OpenSearch/docs/plugins/discovery-ec2.asciidoc

[[discovery-ec2]]
=== EC2 Discovery Plugin

The EC2 discovery plugin provides a list of seed addresses to the
{ref}/modules-discovery-hosts-providers.html[discovery process] by querying the
https://github.com/aws/aws-sdk-java[AWS API] for a list of EC2 instances
matching certain criteria determined by the <<discovery-ec2-usage,plugin
settings>>.

*If you are looking for a hosted solution of {es} on AWS, please visit
https://www.elastic.co/cloud.*

:plugin_name: discovery-ec2
include::install_remove.asciidoc[]

[[discovery-ec2-usage]]
==== Using the EC2 discovery plugin

The `discovery-ec2` plugin allows {es} to find the master-eligible nodes in a
cluster running on AWS EC2 by querying the
https://github.com/aws/aws-sdk-java[AWS API] for the addresses of the EC2
instances running these nodes.

It is normally a good idea to restrict the discovery process just to the
master-eligible nodes in the cluster. This plugin allows you to identify these
nodes by certain criteria including their tags, their membership of security
groups, and their placement within availability zones. The discovery process
will work correctly even if it finds master-ineligible nodes, but master
elections will be more efficient if this can be avoided.

The interaction with the AWS API can be authenticated using the
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html[instance
role], or else custom credentials can be supplied.

===== Enabling EC2 discovery

To enable EC2 discovery, configure {es} to use the `ec2` seed hosts provider:

[source,yaml]
----
discovery.seed_providers: ec2
----

===== Configuring EC2 discovery

EC2 discovery supports a number of settings. Some settings are sensitive and
must be stored in the {ref}/secure-settings.html[{es} keystore]. For example,
to authenticate using a particular access key and secret key, add these keys to
the keystore by running the following commands:

[source,sh]
----
bin/elasticsearch-keystore add discovery.ec2.access_key
bin/elasticsearch-keystore add discovery.ec2.secret_key
----

The available settings for the EC2 discovery plugin are as follows.

`discovery.ec2.access_key` ({ref}/secure-settings.html[Secure], {ref}/secure-settings.html#reloadable-secure-settings[reloadable])::

    An EC2 access key. If set, you must also set `discovery.ec2.secret_key`.
    If unset, `discovery-ec2` will instead use the instance role. This setting
    is sensitive and must be stored in the {es} keystore.

`discovery.ec2.secret_key` ({ref}/secure-settings.html[Secure], {ref}/secure-settings.html#reloadable-secure-settings[reloadable])::

    An EC2 secret key. If set, you must also set `discovery.ec2.access_key`.
    This setting is sensitive and must be stored in the {es} keystore.

`discovery.ec2.session_token` ({ref}/secure-settings.html[Secure], {ref}/secure-settings.html#reloadable-secure-settings[reloadable])::

    An EC2 session token. If set, you must also set `discovery.ec2.access_key`
    and `discovery.ec2.secret_key`. This setting is sensitive and must be
    stored in the {es} keystore.

`discovery.ec2.endpoint`::

    The EC2 service endpoint to which to connect. See
    https://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region to find
    the appropriate endpoint for the region. This setting defaults to
    `ec2.us-east-1.amazonaws.com` which is appropriate for clusters running in
    the `us-east-1` region.

`discovery.ec2.protocol`::

    The protocol to use to connect to the EC2 service endpoint, which may be
    either `http` or `https`. Defaults to `https`.

`discovery.ec2.proxy.host`::

    The address or host name of an HTTP proxy through which to connect to EC2.
    If not set, no proxy is used.

`discovery.ec2.proxy.port`::

    When the address of an HTTP proxy is given in `discovery.ec2.proxy.host`,
    this setting determines the port to use to connect to the proxy. Defaults to
    `80`.

`discovery.ec2.proxy.username` ({ref}/secure-settings.html[Secure], {ref}/secure-settings.html#reloadable-secure-settings[reloadable])::

    When the address of an HTTP proxy is given in `discovery.ec2.proxy.host`,
    this setting determines the username to use to connect to the proxy. When
    not set, no username is used. This setting is sensitive and must be stored
    in the {es} keystore.

`discovery.ec2.proxy.password` ({ref}/secure-settings.html[Secure], {ref}/secure-settings.html#reloadable-secure-settings[reloadable])::

    When the address of an HTTP proxy is given in `discovery.ec2.proxy.host`,
    this setting determines the password to use to connect to the proxy. When
    not set, no password is used. This setting is sensitive and must be stored
    in the {es} keystore.

`discovery.ec2.read_timeout`::

    The socket timeout for connections to EC2,
    {ref}/common-options.html#time-units[including the units]. For example, a
    value of `60s` specifies a 60-second timeout. Defaults to 50 seconds.

`discovery.ec2.groups`::

    A list of the names or IDs of the security groups to use for discovery. The
    `discovery.ec2.any_group` setting determines the behaviour of this setting.
    Defaults to an empty list, meaning that security group membership is
    ignored by EC2 discovery.

`discovery.ec2.any_group`::

    Defaults to `true`, meaning that instances belonging to _any_ of the
    security groups specified in `discovery.ec2.groups` will be used for
    discovery. If set to `false`, only instances that belong to _all_ of the
    security groups specified in `discovery.ec2.groups` will be used for
    discovery.

`discovery.ec2.host_type`::

+
--

Each EC2 instance has a number of different addresses that might be suitable
for discovery. This setting allows you to select which of these addresses is
used by the discovery process. It can be set to one of `private_ip`,
`public_ip`, `private_dns`, `public_dns` or `tag:TAGNAME` where `TAGNAME`
refers to a name of a tag. This setting defaults to `private_ip`.

If you set `discovery.ec2.host_type` to a value of the form `tag:TAGNAME` then
the value of the tag `TAGNAME` attached to each instance will be used as that
instance's address for discovery. Instances which do not have this tag set will
be ignored by the discovery process.

For example if you tag some EC2 instances with a tag named
`elasticsearch-host-name` and set `host_type: tag:elasticsearch-host-name` then
the `discovery-ec2` plugin will read each instance's host name from the value
of the `elasticsearch-host-name` tag.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html[Read more
about EC2 Tags].

--

`discovery.ec2.availability_zones`::

    A list of the names of the availability zones to use for discovery. The
    name of an availability zone is the
    https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html[region
    code followed by a letter], such as `us-east-1a`. Only instances placed in
    one of the given availability zones will be used for discovery.

[[discovery-ec2-filtering]]
`discovery.ec2.tag.TAGNAME`::

+
--

A list of the values of a tag called `TAGNAME` to use for discovery. If set,
only instances that are tagged with one of the given values will be used for
discovery. For instance, the following settings will only use nodes with a
`role` tag set to `master` and an `environment` tag set to either `dev` or
`staging`.

[source,yaml]
----
discovery.ec2.tag.role: master
discovery.ec2.tag.environment: dev,staging
----

NOTE: The names of tags used for discovery may only contain ASCII letters,
numbers, hyphens and underscores. In particular you cannot use tags whose name
includes a colon.

--

`discovery.ec2.node_cache_time`::

    Sets the length of time for which the collection of discovered instances is
    cached. {es} waits at least this long between requests for discovery
    information from the EC2 API. AWS may reject discovery requests if they are
    made too often, and this would cause discovery to fail. Defaults to `10s`.

All **secure** settings of this plugin are
{ref}/secure-settings.html#reloadable-secure-settings[reloadable], allowing you
to update the secure settings for this plugin without needing to restart each
node.


[[discovery-ec2-permissions]]
===== Recommended EC2 permissions

The `discovery-ec2` plugin works by making a `DescribeInstances` call to the AWS
EC2 API. You must configure your AWS account to allow this, which is normally
done using an IAM policy. You can create a custom policy via the IAM Management
Console. It should look similar to this.

[source,js]
----
{
  "Statement": [
    {
      "Action": [
        "ec2:DescribeInstances"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ],
  "Version": "2012-10-17"
}
----
// NOTCONSOLE

[[discovery-ec2-attributes]]
===== Automatic node attributes

The `discovery-ec2` plugin can automatically set the `aws_availability_zone`
node attribute to the availability zone of each node. This node attribute
allows you to ensure that each shard has copies allocated redundantly across
multiple availability zones by using the
{ref}/modules-cluster.html#shard-allocation-awareness[Allocation Awareness]
feature.

In order to enable the automatic definition of the `aws_availability_zone`
attribute, set `cloud.node.auto_attributes` to `true`. For example:

[source,yaml]
----
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
----

The `aws_availability_zone` attribute can be automatically set like this when
using any discovery type. It is not necessary to set `discovery.seed_providers:
ec2`. However this feature does require that the `discovery-ec2` plugin is
installed.

[[discovery-ec2-network-host]]
===== Binding to the correct address

It is important to define `network.host` correctly when deploying a cluster on
EC2. By default each {es} node only binds to `localhost`, which will prevent it
from being discovered by nodes running on any other instances.

You can use the {ref}/modules-network.html[core network host settings] to bind
each node to the desired address, or you can set `network.host` to one of the
following EC2-specific settings provided by the `discovery-ec2` plugin:

[cols="<,<",options="header",]
|==================================================================
|EC2 Host Value      |Description
|`_ec2:privateIpv4_` |The private IP address (ipv4) of the machine.
|`_ec2:privateDns_`  |The private host of the machine.
|`_ec2:publicIpv4_`  |The public IP address (ipv4) of the machine.
|`_ec2:publicDns_`   |The public host of the machine.
|`_ec2:privateIp_`   |Equivalent to `_ec2:privateIpv4_`.
|`_ec2:publicIp_`    |Equivalent to `_ec2:publicIpv4_`.
|`_ec2_`             |Equivalent to `_ec2:privateIpv4_`.
|==================================================================

These values are acceptable when using any discovery type. They do not require
you to set `discovery.seed_providers: ec2`. However they do require that the
`discovery-ec2` plugin is installed.

[[cloud-aws-best-practices]]
==== Best Practices in AWS

This section contains some other information about designing and managing an
{es} cluster on your own AWS infrastructure. If you would prefer to avoid these
operational details then you may be interested in a hosted {es} installation
available on AWS-based infrastructure from https://www.elastic.co/cloud.

===== Storage

EC2 instances offer a number of different kinds of storage. Please be aware of
the following when selecting the storage for your cluster:

* https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html[Instance
Store] is recommended for {es} clusters as it offers excellent performance and
is cheaper than EBS-based storage. {es} is designed to work well with this kind
of ephemeral storage because it replicates each shard across multiple nodes. If
a node fails and its Instance Store is lost then {es} will rebuild any lost
shards from other copies.

* https://aws.amazon.com/ebs/[EBS-based storage] may be acceptable
for smaller clusters (1-2 nodes). Be sure to use provisioned IOPS to ensure
your cluster has satisfactory performance.

* https://aws.amazon.com/efs/[EFS-based storage] is not
recommended or supported as it does not offer satisfactory performance.
Historically, shared network filesystems such as EFS have not always offered
precisely the behaviour that {es} requires of its filesystem, and this has been
known to lead to index corruption. Although EFS offers durability, shared
storage, and the ability to grow and shrink filesystems dynamically, you can
achieve the same benefits using {es} directly.

===== Choice of AMI

Prefer the https://aws.amazon.com/amazon-linux-ami/[Amazon Linux AMIs] as these
allow you to benefit from the lightweight nature, support, and EC2-specific
performance enhancements that these images offer.

===== Networking

* Smaller instance types have limited network performance, in terms of both
https://lab.getbase.com/how-we-discovered-limitations-on-the-aws-tcp-stack/[bandwidth
and number of connections]. If networking is a bottleneck, avoid
https://aws.amazon.com/ec2/instance-types/[instance types] with networking
labelled as `Moderate` or `Low`.

* It is a good idea to distribute your nodes across multiple
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html[availability
zones] and use {ref}/modules-cluster.html#shard-allocation-awareness[shard
allocation awareness] to ensure that each shard has copies in more than one
availability zone.

* Do not span a cluster across regions. {es} expects that node-to-node
connections within a cluster are reasonably reliable and offer high bandwidth
and low latency, and these properties do not hold for connections between
regions. Although an {es} cluster will behave correctly when node-to-node
connections are unreliable or slow, it is not optimised for this case and its
performance may suffer. If you wish to geographically distribute your data, you
should provision multiple clusters and use features such as
{ref}/modules-cross-cluster-search.html[cross-cluster search] and
{ref}/xpack-ccr.html[cross-cluster replication].

===== Other recommendations

* If you have split your nodes into roles, consider
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html[tagging the
EC2 instances] by role to make it easier to filter and view your EC2 instances
in the AWS console.

* Consider
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html#Using_ChangingDisableAPITermination[enabling
termination protection] for all of your data and master-eligible nodes. This
will help to prevent accidental termination of these nodes which could
temporarily reduce the resilience of the cluster and which could cause a
potentially disruptive reallocation of shards.

* If running your cluster using one or more
https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html[auto-scaling
groups], consider protecting your data and master-eligible nodes
https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-instance-termination.html#instance-protection-instance[against
termination during scale-in]. This will help to prevent automatic termination
of these nodes which could temporarily reduce the resilience of the cluster and
which could cause a potentially disruptive reallocation of shards. If these
instances are protected against termination during scale-in then you can use
{ref}/shard-allocation-filtering.html[shard allocation filtering] to gracefully
migrate any data off these nodes before terminating them manually.