migrate branch for cloud-gce
This commit is contained in:
Simon Willnauer
commit
3ab627b72f
|
|
@ -0,0 +1,5 @@
|
|||
ElasticSearch
|
||||
Copyright 2009-2013 ElasticSearch
|
||||
|
||||
This product includes software developed by The Apache Software
|
||||
Foundation (http://www.apache.org/).
|
||||
|
|
@ -0,0 +1,445 @@
|
|||
Google Compute Engine Cloud Plugin for Elasticsearch
|
||||
====================================================
|
||||
|
||||
The GCE Cloud plugin allows to use GCE API for the unicast discovery mechanism.
|
||||
|
||||
In order to install the plugin, run:
|
||||
|
||||
```sh
|
||||
bin/plugin install elasticsearch/elasticsearch-cloud-gce/2.5.0
|
||||
```
|
||||
|
||||
You need to install a version matching your Elasticsearch version:
|
||||
|
||||
| Elasticsearch | GCE Cloud Plugin | Docs |
|
||||
|------------------------|-------------------|------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| master | Build from source | See below |
|
||||
| es-1.x | Build from source | [2.6.0-SNAPSHOT](https://github.com/elasticsearch/elasticsearch-cloud-gce/tree/es-1.x/#google-compute-engine-cloud-plugin-for-elasticsearch)|
|
||||
| es-1.5 | 2.5.0 | [2.5.0](https://github.com/elastic/elasticsearch-cloud-gce/tree/v2.5.0/#version-250-for-elasticsearch-15) |
|
||||
| es-1.4 | 2.4.1 | [2.4.1](https://github.com/elasticsearch/elasticsearch-cloud-gce/tree/v2.4.1/#version-241-for-elasticsearch-14) |
|
||||
| es-1.3 | 2.3.0 | [2.3.0](https://github.com/elasticsearch/elasticsearch-cloud-gce/tree/v2.3.0/#version-230-for-elasticsearch-13) |
|
||||
| es-1.2 | 2.2.0 | [2.2.0](https://github.com/elasticsearch/elasticsearch-cloud-gce/tree/v2.2.0/#google-compute-engine-cloud-plugin-for-elasticsearch)|
|
||||
| es-1.1 | 2.1.2 | [2.1.2](https://github.com/elasticsearch/elasticsearch-cloud-gce/tree/v2.1.2/#google-compute-engine-cloud-plugin-for-elasticsearch)|
|
||||
| es-1.0 | 2.0.1 | [2.0.1](https://github.com/elasticsearch/elasticsearch-cloud-gce/tree/v2.0.1/#google-compute-engine-cloud-plugin-for-elasticsearch)|
|
||||
| es-0.90 | 1.3.0 | [1.3.0](https://github.com/elasticsearch/elasticsearch-cloud-gce/tree/v1.3.0/#google-compute-engine-cloud-plugin-for-elasticsearch)|
|
||||
|
||||
To build a `SNAPSHOT` version, you need to build it with Maven:
|
||||
|
||||
```bash
|
||||
mvn clean install
|
||||
plugin --install cloud-gce \
|
||||
--url file:target/releases/elasticsearch-cloud-gce-X.X.X-SNAPSHOT.zip
|
||||
```
|
||||
|
||||
|
||||
Google Compute Engine Virtual Machine Discovery
|
||||
===============================
|
||||
|
||||
Google Compute Engine VM discovery allows to use the google APIs to perform automatic discovery (similar to multicast in non hostile
|
||||
multicast environments). Here is a simple sample configuration:
|
||||
|
||||
```yaml
|
||||
cloud:
|
||||
gce:
|
||||
project_id: <your-google-project-id>
|
||||
zone: <your-zone>
|
||||
discovery:
|
||||
type: gce
|
||||
```
|
||||
|
||||
How to start (short story)
|
||||
--------------------------
|
||||
|
||||
* Create Google Compute Engine instance (with compute rw permissions)
|
||||
* Install Elasticsearch
|
||||
* Install Google Compute Engine Cloud plugin
|
||||
* Modify `elasticsearch.yml` file
|
||||
* Start Elasticsearch
|
||||
|
||||
How to start (long story)
|
||||
--------------------------
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Before starting, you should have:
|
||||
|
||||
* Your project ID. Let's say here `es-cloud`. Get it from [Google APIS Console](https://code.google.com/apis/console/).
|
||||
* [Google Cloud SDK](https://developers.google.com/cloud/sdk/)
|
||||
|
||||
If you did not set it yet, you can define your default project you will work on:
|
||||
|
||||
```sh
|
||||
gcloud config set project es-cloud
|
||||
```
|
||||
|
||||
### Creating your first instance
|
||||
|
||||
|
||||
```sh
|
||||
gcutil addinstance myesnode1 \
|
||||
--service_account_scope=compute-rw,storage-full \
|
||||
--persistent_boot_disk
|
||||
```
|
||||
|
||||
You will be asked to open a link in your browser. Login and allow access to listed services.
|
||||
You will get back a verification code. Copy and paste it in your terminal.
|
||||
|
||||
You should get `Authentication successful.` message.
|
||||
|
||||
Then, choose your zone. Let's say here that we choose `europe-west1-a`.
|
||||
|
||||
Choose your compute instance size. Let's say `f1-micro`.
|
||||
|
||||
Choose your OS. Let's say `projects/debian-cloud/global/images/debian-7-wheezy-v20140606`.
|
||||
|
||||
You may be asked to create a ssh key. Follow instructions to create one.
|
||||
|
||||
When done, a report like this one should appears:
|
||||
|
||||
```sh
|
||||
Table of resources:
|
||||
|
||||
+-----------+--------------+-------+---------+--------------+----------------+----------------+----------------+---------+----------------+
|
||||
| name | machine-type | image | network | network-ip | external-ip | disks | zone | status | status-message |
|
||||
+-----------+--------------+-------+---------+--------------+----------------+----------------+----------------+---------+----------------+
|
||||
| myesnode1 | f1-micro | | default | 10.240.20.57 | 192.158.29.199 | boot-myesnode1 | europe-west1-a | RUNNING | |
|
||||
+-----------+--------------+-------+---------+--------------+----------------+----------------+----------------+---------+----------------+
|
||||
```
|
||||
|
||||
You can now connect to your instance:
|
||||
|
||||
```
|
||||
# Connect using google cloud SDK
|
||||
gcloud compute ssh myesnode1 --zone europe-west1-a
|
||||
|
||||
# Or using SSH with external IP address
|
||||
ssh -i ~/.ssh/google_compute_engine 192.158.29.199
|
||||
```
|
||||
|
||||
*Note Regarding Service Account Permissions*
|
||||
|
||||
It's important when creating an instance that the correct permissions are set. At a minimum, you must ensure you have:
|
||||
|
||||
```
|
||||
service_account_scope=compute-rw
|
||||
```
|
||||
|
||||
Failing to set this will result in unauthorized messages when starting Elasticsearch.
|
||||
See [Machine Permissions](#machine-permissions).
|
||||
|
||||
Once connected, install Elasticsearch:
|
||||
|
||||
```sh
|
||||
sudo apt-get update
|
||||
|
||||
# Download Elasticsearch
|
||||
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.deb
|
||||
|
||||
# Prepare Java installation
|
||||
sudo apt-get install java7-runtime-headless
|
||||
|
||||
# Prepare Elasticsearch installation
|
||||
sudo dpkg -i elasticsearch-1.2.1.deb
|
||||
```
|
||||
|
||||
### Install elasticsearch cloud gce plugin
|
||||
|
||||
Install the plugin:
|
||||
|
||||
```sh
|
||||
# Use Plugin Manager to install it
|
||||
sudo /usr/share/elasticsearch/bin/plugin --install elasticsearch/elasticsearch-cloud-gce/2.2.0
|
||||
|
||||
# Configure it:
|
||||
sudo vi /etc/elasticsearch/elasticsearch.yml
|
||||
```
|
||||
|
||||
And add the following lines:
|
||||
|
||||
```yaml
|
||||
cloud:
|
||||
gce:
|
||||
project_id: es-cloud
|
||||
zone: europe-west1-a
|
||||
discovery:
|
||||
type: gce
|
||||
```
|
||||
|
||||
|
||||
Start elasticsearch:
|
||||
|
||||
```sh
|
||||
sudo /etc/init.d/elasticsearch start
|
||||
```
|
||||
|
||||
If anything goes wrong, you should check logs:
|
||||
|
||||
```sh
|
||||
tail -f /var/log/elasticsearch/elasticsearch.log
|
||||
```
|
||||
|
||||
If needed, you can change log level to `TRACE` by modifying `sudo vi /etc/elasticsearch/logging.yml`:
|
||||
|
||||
```yaml
|
||||
# discovery
|
||||
discovery.gce: TRACE
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Cloning your existing machine
|
||||
|
||||
In order to build a cluster on many nodes, you can clone your configured instance to new nodes.
|
||||
You won't have to reinstall everything!
|
||||
|
||||
First create an image of your running instance and upload it to Google Cloud Storage:
|
||||
|
||||
```sh
|
||||
# Create an image of yur current instance
|
||||
sudo /usr/bin/gcimagebundle -d /dev/sda -o /tmp/
|
||||
|
||||
# An image has been created in `/tmp` directory:
|
||||
ls /tmp
|
||||
e4686d7f5bf904a924ae0cfeb58d0827c6d5b966.image.tar.gz
|
||||
|
||||
# Upload your image to Google Cloud Storage:
|
||||
# Create a bucket to hold your image, let's say `esimage`:
|
||||
gsutil mb gs://esimage
|
||||
|
||||
# Copy your image to this bucket:
|
||||
gsutil cp /tmp/e4686d7f5bf904a924ae0cfeb58d0827c6d5b966.image.tar.gz gs://esimage
|
||||
|
||||
# Then add your image to images collection:
|
||||
gcutil addimage elasticsearch-1-2-1 gs://esimage/e4686d7f5bf904a924ae0cfeb58d0827c6d5b966.image.tar.gz
|
||||
|
||||
# If the previous command did not work for you, logout from your instance
|
||||
# and launch the same command from your local machine.
|
||||
```
|
||||
|
||||
### Start new instances
|
||||
|
||||
As you have now an image, you can create as many instances as you need:
|
||||
|
||||
```sh
|
||||
# Just change node name (here myesnode2)
|
||||
gcutil addinstance --image=elasticsearch-1-2-1 myesnode2
|
||||
|
||||
# If you want to provide all details directly, you can use:
|
||||
gcutil addinstance --image=elasticsearch-1-2-1 \
|
||||
--kernel=projects/google/global/kernels/gce-v20130603 myesnode2 \
|
||||
--zone europe-west1-a --machine_type f1-micro --service_account_scope=compute-rw \
|
||||
--persistent_boot_disk
|
||||
```
|
||||
|
||||
### Remove an instance (aka shut it down)
|
||||
|
||||
You can use [Google Cloud Console](https://cloud.google.com/console) or CLI to manage your instances:
|
||||
|
||||
```sh
|
||||
# Stopping and removing instances
|
||||
gcutil deleteinstance myesnode1 myesnode2 \
|
||||
--zone=europe-west1-a
|
||||
|
||||
# Consider removing disk as well if you don't need them anymore
|
||||
gcutil deletedisk boot-myesnode1 boot-myesnode2 \
|
||||
--zone=europe-west1-a
|
||||
```
|
||||
|
||||
Using zones
|
||||
-----------
|
||||
|
||||
`cloud.gce.zone` helps to retrieve instances running in a given zone. It should be one of the
|
||||
[GCE supported zones](https://developers.google.com/compute/docs/zones#available).
|
||||
|
||||
The GCE discovery can support multi zones although you need to be aware of network latency between zones.
|
||||
To enable discovery across more than one zone, just enter add your zone list to `cloud.gce.zone` setting:
|
||||
|
||||
```yaml
|
||||
cloud:
|
||||
gce:
|
||||
project_id: <your-google-project-id>
|
||||
zone: ["<your-zone1>", "<your-zone2>"]
|
||||
discovery:
|
||||
type: gce
|
||||
```
|
||||
|
||||
|
||||
|
||||
Filtering by tags
|
||||
-----------------
|
||||
|
||||
The GCE discovery can also filter machines to include in the cluster based on tags using `discovery.gce.tags` settings.
|
||||
For example, setting `discovery.gce.tags` to `dev` will only filter instances having a tag set to `dev`. Several tags
|
||||
set will require all of those tags to be set for the instance to be included.
|
||||
|
||||
One practical use for tag filtering is when an GCE cluster contains many nodes that are not running
|
||||
elasticsearch. In this case (particularly with high ping_timeout values) there is a risk that a new node's discovery
|
||||
phase will end before it has found the cluster (which will result in it declaring itself master of a new cluster
|
||||
with the same name - highly undesirable). Adding tag on elasticsearch GCE nodes and then filtering by that
|
||||
tag will resolve this issue.
|
||||
|
||||
Add your tag when building the new instance:
|
||||
|
||||
```sh
|
||||
gcutil --project=es-cloud addinstance myesnode1 \
|
||||
--service_account_scope=compute-rw \
|
||||
--persistent_boot_disk \
|
||||
--tags=elasticsearch,dev
|
||||
```
|
||||
|
||||
Then, define it in `elasticsearch.yml`:
|
||||
|
||||
```yaml
|
||||
cloud:
|
||||
gce:
|
||||
project_id: es-cloud
|
||||
zone: europe-west1-a
|
||||
discovery:
|
||||
type: gce
|
||||
gce:
|
||||
tags: elasticsearch, dev
|
||||
```
|
||||
|
||||
Changing default transport port
|
||||
-------------------------------
|
||||
|
||||
By default, elasticsearch GCE plugin assumes that you run elasticsearch on 9300 default port.
|
||||
But you can specify the port value elasticsearch is meant to use using google compute engine metadata `es_port`:
|
||||
|
||||
### When creating instance
|
||||
|
||||
Add `--metadata=es_port:9301` option:
|
||||
|
||||
```sh
|
||||
# when creating first instance
|
||||
gcutil addinstance myesnode1 \
|
||||
--service_account_scope=compute-rw,storage-full \
|
||||
--persistent_boot_disk \
|
||||
--metadata=es_port:9301
|
||||
|
||||
# when creating an instance from an image
|
||||
gcutil addinstance --image=elasticsearch-1-0-0-RC1 \
|
||||
--kernel=projects/google/global/kernels/gce-v20130603 myesnode2 \
|
||||
--zone europe-west1-a --machine_type f1-micro --service_account_scope=compute-rw \
|
||||
--persistent_boot_disk --metadata=es_port:9301
|
||||
```
|
||||
|
||||
### On a running instance
|
||||
|
||||
```sh
|
||||
# Get metadata fingerprint
|
||||
gcutil getinstance myesnode1 --zone=europe-west1-a
|
||||
+------------------------+---------------------------------------------------------------------------------------------------------+
|
||||
| property | value |
|
||||
+------------------------+---------------------------------------------------------------------------------------------------------+
|
||||
| metadata | |
|
||||
| fingerprint | 42WmSpB8rSM= |
|
||||
+------------------------+---------------------------------------------------------------------------------------------------------+
|
||||
|
||||
# Use that fingerprint
|
||||
gcutil setinstancemetadata myesnode1 \
|
||||
--zone=europe-west1-a \
|
||||
--metadata=es_port:9301 \
|
||||
--fingerprint=42WmSpB8rSM=
|
||||
```
|
||||
|
||||
|
||||
Tips
|
||||
----
|
||||
|
||||
### Store project id locally
|
||||
|
||||
If you don't want to repeat the project id each time, you can save it in `~/.gcutil.flags` file using:
|
||||
|
||||
```sh
|
||||
gcutil getproject --project=es-cloud --cache_flag_values
|
||||
```
|
||||
|
||||
`~/.gcutil.flags` file now contains:
|
||||
|
||||
```
|
||||
--project=es-cloud
|
||||
```
|
||||
|
||||
### Machine Permissions
|
||||
|
||||
**Creating machines with gcutil**
|
||||
|
||||
Ensure the following flags are set:
|
||||
|
||||
````
|
||||
--service_account_scope=compute-rw
|
||||
```
|
||||
|
||||
**Creating with console (web)**
|
||||
|
||||
When creating an instance using the web portal, click **Show advanced options**.
|
||||
|
||||
At the bottom of the page, under `PROJECT ACCESS`, choose `>> Compute >> Read Write`.
|
||||
|
||||
**Creating with knife google**
|
||||
|
||||
Set the service account scopes when creating the machine:
|
||||
|
||||
```
|
||||
$ knife google server create www1 \
|
||||
-m n1-standard-1 \
|
||||
-I debian-7-wheezy-v20131120 \
|
||||
-Z us-central1-a \
|
||||
-i ~/.ssh/id_rsa \
|
||||
-x jdoe \
|
||||
--gce-service-account-scopes https://www.googleapis.com/auth/compute.full_control
|
||||
```
|
||||
|
||||
Or, you may use the alias:
|
||||
|
||||
```
|
||||
--gce-service-account-scopes compute-rw
|
||||
```
|
||||
|
||||
If you have created a machine without the correct permissions, you will see `403 unauthorized` error messages. The only
|
||||
way to alter these permissions is to delete the instance (NOT THE DISK). Then create another with the correct permissions.
|
||||
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Integrations tests in this plugin require working GCE configuration and therefore disabled by default.
|
||||
To enable tests prepare a config file elasticsearch.yml with the following content:
|
||||
|
||||
```
|
||||
cloud:
|
||||
gce:
|
||||
project_id: es-cloud
|
||||
zone: europe-west1-a
|
||||
discovery:
|
||||
type: gce
|
||||
```
|
||||
|
||||
Replaces `project_id` and `zone` with your settings.
|
||||
|
||||
To run test:
|
||||
|
||||
```sh
|
||||
mvn -Dtests.gce=true -Dtests.config=/path/to/config/file/elasticsearch.yml clean test
|
||||
```
|
||||
|
||||
|
||||
License
|
||||
-------
|
||||
|
||||
This software is licensed under the Apache 2 license, quoted below.
|
||||
|
||||
Copyright 2009-2014 Elasticsearch <http://www.elasticsearch.org>
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not
|
||||
use this file except in compliance with the License. You may obtain a copy of
|
||||
the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
License for the specific language governing permissions and limitations under
|
||||
the License.
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- Licensed to Elasticsearch under one or more contributor
|
||||
license agreements. See the NOTICE file distributed with this work for additional
|
||||
information regarding copyright ownership. Elasticsearch licenses this file to you
|
||||
under the Apache License, Version 2.0 (the "License"); you may not use this
|
||||
file except in compliance with the License. You may obtain a copy of the
|
||||
License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by
|
||||
applicable law or agreed to in writing, software distributed under the License
|
||||
is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the specific language
|
||||
governing permissions and limitations under the License. -->
|
||||
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<groupId>org.elasticsearch.plugin</groupId>
|
||||
<artifactId>elasticsearch-cloud-gce</artifactId>
|
||||
|
||||
|
||||
<name>Elasticsearch Google Compute Engine cloud plugin</name>
|
||||
<description>The Google Compute Engine (GCE) Cloud plugin allows to use GCE API for the unicast discovery mechanism.</description>
|
||||
|
||||
<parent>
|
||||
<groupId>org.elasticsearch</groupId>
|
||||
<artifactId>elasticsearch-plugin</artifactId>
|
||||
<version>2.0.0-SNAPSHOT</version>
|
||||
</parent>
|
||||
|
||||
|
||||
<properties>
|
||||
<google.gce.version>v1-rev59-1.20.0</google.gce.version>
|
||||
<es.plugin.port>9300</es.plugin.port>
|
||||
<!-- currently has no unit tests -->
|
||||
<tests.ifNoTests>warn</tests.ifNoTests>
|
||||
<tests.jvms>1</tests.jvms>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<!-- Google APIs -->
|
||||
<dependency>
|
||||
<groupId>com.google.apis</groupId>
|
||||
<artifactId>google-api-services-compute</artifactId>
|
||||
<version>${google.gce.version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<extensions>
|
||||
<extension>
|
||||
<groupId>org.kuali.maven.wagons</groupId>
|
||||
<artifactId>maven-s3-wagon</artifactId>
|
||||
<version>1.1.19</version>
|
||||
</extension>
|
||||
</extensions>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-assembly-plugin</artifactId>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
</project>
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
<?xml version="1.0"?>
|
||||
<!-- Licensed to ElasticSearch under one or more contributor
|
||||
license agreements. See the NOTICE file distributed with this work for additional
|
||||
information regarding copyright ownership. ElasticSearch licenses this file to you
|
||||
under the Apache License, Version 2.0 (the "License"); you may not use this
|
||||
file except in compliance with the License. You may obtain a copy of the
|
||||
License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by
|
||||
applicable law or agreed to in writing, software distributed under the License
|
||||
is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the specific language
|
||||
governing permissions and limitations under the License. -->
|
||||
|
||||
<assembly>
|
||||
<id>plugin</id>
|
||||
<formats>
|
||||
<format>zip</format>
|
||||
</formats>
|
||||
<includeBaseDirectory>false</includeBaseDirectory>
|
||||
<dependencySets>
|
||||
<dependencySet>
|
||||
<outputDirectory>/</outputDirectory>
|
||||
<useProjectArtifact>true</useProjectArtifact>
|
||||
<useTransitiveFiltering>true</useTransitiveFiltering>
|
||||
<excludes>
|
||||
<exclude>org.elasticsearch:elasticsearch</exclude>
|
||||
</excludes>
|
||||
</dependencySet>
|
||||
<dependencySet>
|
||||
<outputDirectory>/</outputDirectory>
|
||||
<useProjectArtifact>true</useProjectArtifact>
|
||||
<useTransitiveFiltering>true</useTransitiveFiltering>
|
||||
<includes>
|
||||
<include>com.google.api-client:google-api-client</include>
|
||||
<include>com.google.apis:google-api-services-compute</include>
|
||||
<include>com.google.oauth-client:google-oauth-client-jetty</include>
|
||||
<include>com.google.http-client:google-http-client-jackson2</include>
|
||||
</includes>
|
||||
</dependencySet>
|
||||
</dependencySets>
|
||||
</assembly>
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.cloud.gce;
|
||||
|
||||
import com.google.api.services.compute.model.Instance;
|
||||
|
||||
import java.util.Collection;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
public interface GceComputeService {
|
||||
static final public class Fields {
|
||||
public static final String PROJECT = "cloud.gce.project_id";
|
||||
public static final String ZONE = "cloud.gce.zone";
|
||||
public static final String REFRESH = "cloud.gce.refresh_interval";
|
||||
public static final String TAGS = "discovery.gce.tags";
|
||||
public static final String VERSION = "Elasticsearch/GceCloud/1.0";
|
||||
}
|
||||
|
||||
public Collection<Instance> instances();
|
||||
}
|
||||
|
|
@ -0,0 +1,161 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.cloud.gce;
|
||||
|
||||
import com.google.api.client.googleapis.compute.ComputeCredential;
|
||||
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
|
||||
import com.google.api.client.http.HttpTransport;
|
||||
import com.google.api.client.json.JsonFactory;
|
||||
import com.google.api.client.json.jackson2.JacksonFactory;
|
||||
import com.google.api.services.compute.Compute;
|
||||
import com.google.api.services.compute.model.Instance;
|
||||
import com.google.api.services.compute.model.InstanceList;
|
||||
import com.google.common.base.Function;
|
||||
import com.google.common.collect.Iterables;
|
||||
import com.google.common.collect.Lists;
|
||||
import org.elasticsearch.ElasticsearchException;
|
||||
import org.elasticsearch.common.component.AbstractLifecycleComponent;
|
||||
import org.elasticsearch.common.inject.Inject;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.common.settings.SettingsFilter;
|
||||
import org.elasticsearch.common.unit.TimeValue;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
public class GceComputeServiceImpl extends AbstractLifecycleComponent<GceComputeServiceImpl>
|
||||
implements GceComputeService {
|
||||
|
||||
private final String project;
|
||||
private final List<String> zoneList;
|
||||
|
||||
// Forcing Google Token API URL as set in GCE SDK to
|
||||
// http://metadata/computeMetadata/v1/instance/service-accounts/default/token
|
||||
// See https://developers.google.com/compute/docs/metadata#metadataserver
|
||||
private static final String TOKEN_SERVER_ENCODED_URL = "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token";
|
||||
|
||||
@Override
|
||||
public Collection<Instance> instances() {
|
||||
|
||||
logger.debug("get instances for project [{}], zoneList [{}]", project, zoneList);
|
||||
|
||||
List<List<Instance>> instanceListByZone = Lists.transform(zoneList, new Function<String, List<Instance>>() {
|
||||
@Override
|
||||
public List<Instance> apply(String zoneId) {
|
||||
try {
|
||||
Compute.Instances.List list = client().instances().list(project, zoneId);
|
||||
InstanceList instanceList = list.execute();
|
||||
if (instanceList.isEmpty()) {
|
||||
return new ArrayList();
|
||||
}
|
||||
|
||||
return instanceList.getItems();
|
||||
} catch (IOException e) {
|
||||
logger.warn("Problem fetching instance list for zone {}", zoneId);
|
||||
logger.debug("Full exception:", e);
|
||||
|
||||
return new ArrayList();
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
//Collapse instances from all zones into one neat list
|
||||
List<Instance> instanceList = Lists.newArrayList(Iterables.concat(instanceListByZone));
|
||||
|
||||
if (instanceList.size() == 0) {
|
||||
logger.warn("disabling GCE discovery. Can not get list of nodes");
|
||||
}
|
||||
|
||||
return instanceList;
|
||||
}
|
||||
|
||||
private Compute client;
|
||||
private TimeValue refreshInterval = null;
|
||||
private long lastRefresh;
|
||||
|
||||
/** Global instance of the HTTP transport. */
|
||||
private static HttpTransport HTTP_TRANSPORT;
|
||||
|
||||
/** Global instance of the JSON factory. */
|
||||
private static JsonFactory JSON_FACTORY;
|
||||
|
||||
@Inject
|
||||
public GceComputeServiceImpl(Settings settings, SettingsFilter settingsFilter) {
|
||||
super(settings);
|
||||
this.project = settings.get(Fields.PROJECT);
|
||||
String[] zoneList = settings.getAsArray(Fields.ZONE);
|
||||
this.zoneList = Lists.newArrayList(zoneList);
|
||||
}
|
||||
|
||||
public synchronized Compute client() {
|
||||
if (refreshInterval != null && refreshInterval.millis() != 0) {
|
||||
if (client != null &&
|
||||
(refreshInterval.millis() < 0 || (System.currentTimeMillis() - lastRefresh) < refreshInterval.millis())) {
|
||||
if (logger.isTraceEnabled()) logger.trace("using cache to retrieve client");
|
||||
return client;
|
||||
}
|
||||
lastRefresh = System.currentTimeMillis();
|
||||
}
|
||||
|
||||
try {
|
||||
HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();
|
||||
JSON_FACTORY = new JacksonFactory();
|
||||
|
||||
logger.info("starting GCE discovery service");
|
||||
|
||||
ComputeCredential credential = new ComputeCredential.Builder(HTTP_TRANSPORT, JSON_FACTORY)
|
||||
.setTokenServerEncodedUrl(TOKEN_SERVER_ENCODED_URL)
|
||||
.build();
|
||||
|
||||
credential.refreshToken();
|
||||
|
||||
logger.debug("token [{}] will expire in [{}] s", credential.getAccessToken(), credential.getExpiresInSeconds());
|
||||
refreshInterval = TimeValue.timeValueSeconds(credential.getExpiresInSeconds()-1);
|
||||
|
||||
// Once done, let's use this token
|
||||
this.client = new Compute.Builder(HTTP_TRANSPORT, JSON_FACTORY, null)
|
||||
.setApplicationName(Fields.VERSION)
|
||||
.setHttpRequestInitializer(credential)
|
||||
.build();
|
||||
} catch (Exception e) {
|
||||
logger.warn("unable to start GCE discovery service: {} : {}", e.getClass().getName(), e.getMessage());
|
||||
throw new IllegalArgumentException("unable to start GCE discovery service", e);
|
||||
}
|
||||
|
||||
return this.client;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void doStart() throws ElasticsearchException {
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void doStop() throws ElasticsearchException {
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void doClose() throws ElasticsearchException {
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.cloud.gce;
|
||||
|
||||
import org.elasticsearch.common.inject.AbstractModule;
|
||||
import org.elasticsearch.common.inject.Inject;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
public class GceModule extends AbstractModule {
|
||||
private Settings settings;
|
||||
|
||||
@Inject
|
||||
public GceModule(Settings settings) {
|
||||
this.settings = settings;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void configure() {
|
||||
bind(GceComputeService.class)
|
||||
.to(settings.getAsClass("cloud.gce.api.impl", GceComputeServiceImpl.class))
|
||||
.asEagerSingleton();
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.discovery.gce;
|
||||
|
||||
import org.elasticsearch.cluster.ClusterName;
|
||||
import org.elasticsearch.cluster.ClusterService;
|
||||
import org.elasticsearch.cluster.settings.ClusterDynamicSettings;
|
||||
import org.elasticsearch.cluster.settings.DynamicSettings;
|
||||
import org.elasticsearch.common.inject.Inject;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.discovery.DiscoverySettings;
|
||||
import org.elasticsearch.discovery.zen.ZenDiscovery;
|
||||
import org.elasticsearch.discovery.zen.elect.ElectMasterService;
|
||||
import org.elasticsearch.discovery.zen.ping.ZenPingService;
|
||||
import org.elasticsearch.node.settings.NodeSettingsService;
|
||||
import org.elasticsearch.threadpool.ThreadPool;
|
||||
import org.elasticsearch.transport.TransportService;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
public class GceDiscovery extends ZenDiscovery {
|
||||
|
||||
@Inject
|
||||
public GceDiscovery(Settings settings, ClusterName clusterName, ThreadPool threadPool, TransportService transportService,
|
||||
ClusterService clusterService, NodeSettingsService nodeSettingsService, ZenPingService pingService,
|
||||
DiscoverySettings discoverySettings,
|
||||
ElectMasterService electMasterService, @ClusterDynamicSettings DynamicSettings dynamicSettings) {
|
||||
super(settings, clusterName, threadPool, transportService, clusterService, nodeSettingsService,
|
||||
pingService, electMasterService, discoverySettings, dynamicSettings);
|
||||
|
||||
// TODO Add again force disable multicast
|
||||
// See related issue in AWS plugin https://github.com/elastic/elasticsearch-cloud-aws/issues/179
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.discovery.gce;
|
||||
|
||||
import org.elasticsearch.discovery.Discovery;
|
||||
import org.elasticsearch.discovery.zen.ZenDiscoveryModule;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
public class GceDiscoveryModule extends ZenDiscoveryModule {
|
||||
|
||||
@Override
|
||||
protected void bindDiscovery() {
|
||||
bind(Discovery.class).to(GceDiscovery.class).asEagerSingleton();
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,256 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.discovery.gce;
|
||||
|
||||
import com.google.api.services.compute.model.AccessConfig;
|
||||
import com.google.api.services.compute.model.Instance;
|
||||
import com.google.api.services.compute.model.NetworkInterface;
|
||||
import org.elasticsearch.Version;
|
||||
import org.elasticsearch.cloud.gce.GceComputeService;
|
||||
import org.elasticsearch.cluster.node.DiscoveryNode;
|
||||
import org.elasticsearch.common.Strings;
|
||||
import org.elasticsearch.common.component.AbstractComponent;
|
||||
import org.elasticsearch.common.inject.Inject;
|
||||
import org.elasticsearch.common.network.NetworkService;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.common.transport.TransportAddress;
|
||||
import org.elasticsearch.common.unit.TimeValue;
|
||||
import org.elasticsearch.discovery.zen.ping.unicast.UnicastHostsProvider;
|
||||
import org.elasticsearch.transport.TransportService;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.net.InetAddress;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collection;
|
||||
import java.util.List;
|
||||
|
||||
import static org.elasticsearch.cloud.gce.GceComputeService.Fields;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
public class GceUnicastHostsProvider extends AbstractComponent implements UnicastHostsProvider {
|
||||
|
||||
static final class Status {
|
||||
private static final String TERMINATED = "TERMINATED";
|
||||
}
|
||||
|
||||
private final GceComputeService gceComputeService;
|
||||
private TransportService transportService;
|
||||
private NetworkService networkService;
|
||||
|
||||
private final Version version;
|
||||
private final String project;
|
||||
private final String zone;
|
||||
private final String[] tags;
|
||||
|
||||
private final TimeValue refreshInterval;
|
||||
private long lastRefresh;
|
||||
private List<DiscoveryNode> cachedDiscoNodes;
|
||||
|
||||
@Inject
|
||||
public GceUnicastHostsProvider(Settings settings, GceComputeService gceComputeService,
|
||||
TransportService transportService,
|
||||
NetworkService networkService,
|
||||
Version version) {
|
||||
super(settings);
|
||||
this.gceComputeService = gceComputeService;
|
||||
this.transportService = transportService;
|
||||
this.networkService = networkService;
|
||||
this.version = version;
|
||||
|
||||
this.refreshInterval = settings.getAsTime(Fields.REFRESH, TimeValue.timeValueSeconds(0));
|
||||
this.project = settings.get(Fields.PROJECT);
|
||||
this.zone = settings.get(Fields.ZONE);
|
||||
|
||||
// Check that we have all needed properties
|
||||
checkProperty(Fields.PROJECT, project);
|
||||
checkProperty(Fields.ZONE, zone);
|
||||
|
||||
this.tags = settings.getAsArray(Fields.TAGS);
|
||||
if (logger.isDebugEnabled()) {
|
||||
logger.debug("using tags {}", Arrays.asList(this.tags));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* We build the list of Nodes from GCE Management API
|
||||
* Information can be cached using `plugins.refresh_interval` property if needed.
|
||||
* Setting `plugins.refresh_interval` to `-1` will cause infinite caching.
|
||||
* Setting `plugins.refresh_interval` to `0` will disable caching (default).
|
||||
*/
|
||||
@Override
|
||||
public List<DiscoveryNode> buildDynamicNodes() {
|
||||
if (refreshInterval.millis() != 0) {
|
||||
if (cachedDiscoNodes != null &&
|
||||
(refreshInterval.millis() < 0 || (System.currentTimeMillis() - lastRefresh) < refreshInterval.millis())) {
|
||||
if (logger.isTraceEnabled()) logger.trace("using cache to retrieve node list");
|
||||
return cachedDiscoNodes;
|
||||
}
|
||||
lastRefresh = System.currentTimeMillis();
|
||||
}
|
||||
logger.debug("start building nodes list using GCE API");
|
||||
|
||||
cachedDiscoNodes = new ArrayList<>();
|
||||
String ipAddress = null;
|
||||
try {
|
||||
InetAddress inetAddress = networkService.resolvePublishHostAddress(null);
|
||||
if (inetAddress != null) {
|
||||
ipAddress = inetAddress.getHostAddress();
|
||||
}
|
||||
} catch (IOException e) {
|
||||
// We can't find the publish host address... Hmmm. Too bad :-(
|
||||
// We won't simply filter it
|
||||
}
|
||||
|
||||
try {
|
||||
Collection<Instance> instances = gceComputeService.instances();
|
||||
|
||||
if (instances == null) {
|
||||
logger.trace("no instance found for project [{}], zone [{}].", this.project, this.zone);
|
||||
return cachedDiscoNodes;
|
||||
}
|
||||
|
||||
for (Instance instance : instances) {
|
||||
String name = instance.getName();
|
||||
String type = instance.getMachineType();
|
||||
|
||||
String status = instance.getStatus();
|
||||
logger.trace("gce instance {} with status {} found.", name, status);
|
||||
|
||||
// We don't want to connect to TERMINATED status instances
|
||||
// See https://github.com/elasticsearch/elasticsearch-cloud-gce/issues/3
|
||||
if (Status.TERMINATED.equals(status)) {
|
||||
logger.debug("node {} is TERMINATED. Ignoring", name);
|
||||
continue;
|
||||
}
|
||||
|
||||
// see if we need to filter by tag
|
||||
boolean filterByTag = false;
|
||||
if (tags.length > 0) {
|
||||
logger.trace("start filtering instance {} with tags {}.", name, tags);
|
||||
if (instance.getTags() == null || instance.getTags().isEmpty()
|
||||
|| instance.getTags().getItems() == null || instance.getTags().getItems().isEmpty()) {
|
||||
// If this instance have no tag, we filter it
|
||||
logger.trace("no tags for this instance but we asked for tags. {} won't be part of the cluster.", name);
|
||||
filterByTag = true;
|
||||
} else {
|
||||
// check that all tags listed are there on the instance
|
||||
logger.trace("comparing instance tags {} with tags filter {}.", instance.getTags().getItems(), tags);
|
||||
for (String tag : tags) {
|
||||
boolean found = false;
|
||||
for (String instancetag : instance.getTags().getItems()) {
|
||||
if (instancetag.equals(tag)) {
|
||||
found = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (!found) {
|
||||
filterByTag = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if (filterByTag) {
|
||||
logger.trace("filtering out instance {} based tags {}, not part of {}", name, tags,
|
||||
instance.getTags() == null || instance.getTags().getItems() == null ? "" : "");
|
||||
continue;
|
||||
} else {
|
||||
logger.trace("instance {} with tags {} is added to discovery", name, tags);
|
||||
}
|
||||
|
||||
String ip_public = null;
|
||||
String ip_private = null;
|
||||
|
||||
List<NetworkInterface> interfaces = instance.getNetworkInterfaces();
|
||||
|
||||
for (NetworkInterface networkInterface : interfaces) {
|
||||
if (ip_public == null) {
|
||||
// Trying to get Public IP Address (For future use)
|
||||
if (networkInterface.getAccessConfigs() != null) {
|
||||
for (AccessConfig accessConfig : networkInterface.getAccessConfigs()) {
|
||||
if (Strings.hasText(accessConfig.getNatIP())) {
|
||||
ip_public = accessConfig.getNatIP();
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (ip_private == null) {
|
||||
ip_private = networkInterface.getNetworkIP();
|
||||
}
|
||||
|
||||
// If we have both public and private, we can stop here
|
||||
if (ip_private != null && ip_public != null) break;
|
||||
}
|
||||
|
||||
try {
|
||||
if (ip_private.equals(ipAddress)) {
|
||||
// We found the current node.
|
||||
// We can ignore it in the list of DiscoveryNode
|
||||
logger.trace("current node found. Ignoring {} - {}", name, ip_private);
|
||||
} else {
|
||||
String address = ip_private;
|
||||
// Test if we have es_port metadata defined here
|
||||
if (instance.getMetadata() != null && instance.getMetadata().containsKey("es_port")) {
|
||||
Object es_port = instance.getMetadata().get("es_port");
|
||||
logger.trace("es_port is defined with {}", es_port);
|
||||
if (es_port instanceof String) {
|
||||
address = address.concat(":").concat((String) es_port);
|
||||
} else {
|
||||
// Ignoring other values
|
||||
logger.trace("es_port is instance of {}. Ignoring...", es_port.getClass().getName());
|
||||
}
|
||||
}
|
||||
|
||||
// ip_private is a single IP Address. We need to build a TransportAddress from it
|
||||
TransportAddress[] addresses = transportService.addressesFromString(address);
|
||||
|
||||
// If user has set `es_port` metadata, we don't need to ping all ports
|
||||
// we only limit to 1 addresses, makes no sense to ping 100 ports
|
||||
logger.trace("adding {}, type {}, address {}, transport_address {}, status {}", name, type,
|
||||
ip_private, addresses[0], status);
|
||||
cachedDiscoNodes.add(new DiscoveryNode("#cloud-" + name + "-" + 0, addresses[0], version.minimumCompatibilityVersion()));
|
||||
}
|
||||
} catch (Exception e) {
|
||||
logger.warn("failed to add {}, address {}", e, name, ip_private);
|
||||
}
|
||||
|
||||
}
|
||||
} catch (Throwable e) {
|
||||
logger.warn("Exception caught during discovery {} : {}", e.getClass().getName(), e.getMessage());
|
||||
logger.trace("Exception caught during discovery", e);
|
||||
}
|
||||
|
||||
logger.debug("{} node(s) added", cachedDiscoNodes.size());
|
||||
logger.debug("using dynamic discovery nodes {}", cachedDiscoNodes);
|
||||
|
||||
return cachedDiscoNodes;
|
||||
}
|
||||
|
||||
private void checkProperty(String name, String value) {
|
||||
if (!Strings.hasText(value)) {
|
||||
logger.warn("{} is not set.", name);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,70 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.plugin.cloud.gce;
|
||||
|
||||
import org.elasticsearch.cloud.gce.GceModule;
|
||||
import org.elasticsearch.common.component.LifecycleComponent;
|
||||
import org.elasticsearch.common.inject.Module;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.plugins.AbstractPlugin;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
public class CloudGcePlugin extends AbstractPlugin {
|
||||
|
||||
private final Settings settings;
|
||||
|
||||
public CloudGcePlugin(Settings settings) {
|
||||
this.settings = settings;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String name() {
|
||||
return "cloud-gce";
|
||||
}
|
||||
|
||||
@Override
|
||||
public String description() {
|
||||
return "Cloud Google Compute Engine Plugin";
|
||||
}
|
||||
|
||||
@Override
|
||||
public Collection<Class<? extends Module>> modules() {
|
||||
Collection<Class<? extends Module>> modules = new ArrayList<>();
|
||||
if (settings.getAsBoolean("cloud.enabled", true)) {
|
||||
modules.add(GceModule.class);
|
||||
}
|
||||
return modules;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Collection<Class<? extends LifecycleComponent>> services() {
|
||||
Collection<Class<? extends LifecycleComponent>> services = new ArrayList<>();
|
||||
if (settings.getAsBoolean("cloud.enabled", true)) {
|
||||
// services.add(GceComputeServiceImpl.class);
|
||||
}
|
||||
return services;
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
# Licensed to ElasticSearch under one or more contributor
|
||||
# license agreements. See the NOTICE file distributed with this work for additional
|
||||
# information regarding copyright ownership. ElasticSearch licenses this file to you
|
||||
# under the Apache License, Version 2.0 (the "License"); you may not use this
|
||||
# file except in compliance with the License. You may obtain a copy of the
|
||||
# License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by
|
||||
# applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the specific language
|
||||
# governing permissions and limitations under the License.
|
||||
plugin=org.elasticsearch.plugin.cloud.gce.CloudGcePlugin
|
||||
version=${project.version}
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.cloud.gce;
|
||||
|
||||
import org.elasticsearch.common.Strings;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.env.Environment;
|
||||
import org.elasticsearch.env.FailedToResolveConfigException;
|
||||
import org.elasticsearch.plugins.PluginsService;
|
||||
import org.elasticsearch.test.ElasticsearchIntegrationTest;
|
||||
import org.elasticsearch.test.ElasticsearchIntegrationTest.ThirdParty;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
@ThirdParty
|
||||
public abstract class AbstractGceTest extends ElasticsearchIntegrationTest {
|
||||
|
||||
@Override
|
||||
protected Settings nodeSettings(int nodeOrdinal) {
|
||||
Settings.Builder settings = Settings.builder()
|
||||
.put(super.nodeSettings(nodeOrdinal))
|
||||
.put("path.home", createTempDir())
|
||||
.put("plugins." + PluginsService.LOAD_PLUGIN_FROM_CLASSPATH, true);
|
||||
|
||||
Environment environment = new Environment(settings.build());
|
||||
|
||||
// if explicit, just load it and don't load from env
|
||||
try {
|
||||
if (Strings.hasText(System.getProperty("tests.config"))) {
|
||||
settings.loadFromUrl(environment.resolveConfig(System.getProperty("tests.config")));
|
||||
} else {
|
||||
throw new IllegalStateException("to run integration tests, you need to set -Dtests.thirdparty=true and -Dtests.config=/path/to/elasticsearch.yml");
|
||||
}
|
||||
} catch (FailedToResolveConfigException exception) {
|
||||
throw new IllegalStateException("your test configuration file is incorrect: " + System.getProperty("tests.config"), exception);
|
||||
}
|
||||
return settings.build();
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -0,0 +1,223 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.discovery.gce;
|
||||
|
||||
import com.google.common.collect.Lists;
|
||||
import org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse;
|
||||
import org.elasticsearch.cloud.gce.GceComputeService;
|
||||
import org.elasticsearch.cloud.gce.GceComputeService.Fields;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.discovery.gce.mock.*;
|
||||
import org.elasticsearch.plugins.PluginsService;
|
||||
import org.elasticsearch.test.ElasticsearchIntegrationTest;
|
||||
import org.junit.Ignore;
|
||||
import org.junit.Test;
|
||||
|
||||
import java.io.IOException;
|
||||
|
||||
import static org.hamcrest.Matchers.notNullValue;
|
||||
|
||||
@ElasticsearchIntegrationTest.ClusterScope(
|
||||
scope = ElasticsearchIntegrationTest.Scope.TEST,
|
||||
numDataNodes = 0,
|
||||
numClientNodes = 0,
|
||||
transportClientRatio = 0.0)
|
||||
public class GceComputeEngineTest extends ElasticsearchIntegrationTest {
|
||||
|
||||
public static int getPort(int nodeOrdinal) {
|
||||
try {
|
||||
return PropertiesHelper.getAsInt("plugin.port")
|
||||
+ nodeOrdinal * 10;
|
||||
} catch (IOException e) {
|
||||
}
|
||||
|
||||
return -1;
|
||||
}
|
||||
|
||||
protected void checkNumberOfNodes(int expected) {
|
||||
NodesInfoResponse nodeInfos = client().admin().cluster().prepareNodesInfo().execute().actionGet();
|
||||
assertNotNull(nodeInfos);
|
||||
assertNotNull(nodeInfos.getNodes());
|
||||
assertEquals(expected, nodeInfos.getNodes().length);
|
||||
}
|
||||
|
||||
protected Settings settingsBuilder(int nodeOrdinal, Class<? extends GceComputeService> mock, Settings settings) {
|
||||
Settings.Builder builder = Settings.settingsBuilder()
|
||||
.put("discovery.type", "gce")
|
||||
.put("cloud.gce.api.impl", mock)
|
||||
// We need the network to make the mock working
|
||||
.put("node.mode", "network")
|
||||
// Make the tests run faster
|
||||
.put("discovery.zen.join.timeout", "100ms")
|
||||
.put("discovery.zen.ping.timeout", "10ms")
|
||||
.put("discovery.initial_state_timeout", "300ms")
|
||||
// We use a specific port for each node
|
||||
.put("transport.tcp.port", getPort(nodeOrdinal))
|
||||
// We disable http
|
||||
.put("http.enabled", false)
|
||||
// We force plugin loading
|
||||
.put("plugins." + PluginsService.LOAD_PLUGIN_FROM_CLASSPATH, true)
|
||||
.put(settings)
|
||||
.put(super.nodeSettings(nodeOrdinal));
|
||||
|
||||
return builder.build();
|
||||
}
|
||||
|
||||
protected void startNode(int nodeOrdinal, Class<? extends GceComputeService> mock, Settings settings) {
|
||||
logger.info("--> start node #{}, mock [{}], settings [{}]", nodeOrdinal, mock.getSimpleName(), settings.getAsMap());
|
||||
internalCluster().startNode(settingsBuilder(
|
||||
nodeOrdinal,
|
||||
mock,
|
||||
settings));
|
||||
assertThat(client().admin().cluster().prepareState().setMasterNodeTimeout("1s").execute().actionGet().getState().nodes().masterNodeId(), notNullValue());
|
||||
}
|
||||
|
||||
@Test @Ignore
|
||||
public void nodes_with_different_tags_and_no_tag_set() {
|
||||
startNode(1,
|
||||
GceComputeServiceTwoNodesDifferentTagsMock.class,
|
||||
Settings.EMPTY);
|
||||
startNode(2,
|
||||
GceComputeServiceTwoNodesDifferentTagsMock.class,
|
||||
Settings.EMPTY);
|
||||
|
||||
// We expect having 2 nodes as part of the cluster, let's test that
|
||||
checkNumberOfNodes(2);
|
||||
}
|
||||
|
||||
/**
|
||||
* We need to ignore this test from elasticsearch version 1.2.1 as
|
||||
* expected nodes running is 2 and this test will create 2 clusters with one node each.
|
||||
* @see org.elasticsearch.test.ElasticsearchIntegrationTest#ensureClusterSizeConsistency()
|
||||
* TODO Reactivate when it will be possible to set the number of running nodes
|
||||
*/
|
||||
@Test @Ignore
|
||||
public void nodes_with_different_tags_and_one_tag_set() {
|
||||
startNode(1,
|
||||
GceComputeServiceTwoNodesDifferentTagsMock.class,
|
||||
Settings.settingsBuilder().put(Fields.TAGS, "elasticsearch").build());
|
||||
startNode(2,
|
||||
GceComputeServiceTwoNodesDifferentTagsMock.class,
|
||||
Settings.settingsBuilder().put(Fields.TAGS, "elasticsearch").build());
|
||||
|
||||
// We expect having 1 nodes as part of the cluster, let's test that
|
||||
checkNumberOfNodes(1);
|
||||
}
|
||||
|
||||
/**
|
||||
* We need to ignore this test from elasticsearch version 1.2.1 as
|
||||
* expected nodes running is 2 and this test will create 2 clusters with one node each.
|
||||
* @see org.elasticsearch.test.ElasticsearchIntegrationTest#ensureClusterSizeConsistency()
|
||||
* TODO Reactivate when it will be possible to set the number of running nodes
|
||||
*/
|
||||
@Test @Ignore
|
||||
public void nodes_with_different_tags_and_two_tag_set() {
|
||||
startNode(1,
|
||||
GceComputeServiceTwoNodesDifferentTagsMock.class,
|
||||
Settings.settingsBuilder().put(Fields.TAGS, Lists.newArrayList("elasticsearch", "dev")).build());
|
||||
startNode(2,
|
||||
GceComputeServiceTwoNodesDifferentTagsMock.class,
|
||||
Settings.settingsBuilder().put(Fields.TAGS, Lists.newArrayList("elasticsearch", "dev")).build());
|
||||
|
||||
// We expect having 1 nodes as part of the cluster, let's test that
|
||||
checkNumberOfNodes(1);
|
||||
}
|
||||
|
||||
@Test @Ignore
|
||||
public void nodes_with_same_tags_and_no_tag_set() {
|
||||
startNode(1,
|
||||
GceComputeServiceTwoNodesSameTagsMock.class,
|
||||
Settings.EMPTY);
|
||||
startNode(2,
|
||||
GceComputeServiceTwoNodesSameTagsMock.class,
|
||||
Settings.EMPTY);
|
||||
|
||||
// We expect having 2 nodes as part of the cluster, let's test that
|
||||
checkNumberOfNodes(2);
|
||||
}
|
||||
|
||||
@Test @Ignore
|
||||
public void nodes_with_same_tags_and_one_tag_set() {
|
||||
startNode(1,
|
||||
GceComputeServiceTwoNodesSameTagsMock.class,
|
||||
Settings.settingsBuilder().put(Fields.TAGS, "elasticsearch").build());
|
||||
startNode(2,
|
||||
GceComputeServiceTwoNodesSameTagsMock.class,
|
||||
Settings.settingsBuilder().put(Fields.TAGS, "elasticsearch").build());
|
||||
|
||||
// We expect having 2 nodes as part of the cluster, let's test that
|
||||
checkNumberOfNodes(2);
|
||||
}
|
||||
|
||||
@Test @Ignore
|
||||
public void nodes_with_same_tags_and_two_tags_set() {
|
||||
startNode(1,
|
||||
GceComputeServiceTwoNodesSameTagsMock.class,
|
||||
Settings.settingsBuilder().put(Fields.TAGS, Lists.newArrayList("elasticsearch", "dev")).build());
|
||||
startNode(2,
|
||||
GceComputeServiceTwoNodesSameTagsMock.class,
|
||||
Settings.settingsBuilder().put(Fields.TAGS, Lists.newArrayList("elasticsearch", "dev")).build());
|
||||
|
||||
// We expect having 2 nodes as part of the cluster, let's test that
|
||||
checkNumberOfNodes(2);
|
||||
}
|
||||
|
||||
@Test @Ignore
|
||||
public void multiple_zones_and_two_nodes_in_same_zone() {
|
||||
startNode(1,
|
||||
GceComputeServiceTwoNodesOneZoneMock.class,
|
||||
Settings.settingsBuilder().put(Fields.ZONE, Lists.newArrayList("us-central1-a", "us-central1-b",
|
||||
"us-central1-f", "europe-west1-a", "europe-west1-b")).build());
|
||||
startNode(2,
|
||||
GceComputeServiceTwoNodesOneZoneMock.class,
|
||||
Settings.settingsBuilder().put(Fields.ZONE, Lists.newArrayList("us-central1-a", "us-central1-b",
|
||||
"us-central1-f", "europe-west1-a", "europe-west1-b")).build());
|
||||
|
||||
// We expect having 2 nodes as part of the cluster, let's test that
|
||||
checkNumberOfNodes(2);
|
||||
}
|
||||
|
||||
@Test @Ignore
|
||||
public void multiple_zones_and_two_nodes_in_different_zones() {
|
||||
startNode(1,
|
||||
GceComputeServiceTwoNodesTwoZonesMock.class,
|
||||
Settings.settingsBuilder().put(Fields.ZONE, Lists.newArrayList("us-central1-a", "us-central1-b",
|
||||
"us-central1-f", "europe-west1-a", "europe-west1-b")).build());
|
||||
startNode(2,
|
||||
GceComputeServiceTwoNodesTwoZonesMock.class,
|
||||
Settings.settingsBuilder().put(Fields.ZONE, Lists.newArrayList("us-central1-a", "us-central1-b",
|
||||
"us-central1-f", "europe-west1-a", "europe-west1-b")).build());
|
||||
|
||||
// We expect having 2 nodes as part of the cluster, let's test that
|
||||
checkNumberOfNodes(2);
|
||||
}
|
||||
|
||||
/**
|
||||
* For issue https://github.com/elastic/elasticsearch-cloud-gce/issues/43
|
||||
*/
|
||||
@Test @Ignore
|
||||
public void zero_node_43() {
|
||||
startNode(1,
|
||||
GceComputeServiceZeroNodeMock.class,
|
||||
Settings.settingsBuilder().put(Fields.ZONE, Lists.newArrayList("us-central1-a", "us-central1-b",
|
||||
"us-central1-f", "europe-west1-a", "europe-west1-b")).build());
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.discovery.gce;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.Properties;
|
||||
|
||||
/**
|
||||
* Get information from plugin-test.properties file
|
||||
*/
|
||||
public class PropertiesHelper {
|
||||
public static int getAsInt(String name) throws IOException {
|
||||
final Properties properties = new Properties();
|
||||
properties.load(PropertiesHelper.class.getResourceAsStream("/plugin-test.properties"));
|
||||
return Integer.parseInt(properties.getProperty(name));
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,119 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.discovery.gce.mock;
|
||||
|
||||
import com.google.api.services.compute.model.Instance;
|
||||
import com.google.api.services.compute.model.Metadata;
|
||||
import com.google.api.services.compute.model.NetworkInterface;
|
||||
import com.google.api.services.compute.model.Tags;
|
||||
import com.google.common.collect.Lists;
|
||||
import org.elasticsearch.ElasticsearchException;
|
||||
import org.elasticsearch.cloud.gce.GceComputeService;
|
||||
import org.elasticsearch.common.component.AbstractLifecycleComponent;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.discovery.gce.GceComputeEngineTest;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.List;
|
||||
|
||||
import static com.carrotsearch.randomizedtesting.RandomizedTest.randomInt;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
public abstract class GceComputeServiceAbstractMock extends AbstractLifecycleComponent<GceComputeServiceAbstractMock>
|
||||
implements GceComputeService {
|
||||
|
||||
protected abstract List<List<String>> getTags();
|
||||
protected abstract List<String> getZones();
|
||||
private final List<String> zoneList;
|
||||
|
||||
protected GceComputeServiceAbstractMock(Settings settings) {
|
||||
super(settings);
|
||||
|
||||
int numNodes = getTags().size() > getZones().size() ? getTags().size() : getZones().size();
|
||||
|
||||
logger.debug("starting GCE Api Mock with {} nodes:", numNodes);
|
||||
for (int i = 0; i < numNodes; i++) {
|
||||
List<String> tags = getTags().size() > i ? getTags().get(i) : null;
|
||||
String zone = getZones().size() > i ? getZones().get(i) : null;
|
||||
logger.debug(" - node #{}: tags [{}], zone [{}]", i, tags, zone);
|
||||
}
|
||||
|
||||
String[] zoneList = settings.getAsArray(Fields.ZONE);
|
||||
this.zoneList = Lists.newArrayList(zoneList);
|
||||
}
|
||||
|
||||
private Collection<Instance> instances = null;
|
||||
|
||||
private void computeInstances() {
|
||||
instances = new ArrayList<Instance>();
|
||||
int nodeNumber = 0;
|
||||
// For each instance (item of tags)
|
||||
for (List<String> tags : getTags()) {
|
||||
String zone = zoneList.isEmpty() ? "dummy" : zoneList.get(randomInt(zoneList.size()-1));
|
||||
logger.info(" ----> GCE Mock API: Adding node [{}] in zone [{}]", nodeNumber, zone);
|
||||
Instance instance = new Instance();
|
||||
instance.setName("Mock Node " + tags);
|
||||
instance.setMachineType("Mock Type machine");
|
||||
instance.setStatus("STARTED");
|
||||
instance.setZone(zone);
|
||||
Tags instanceTags = new Tags();
|
||||
instanceTags.setItems(tags);
|
||||
instance.setTags(instanceTags);
|
||||
NetworkInterface networkInterface = new NetworkInterface();
|
||||
networkInterface.setNetworkIP("localhost");
|
||||
List<NetworkInterface> networkInterfaces = new ArrayList<NetworkInterface>();
|
||||
networkInterfaces.add(networkInterface);
|
||||
instance.setNetworkInterfaces(networkInterfaces);
|
||||
|
||||
// Add metadata es_port:930X where X is the instance number
|
||||
Metadata metadata = new Metadata();
|
||||
metadata.put("es_port", "" + GceComputeEngineTest.getPort(nodeNumber));
|
||||
instance.setMetadata(metadata);
|
||||
|
||||
instances.add(instance);
|
||||
|
||||
nodeNumber++;
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public Collection<Instance> instances() {
|
||||
if (instances == null || instances.size() == 0) {
|
||||
computeInstances();
|
||||
}
|
||||
|
||||
return instances;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void doStart() throws ElasticsearchException {
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void doStop() throws ElasticsearchException {
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void doClose() throws ElasticsearchException {
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.discovery.gce.mock;
|
||||
|
||||
import org.elasticsearch.common.inject.Inject;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
public class GceComputeServiceTwoNodesDifferentTagsMock extends GceComputeServiceAbstractMock {
|
||||
private static List<List<String>> tags = Arrays.asList(
|
||||
Arrays.asList("dev"),
|
||||
Arrays.asList("elasticsearch", "dev"));
|
||||
|
||||
@Override
|
||||
protected List<List<String>> getTags() {
|
||||
return tags;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected List<String> getZones() {
|
||||
return new ArrayList();
|
||||
}
|
||||
|
||||
@Inject
|
||||
protected GceComputeServiceTwoNodesDifferentTagsMock(Settings settings) {
|
||||
super(settings);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.discovery.gce.mock;
|
||||
|
||||
import org.elasticsearch.common.inject.Inject;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
public class GceComputeServiceTwoNodesOneZoneMock extends GceComputeServiceAbstractMock {
|
||||
private static List<String> zones = Arrays.asList("us-central1-a", "us-central1-a");
|
||||
|
||||
@Override
|
||||
protected List<List<String>> getTags() {
|
||||
return new ArrayList<>();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected List<String> getZones() {
|
||||
return zones;
|
||||
}
|
||||
|
||||
@Inject
|
||||
protected GceComputeServiceTwoNodesOneZoneMock(Settings settings) {
|
||||
super(settings);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.discovery.gce.mock;
|
||||
|
||||
import org.elasticsearch.common.inject.Inject;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
public class GceComputeServiceTwoNodesSameTagsMock extends GceComputeServiceAbstractMock {
|
||||
private static List<List<String>> tags = Arrays.asList(
|
||||
Arrays.asList("elasticsearch", "dev"),
|
||||
Arrays.asList("elasticsearch", "dev"));
|
||||
|
||||
|
||||
@Override
|
||||
protected List<List<String>> getTags() {
|
||||
return tags;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected List<String> getZones() {
|
||||
return new ArrayList<>();
|
||||
}
|
||||
|
||||
@Inject
|
||||
protected GceComputeServiceTwoNodesSameTagsMock(Settings settings) {
|
||||
super(settings);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.discovery.gce.mock;
|
||||
|
||||
import org.elasticsearch.common.inject.Inject;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
public class GceComputeServiceTwoNodesTwoZonesMock extends GceComputeServiceAbstractMock {
|
||||
private static List<String> zones = Arrays.asList("us-central1-a", "europe-west1-a");
|
||||
|
||||
@Override
|
||||
protected List<List<String>> getTags() {
|
||||
return new ArrayList();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected List<String> getZones() {
|
||||
return zones;
|
||||
}
|
||||
|
||||
@Inject
|
||||
protected GceComputeServiceTwoNodesTwoZonesMock(Settings settings) {
|
||||
super(settings);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,78 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.discovery.gce.mock;
|
||||
|
||||
import com.google.api.services.compute.model.Instance;
|
||||
import com.google.common.base.Function;
|
||||
import com.google.common.collect.Iterables;
|
||||
import com.google.common.collect.Lists;
|
||||
import org.elasticsearch.common.inject.Inject;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
public class GceComputeServiceZeroNodeMock extends GceComputeServiceAbstractMock {
|
||||
|
||||
@Override
|
||||
protected List<List<String>> getTags() {
|
||||
return new ArrayList();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected List<String> getZones() {
|
||||
return new ArrayList();
|
||||
}
|
||||
|
||||
private final List<String> zoneList;
|
||||
|
||||
@Override
|
||||
public Collection<Instance> instances() {
|
||||
logger.debug("get instances for zoneList [{}]", zoneList);
|
||||
|
||||
List<List<Instance>> instanceListByZone = Lists.transform(zoneList, new Function<String, List<Instance>>() {
|
||||
@Override
|
||||
public List<Instance> apply(String zoneId) {
|
||||
// If we return null here we will get a trace as explained in issue 43
|
||||
return new ArrayList();
|
||||
}
|
||||
});
|
||||
|
||||
//Collapse instances from all zones into one neat list
|
||||
List<Instance> instanceList = Lists.newArrayList(Iterables.concat(instanceListByZone));
|
||||
|
||||
if (instanceList.size() == 0) {
|
||||
logger.warn("disabling GCE discovery. Can not get list of nodes");
|
||||
}
|
||||
|
||||
return instanceList;
|
||||
}
|
||||
|
||||
@Inject
|
||||
protected GceComputeServiceZeroNodeMock(Settings settings) {
|
||||
super(settings);
|
||||
String[] zoneList = settings.getAsArray(Fields.ZONE);
|
||||
this.zoneList = Lists.newArrayList(zoneList);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.gce.itest;
|
||||
|
||||
import org.elasticsearch.action.admin.cluster.state.ClusterStateResponse;
|
||||
import org.elasticsearch.cloud.gce.AbstractGceTest;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.plugins.PluginsService;
|
||||
import org.elasticsearch.test.ElasticsearchIntegrationTest;
|
||||
import org.hamcrest.Matchers;
|
||||
import org.junit.Test;
|
||||
|
||||
/**
|
||||
* This test needs GCE to run and -Dtests.gce=true to be set
|
||||
* and -Dtests.config=/path/to/elasticsearch.yml
|
||||
* TODO: By now, it will only work from GCE platform as we don't support yet external auth.
|
||||
* See https://github.com/elasticsearch/elasticsearch-cloud-gce/issues/10
|
||||
* @see org.elasticsearch.cloud.gce.AbstractGceTest
|
||||
*/
|
||||
@ElasticsearchIntegrationTest.ClusterScope(
|
||||
scope = ElasticsearchIntegrationTest.Scope.SUITE,
|
||||
numDataNodes = 1,
|
||||
transportClientRatio = 0.0)
|
||||
public class GceSimpleITest extends AbstractGceTest {
|
||||
|
||||
@Override
|
||||
protected Settings nodeSettings(int nodeOrdinal) {
|
||||
return Settings.builder()
|
||||
.put(super.nodeSettings(nodeOrdinal))
|
||||
.put("plugins." + PluginsService.LOAD_PLUGIN_FROM_CLASSPATH, true)
|
||||
.build();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void one_node_should_run() {
|
||||
// Do nothing... Just start :-)
|
||||
// but let's check that we have at least 1 node (local node)
|
||||
ClusterStateResponse clusterState = client().admin().cluster().prepareState().execute().actionGet();
|
||||
|
||||
assertThat(clusterState.getState().getNodes().getSize(), Matchers.greaterThanOrEqualTo(1));
|
||||
}
|
||||
|
||||
@Override
|
||||
public Settings indexSettings() {
|
||||
// During restore we frequently restore index to exactly the same state it was before, that might cause the same
|
||||
// checksum file to be written twice during restore operation
|
||||
return Settings.builder().put(super.indexSettings())
|
||||
.build();
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
# This file contains specific needed properties for tests.
|
||||
# You can use placeholders to extract values from Maven for example
|
||||
plugin.port=${es.plugin.port}
|
||||
Loading…
Reference in New Issue