HBASE-18418 Remove apache_hbase_topology from dev-support
This commit is contained in:
parent
c16eb7881f
commit
3acb081787
|
@ -1,24 +0,0 @@
|
||||||
# Licensed to the Apache Software Foundation (ASF) under one
|
|
||||||
# or more contributor license agreements. See the NOTICE file
|
|
||||||
# distributed with this work for additional information
|
|
||||||
# regarding copyright ownership. The ASF licenses this file
|
|
||||||
# to you under the Apache License, Version 2.0 (the
|
|
||||||
# "License"); you may not use this file except in compliance
|
|
||||||
# with the License. You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
FROM debian:wheezy
|
|
||||||
|
|
||||||
ENV TOPOLOGY_NAME=apache_hbase
|
|
||||||
ADD . /root/clusterdock/clusterdock/topologies/${TOPOLOGY_NAME}
|
|
||||||
|
|
||||||
RUN find /root -type f -name id_rsa -exec chmod 600 {} \;
|
|
||||||
|
|
||||||
VOLUME /root/clusterdock/clusterdock/topologies/${TOPOLOGY_NAME}
|
|
||||||
CMD ["/true"]
|
|
|
@ -1,49 +0,0 @@
|
||||||
<!---
|
|
||||||
Licensed to the Apache Software Foundation (ASF) under one
|
|
||||||
or more contributor license agreements. See the NOTICE file
|
|
||||||
distributed with this work for additional information
|
|
||||||
regarding copyright ownership. The ASF licenses this file
|
|
||||||
to you under the Apache License, Version 2.0 (the
|
|
||||||
"License"); you may not use this file except in compliance
|
|
||||||
with the License. You may obtain a copy of the License at
|
|
||||||
|
|
||||||
http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
|
|
||||||
Unless required by applicable law or agreed to in writing,
|
|
||||||
software distributed under the License is distributed on an
|
|
||||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
||||||
KIND, either express or implied. See the License for the
|
|
||||||
specific language governing permissions and limitations
|
|
||||||
under the License.
|
|
||||||
-->
|
|
||||||
# apache_hbase clusterdock topology
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
*clusterdock* is a framework for creating Docker-based container clusters. Unlike regular Docker
|
|
||||||
containers, which tend to run single processes and then exit once the process terminates, these
|
|
||||||
container clusters are characterized by the execution of an init process in daemon mode. As such,
|
|
||||||
the containers act more like "fat containers" or "light VMs;" entities with accessible IP addresses
|
|
||||||
which emulate standalone hosts.
|
|
||||||
|
|
||||||
*clusterdock* relies upon the notion of a topology to define how clusters should be built into
|
|
||||||
images and then what to do with those images to start Docker container clusters.
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
The *clusterdock* framework is designed to be run out of its own container while affecting
|
|
||||||
operations on the host. To avoid problems that might result from incorrectly
|
|
||||||
formatting this framework invocation, a Bash helper script (`clusterdock.sh`) can be sourced on a
|
|
||||||
host that has Docker installed. Afterwards, running any of the binaries intended to carry
|
|
||||||
out *clusterdock* actions can be done using the `clusterdock_run` command.
|
|
||||||
```
|
|
||||||
wget https://raw.githubusercontent.com/cloudera/clusterdock/master/clusterdock.sh
|
|
||||||
# ALWAYS INSPECT SCRIPTS FROM THE INTERNET BEFORE SOURCING THEM.
|
|
||||||
source clusterdock.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
Since the *clusterdock* framework itself lives outside of Apache HBase, an environmental variable
|
|
||||||
is used to let the helper script know where to find an image of the *apache_hbase* topology. To
|
|
||||||
start a four-node Apache HBase cluster with default versions, you would simply run
|
|
||||||
```
|
|
||||||
CLUSTERDOCK_TOPOLOGY_IMAGE=apache_hbase_topology_location clusterdock_run \
|
|
||||||
./bin/start_cluster apache_hbase --secondary-nodes='node-{2..4}'
|
|
||||||
```
|
|
|
@ -1,15 +0,0 @@
|
||||||
# Licensed to the Apache Software Foundation (ASF) under one
|
|
||||||
# or more contributor license agreements. See the NOTICE file
|
|
||||||
# distributed with this work for additional information
|
|
||||||
# regarding copyright ownership. The ASF licenses this file
|
|
||||||
# to you under the Apache License, Version 2.0 (the
|
|
||||||
# "License"); you may not use this file except in compliance
|
|
||||||
# with the License. You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
|
@ -1,421 +0,0 @@
|
||||||
#!/usr/bin/env python
|
|
||||||
# Licensed to the Apache Software Foundation (ASF) under one
|
|
||||||
# or more contributor license agreements. See the NOTICE file
|
|
||||||
# distributed with this work for additional information
|
|
||||||
# regarding copyright ownership. The ASF licenses this file
|
|
||||||
# to you under the Apache License, Version 2.0 (the
|
|
||||||
# "License"); you may not use this file except in compliance
|
|
||||||
# with the License. You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
"""The actions module for the apache_hbase topology. The behavior to be carried out by the
|
|
||||||
build_cluster and start_cluster clusterdock scripts are to be defined through the build and
|
|
||||||
start functions, respectively.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import logging
|
|
||||||
import tarfile
|
|
||||||
from ConfigParser import ConfigParser
|
|
||||||
from os import EX_OK, listdir, makedirs, remove # pylint: disable=ungrouped-imports
|
|
||||||
# Follow convention of grouping from module imports
|
|
||||||
# after normal imports.
|
|
||||||
from os.path import exists, join
|
|
||||||
from shutil import move
|
|
||||||
from socket import getfqdn
|
|
||||||
from sys import stdout
|
|
||||||
from uuid import uuid4
|
|
||||||
|
|
||||||
# pylint: disable=import-error
|
|
||||||
# clusterdock topologies get access to the clusterdock package at run time, but their reference
|
|
||||||
# to clusterdock modules will confuse pylint, so we have to disable it.
|
|
||||||
|
|
||||||
import requests
|
|
||||||
from docker import Client
|
|
||||||
|
|
||||||
from clusterdock import Constants
|
|
||||||
from clusterdock.cluster import Cluster, Node, NodeGroup
|
|
||||||
from clusterdock.docker_utils import (build_image, get_clusterdock_container_id,
|
|
||||||
get_host_port_binding, is_image_available_locally, pull_image)
|
|
||||||
from clusterdock.utils import strip_components_from_tar, XmlConfiguration
|
|
||||||
|
|
||||||
# We disable a couple of Pylint conventions because it assumes that module level variables must be
|
|
||||||
# named as if they're constants (which isn't the case here).
|
|
||||||
logger = logging.getLogger(__name__) # pylint: disable=invalid-name
|
|
||||||
logger.setLevel(logging.INFO)
|
|
||||||
|
|
||||||
client = Client() # pylint: disable=invalid-name
|
|
||||||
|
|
||||||
DEFAULT_APACHE_NAMESPACE = Constants.DEFAULT.apache_namespace # pylint: disable=no-member
|
|
||||||
|
|
||||||
def _copy_container_folder_to_host(container_id, source_folder, destination_folder,
|
|
||||||
host_folder=None):
|
|
||||||
if not exists(destination_folder):
|
|
||||||
makedirs(destination_folder)
|
|
||||||
stream, _ = client.get_archive(container_id, source_folder)
|
|
||||||
tar_filename = join(destination_folder, 'container_folder.tar')
|
|
||||||
with open(tar_filename, 'wb') as file_descriptor:
|
|
||||||
file_descriptor.write(stream.read())
|
|
||||||
tar = tarfile.open(name=tar_filename)
|
|
||||||
tar.extractall(path=destination_folder, members=strip_components_from_tar(tar))
|
|
||||||
tar.close()
|
|
||||||
remove(tar_filename)
|
|
||||||
logger.info("Extracted container folder %s to %s.", source_folder,
|
|
||||||
host_folder if host_folder else destination_folder)
|
|
||||||
|
|
||||||
def _create_configs_from_file(filename, cluster_config_dir, wildcards):
|
|
||||||
configurations = ConfigParser(allow_no_value=True)
|
|
||||||
configurations.read(filename)
|
|
||||||
|
|
||||||
for config_file in configurations.sections():
|
|
||||||
logger.info("Updating %s...", config_file)
|
|
||||||
# For XML configuration files, run things through XmlConfiguration.
|
|
||||||
if config_file.endswith('.xml'):
|
|
||||||
XmlConfiguration(
|
|
||||||
{item[0]: item[1].format(**wildcards)
|
|
||||||
for item in configurations.items(config_file)}
|
|
||||||
).write_to_file(join(cluster_config_dir, config_file))
|
|
||||||
# For everything else, recognize whether a line in the configuration should simply be
|
|
||||||
# appended to the bottom of a file or processed in some way. The presence of +++ will
|
|
||||||
# lead to the evaluation of the following string through the end of the line.
|
|
||||||
else:
|
|
||||||
lines = []
|
|
||||||
for item in configurations.items(config_file):
|
|
||||||
if item[0].startswith('+++'):
|
|
||||||
command = item[0].lstrip('+ ').format(**wildcards)
|
|
||||||
|
|
||||||
# Yes, we use eval here. This is potentially dangerous, but intention.
|
|
||||||
lines.append(str(eval(command))) # pylint: disable=eval-used
|
|
||||||
elif item[0] == "body":
|
|
||||||
lines.append(item[1].format(**wildcards))
|
|
||||||
else:
|
|
||||||
lines.append(item[0].format(**wildcards))
|
|
||||||
with open(join(cluster_config_dir, config_file), 'w') as conf:
|
|
||||||
conf.write("".join(["{0}\n".format(line) for line in lines]))
|
|
||||||
|
|
||||||
# Keep track of some common web UI ports that we'll expose to users later (e.g. to allow a user
|
|
||||||
# to reach the HDFS NameNode web UI over the internet).
|
|
||||||
HBASE_REST_SERVER_PORT = 8080
|
|
||||||
NAMENODE_WEB_UI_PORT = 50070
|
|
||||||
RESOURCEMANAGER_WEB_UI_PORT = 8088
|
|
||||||
|
|
||||||
# When starting or building cluster, CLUSTERDOCK_VOLUME will be the root directory for persistent
|
|
||||||
# files (note that this location will itself be in a Docker container's filesystem).
|
|
||||||
CLUSTERDOCK_VOLUME = '/tmp/clusterdock'
|
|
||||||
|
|
||||||
def start(args):
|
|
||||||
"""This function will be executed when ./bin/start_cluster apache_hbase is invoked."""
|
|
||||||
|
|
||||||
# pylint: disable=too-many-locals
|
|
||||||
# Pylint doesn't want more than 15 local variables in a function; this one has 17. This is about
|
|
||||||
# as low as I want to go because, while I can cheat and stuff unrelated things in a dictionary,
|
|
||||||
# that won't improve readability.
|
|
||||||
|
|
||||||
uuid = str(uuid4())
|
|
||||||
container_cluster_config_dir = join(CLUSTERDOCK_VOLUME, uuid, 'config')
|
|
||||||
makedirs(container_cluster_config_dir)
|
|
||||||
|
|
||||||
for mount in client.inspect_container(get_clusterdock_container_id())['Mounts']:
|
|
||||||
if mount['Destination'] == CLUSTERDOCK_VOLUME:
|
|
||||||
host_cluster_config_dir = join(mount['Source'], uuid, 'config')
|
|
||||||
break
|
|
||||||
else:
|
|
||||||
raise Exception("Could not find source of {0} mount.".format(CLUSTERDOCK_VOLUME))
|
|
||||||
|
|
||||||
# CLUSTERDOCK_VOLUME/uuid/config in the clusterdock container corresponds to
|
|
||||||
# host_cluster_config_dir on the Docker host.
|
|
||||||
logger.debug("Creating directory for cluster configuration files in %s...",
|
|
||||||
host_cluster_config_dir)
|
|
||||||
|
|
||||||
# Generate the image name to use from the command line arguments passed in.
|
|
||||||
image = '/'.join(
|
|
||||||
[item
|
|
||||||
for item in [args.registry_url, args.namespace or DEFAULT_APACHE_NAMESPACE,
|
|
||||||
"clusterdock:{os}_java-{java}_hadoop-{hadoop}_hbase-{hbase}".format(
|
|
||||||
os=args.operating_system, java=args.java_version,
|
|
||||||
hadoop=args.hadoop_version, hbase=args.hbase_version
|
|
||||||
)]
|
|
||||||
if item]
|
|
||||||
)
|
|
||||||
if args.always_pull or not is_image_available_locally(image):
|
|
||||||
pull_image(image)
|
|
||||||
|
|
||||||
# Before starting the cluster, we create a throwaway container from which we copy
|
|
||||||
# configuration files back to the host. We also use this container to run an HBase
|
|
||||||
# command that returns the port of the HBase master web UI. Since we aren't running init here,
|
|
||||||
# we also have to manually pass in JAVA_HOME as an environmental variable.
|
|
||||||
get_hbase_web_ui_port_command = ('/hbase/bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool '
|
|
||||||
'hbase.master.info.port')
|
|
||||||
container_id = client.create_container(image=image, command=get_hbase_web_ui_port_command,
|
|
||||||
environment={'JAVA_HOME': '/java'})['Id']
|
|
||||||
logger.debug("Created temporary container (id: %s) from which to copy configuration files.",
|
|
||||||
container_id)
|
|
||||||
|
|
||||||
# Actually do the copying of Hadoop configs...
|
|
||||||
_copy_container_folder_to_host(container_id, '/hadoop/etc/hadoop',
|
|
||||||
join(container_cluster_config_dir, 'hadoop'),
|
|
||||||
join(host_cluster_config_dir, 'hadoop'))
|
|
||||||
|
|
||||||
# ... and repeat for HBase configs.
|
|
||||||
_copy_container_folder_to_host(container_id, '/hbase/conf',
|
|
||||||
join(container_cluster_config_dir, 'hbase'),
|
|
||||||
join(host_cluster_config_dir, 'hbase'))
|
|
||||||
|
|
||||||
logger.info("The /hbase/lib folder on containers in the cluster will be volume mounted "
|
|
||||||
"into %s...", join(host_cluster_config_dir, 'hbase-lib'))
|
|
||||||
_copy_container_folder_to_host(container_id, '/hbase/lib',
|
|
||||||
join(container_cluster_config_dir, 'hbase-lib'),
|
|
||||||
join(host_cluster_config_dir, 'hbase-lib'))
|
|
||||||
|
|
||||||
# Every node in the cluster will have a shared volume mount from the host for Hadoop and HBase
|
|
||||||
# configuration files as well as the HBase lib folder.
|
|
||||||
shared_volumes = [{join(host_cluster_config_dir, 'hadoop'): '/hadoop/etc/hadoop'},
|
|
||||||
{join(host_cluster_config_dir, 'hbase'): '/hbase/conf'},
|
|
||||||
{join(host_cluster_config_dir, 'hbase-lib'): '/hbase/lib'}]
|
|
||||||
|
|
||||||
# Get the HBase master web UI port, stripping the newline the Docker REST API gives us.
|
|
||||||
client.start(container=container_id)
|
|
||||||
if client.wait(container=container_id) == EX_OK:
|
|
||||||
hbase_master_web_ui_port = client.logs(container=container_id).rstrip()
|
|
||||||
client.remove_container(container=container_id, force=True)
|
|
||||||
else:
|
|
||||||
raise Exception('Failed to remove HBase configuration container.')
|
|
||||||
|
|
||||||
# Create the Node objects. These hold the state of our container nodes and will be started
|
|
||||||
# at Cluster instantiation time.
|
|
||||||
primary_node = Node(hostname=args.primary_node[0], network=args.network,
|
|
||||||
image=image, ports=[NAMENODE_WEB_UI_PORT,
|
|
||||||
hbase_master_web_ui_port,
|
|
||||||
RESOURCEMANAGER_WEB_UI_PORT,
|
|
||||||
HBASE_REST_SERVER_PORT],
|
|
||||||
volumes=shared_volumes)
|
|
||||||
secondary_nodes = []
|
|
||||||
for hostname in args.secondary_nodes:
|
|
||||||
# A list of service directories will be used to name folders on the host and, appended
|
|
||||||
# with an index, in the container, as well (e.g. /data1/node-1/dfs:/dfs1).
|
|
||||||
service_directories = ['dfs', 'yarn']
|
|
||||||
|
|
||||||
# Every Node will have shared_volumes to let one set of configs on the host be propagated
|
|
||||||
# to every container. If --data-directories is specified, this will be appended to allow
|
|
||||||
# containers to use multiple disks on the host.
|
|
||||||
volumes = shared_volumes[:]
|
|
||||||
if args.data_directories:
|
|
||||||
data_directories = args.data_directories.split(',')
|
|
||||||
volumes += [{join(data_directory, uuid, hostname, service_directory):
|
|
||||||
"/{0}{1}".format(service_directory, i)}
|
|
||||||
for i, data_directory in enumerate(data_directories, start=1)
|
|
||||||
for service_directory in service_directories]
|
|
||||||
secondary_nodes.append(Node(hostname=hostname,
|
|
||||||
network=args.network,
|
|
||||||
image=image,
|
|
||||||
volumes=volumes))
|
|
||||||
|
|
||||||
Cluster(topology='apache_hbase',
|
|
||||||
node_groups=[NodeGroup(name='primary', nodes=[primary_node]),
|
|
||||||
NodeGroup(name='secondary', nodes=secondary_nodes)],
|
|
||||||
network_name=args.network).start()
|
|
||||||
|
|
||||||
# When creating configs, pass in a dictionary of wildcards into create_configurations_from_file
|
|
||||||
# to transform placeholders in the configurations.cfg file into real values.
|
|
||||||
_create_configs_from_file(filename=args.configurations,
|
|
||||||
cluster_config_dir=container_cluster_config_dir,
|
|
||||||
wildcards={"primary_node": args.primary_node,
|
|
||||||
"secondary_nodes": args.secondary_nodes,
|
|
||||||
"all_nodes": args.primary_node + args.secondary_nodes,
|
|
||||||
"network": args.network})
|
|
||||||
|
|
||||||
# After creating configurations from the configurations.cfg file, update hdfs-site.xml and
|
|
||||||
# yarn-site.xml to use the data directories passed on the command line.
|
|
||||||
if args.data_directories:
|
|
||||||
_update_config_for_data_dirs(
|
|
||||||
container_cluster_config_dir=container_cluster_config_dir,
|
|
||||||
data_directories=data_directories
|
|
||||||
)
|
|
||||||
|
|
||||||
if not args.dont_start_services:
|
|
||||||
_start_services(primary_node, hbase_master_web_ui_port=hbase_master_web_ui_port)
|
|
||||||
|
|
||||||
def _update_config_for_data_dirs(container_cluster_config_dir, data_directories):
|
|
||||||
logger.info('Updating dfs.datanode.data.dir in hdfs-site.xml...')
|
|
||||||
hdfs_site_xml_filename = join(container_cluster_config_dir, 'hadoop', 'hdfs-site.xml')
|
|
||||||
hdfs_site_xml = XmlConfiguration(
|
|
||||||
properties={'dfs.datanode.data.dir':
|
|
||||||
','.join(["/dfs{0}".format(i)
|
|
||||||
for i, _ in enumerate(data_directories, start=1)])},
|
|
||||||
source_file=hdfs_site_xml_filename
|
|
||||||
)
|
|
||||||
hdfs_site_xml.write_to_file(filename=hdfs_site_xml_filename)
|
|
||||||
|
|
||||||
logger.info('Updating yarn.nodemanager.local-dirs in yarn-site.xml...')
|
|
||||||
yarn_site_xml_filename = join(container_cluster_config_dir, 'hadoop', 'yarn-site.xml')
|
|
||||||
yarn_site_xml = XmlConfiguration(
|
|
||||||
properties={'yarn.nodemanager.local-dirs':
|
|
||||||
','.join(["/yarn{0}".format(i)
|
|
||||||
for i, _ in enumerate(data_directories, start=1)])},
|
|
||||||
source_file=yarn_site_xml_filename
|
|
||||||
)
|
|
||||||
yarn_site_xml.write_to_file(filename=yarn_site_xml_filename)
|
|
||||||
|
|
||||||
def _start_services(primary_node, **kwargs):
|
|
||||||
logger.info("Formatting namenode on %s...", primary_node.fqdn)
|
|
||||||
primary_node.ssh('hdfs namenode -format')
|
|
||||||
|
|
||||||
logger.info("Starting HDFS...")
|
|
||||||
primary_node.ssh('/hadoop/sbin/start-dfs.sh')
|
|
||||||
|
|
||||||
logger.info("Starting YARN...")
|
|
||||||
primary_node.ssh('/hadoop/sbin/start-yarn.sh')
|
|
||||||
|
|
||||||
logger.info('Starting HBase...')
|
|
||||||
primary_node.ssh('/hbase/bin/start-hbase.sh')
|
|
||||||
primary_node.ssh('/hbase/bin/hbase-daemon.sh start rest')
|
|
||||||
|
|
||||||
logger.info("NameNode and HBase master are located on %s. SSH over and have fun!",
|
|
||||||
primary_node.hostname)
|
|
||||||
|
|
||||||
logger.info("The HDFS NameNode web UI can be reached at http://%s:%s",
|
|
||||||
getfqdn(), get_host_port_binding(primary_node.container_id,
|
|
||||||
NAMENODE_WEB_UI_PORT))
|
|
||||||
|
|
||||||
logger.info("The YARN ResourceManager web UI can be reached at http://%s:%s",
|
|
||||||
getfqdn(), get_host_port_binding(primary_node.container_id,
|
|
||||||
RESOURCEMANAGER_WEB_UI_PORT))
|
|
||||||
|
|
||||||
logger.info("The HBase master web UI can be reached at http://%s:%s",
|
|
||||||
getfqdn(), get_host_port_binding(primary_node.container_id,
|
|
||||||
kwargs.get('hbase_master_web_ui_port')))
|
|
||||||
|
|
||||||
logger.info("The HBase REST server can be reached at http://%s:%s",
|
|
||||||
getfqdn(), get_host_port_binding(primary_node.container_id,
|
|
||||||
HBASE_REST_SERVER_PORT))
|
|
||||||
|
|
||||||
def build(args):
|
|
||||||
"""This function will be executed when ./bin/build_cluster apache_hbase is invoked."""
|
|
||||||
|
|
||||||
# pylint: disable=too-many-locals
|
|
||||||
# See start function above for rationale for disabling this warning.
|
|
||||||
|
|
||||||
container_build_dir = join(CLUSTERDOCK_VOLUME, str(uuid4()))
|
|
||||||
makedirs(container_build_dir)
|
|
||||||
|
|
||||||
# If --hbase-git-commit is specified, we build HBase from source.
|
|
||||||
if args.hbase_git_commit:
|
|
||||||
build_hbase_commands = [
|
|
||||||
"git clone https://github.com/apache/hbase.git {0}".format(container_build_dir),
|
|
||||||
"git -C {0} checkout {1}".format(container_build_dir, args.hbase_git_commit),
|
|
||||||
"mvn --batch-mode clean install -DskipTests assembly:single -f {0}/pom.xml".format(
|
|
||||||
container_build_dir
|
|
||||||
)
|
|
||||||
]
|
|
||||||
|
|
||||||
maven_image = Constants.docker_images.maven # pylint: disable=no-member
|
|
||||||
if not is_image_available_locally(maven_image):
|
|
||||||
pull_image(maven_image)
|
|
||||||
|
|
||||||
container_configs = {
|
|
||||||
'command': 'bash -c "{0}"'.format(' && '.join(build_hbase_commands)),
|
|
||||||
'image': maven_image,
|
|
||||||
'host_config': client.create_host_config(volumes_from=get_clusterdock_container_id())
|
|
||||||
}
|
|
||||||
|
|
||||||
maven_container_id = client.create_container(**container_configs)['Id']
|
|
||||||
client.start(container=maven_container_id)
|
|
||||||
for line in client.logs(container=maven_container_id, stream=True):
|
|
||||||
stdout.write(line)
|
|
||||||
stdout.flush()
|
|
||||||
|
|
||||||
# Mimic docker run --rm by blocking on docker wait and then removing the container
|
|
||||||
# if it encountered no errors.
|
|
||||||
if client.wait(container=maven_container_id) == EX_OK:
|
|
||||||
client.remove_container(container=maven_container_id, force=True)
|
|
||||||
else:
|
|
||||||
raise Exception('Error encountered while building HBase.')
|
|
||||||
|
|
||||||
assembly_target_dir = join(container_build_dir, 'hbase-assembly', 'target')
|
|
||||||
for a_file in listdir(assembly_target_dir):
|
|
||||||
if a_file.endswith('bin.tar.gz'):
|
|
||||||
args.hbase_tarball = join(assembly_target_dir, a_file)
|
|
||||||
break
|
|
||||||
|
|
||||||
# Download all the binary tarballs into our temporary directory so that we can add them
|
|
||||||
# into the Docker image we're building.
|
|
||||||
filenames = []
|
|
||||||
for tarball_location in [args.java_tarball, args.hadoop_tarball, args.hbase_tarball]:
|
|
||||||
tarball_filename = tarball_location.rsplit('/', 1)[-1]
|
|
||||||
filenames.append(tarball_filename)
|
|
||||||
|
|
||||||
# Download tarballs given as URLs.
|
|
||||||
if container_build_dir not in tarball_location:
|
|
||||||
get_request = requests.get(tarball_location, stream=True, cookies=(
|
|
||||||
{'oraclelicense': 'accept-securebackup-cookie'}
|
|
||||||
if tarball_location == args.java_tarball
|
|
||||||
else None
|
|
||||||
))
|
|
||||||
# Raise Exception if download failed.
|
|
||||||
get_request.raise_for_status()
|
|
||||||
logger.info("Downloading %s...", tarball_filename)
|
|
||||||
with open(join(container_build_dir, tarball_filename), 'wb') as file_descriptor:
|
|
||||||
for chunk in get_request.iter_content(1024):
|
|
||||||
file_descriptor.write(chunk)
|
|
||||||
else:
|
|
||||||
move(tarball_location, container_build_dir)
|
|
||||||
|
|
||||||
dockerfile_contents = r"""
|
|
||||||
FROM {nodebase_image}
|
|
||||||
COPY {java_tarball} /tarballs/
|
|
||||||
RUN mkdir /java && tar -xf /tarballs/{java_tarball} -C /java --strip-components=1
|
|
||||||
RUN echo "JAVA_HOME=/java" >> /etc/environment
|
|
||||||
|
|
||||||
COPY {hadoop_tarball} /tarballs/
|
|
||||||
RUN mkdir /hadoop && tar -xf /tarballs/{hadoop_tarball} -C /hadoop --strip-components=1
|
|
||||||
COPY {hbase_tarball} /tarballs/
|
|
||||||
RUN mkdir /hbase && tar -xf /tarballs/{hbase_tarball} -C /hbase --strip-components=1
|
|
||||||
|
|
||||||
# Remove tarballs folder.
|
|
||||||
RUN rm -rf /tarballs
|
|
||||||
|
|
||||||
# Set PATH explicitly.
|
|
||||||
RUN echo "PATH=/java/bin:/hadoop/bin:/hbase/bin/:$(echo $PATH)" >> /etc/environment
|
|
||||||
|
|
||||||
# Add hbase user and group before copying root's SSH keys over.
|
|
||||||
RUN groupadd hbase \
|
|
||||||
&& useradd -g hbase hbase \
|
|
||||||
&& cp -R /root/.ssh ~hbase \
|
|
||||||
&& chown -R hbase:hbase ~hbase/.ssh
|
|
||||||
|
|
||||||
# Disable requiretty in /etc/sudoers as required by HBase chaos monkey.
|
|
||||||
RUN sed -i 's/Defaults\s*requiretty/#&/' /etc/sudoers
|
|
||||||
""".format(nodebase_image='/'.join([item
|
|
||||||
for item in [args.registry_url,
|
|
||||||
args.namespace or DEFAULT_APACHE_NAMESPACE,
|
|
||||||
"clusterdock:{os}_nodebase".format(
|
|
||||||
os=args.operating_system
|
|
||||||
)]
|
|
||||||
if item]),
|
|
||||||
java_tarball=filenames[0], hadoop_tarball=filenames[1], hbase_tarball=filenames[2])
|
|
||||||
|
|
||||||
logger.info("Created Dockerfile: %s", dockerfile_contents)
|
|
||||||
|
|
||||||
with open(join(container_build_dir, 'Dockerfile'), 'w') as dockerfile:
|
|
||||||
dockerfile.write(dockerfile_contents)
|
|
||||||
|
|
||||||
image = '/'.join(
|
|
||||||
[item
|
|
||||||
for item in [args.registry_url, args.namespace or DEFAULT_APACHE_NAMESPACE,
|
|
||||||
"clusterdock:{os}_java-{java}_hadoop-{hadoop}_hbase-{hbase}".format(
|
|
||||||
os=args.operating_system, java=args.java_version,
|
|
||||||
hadoop=args.hadoop_version, hbase=args.hbase_version
|
|
||||||
)]
|
|
||||||
if item])
|
|
||||||
|
|
||||||
logger.info("Building image %s...", image)
|
|
||||||
build_image(dockerfile=join(container_build_dir, 'Dockerfile'), tag=image)
|
|
||||||
|
|
||||||
logger.info("Removing build temporary directory...")
|
|
||||||
return [image]
|
|
|
@ -1,80 +0,0 @@
|
||||||
# Licensed to the Apache Software Foundation (ASF) under one
|
|
||||||
# or more contributor license agreements. See the NOTICE file
|
|
||||||
# distributed with this work for additional information
|
|
||||||
# regarding copyright ownership. The ASF licenses this file
|
|
||||||
# to you under the Apache License, Version 2.0 (the
|
|
||||||
# "License"); you may not use this file except in compliance
|
|
||||||
# with the License. You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
#
|
|
||||||
# This configuration file is used by the apache-hbase clusterdock topology to populate configuration
|
|
||||||
# files for an HBase cluster. Section names denote the filenames (e.g. [hadoop/core-site.xml]) with
|
|
||||||
# the section's corresponding items denoting properties.
|
|
||||||
#
|
|
||||||
# Filenames ending with .xml will have items run through an XML parser. That is,
|
|
||||||
#
|
|
||||||
# [hbase/hbase-site.xml]
|
|
||||||
# hbase.cluster.distributed = true
|
|
||||||
#
|
|
||||||
# would lead to the creation of an hbase-site.xml file containing:
|
|
||||||
#
|
|
||||||
# <property>
|
|
||||||
# <name>hbase.cluster.distributed</name>
|
|
||||||
# <value>true</value>
|
|
||||||
# </property>
|
|
||||||
#
|
|
||||||
# Note, also, that items in non-xml files can be copied verbatim by with using the "body:" item (be
|
|
||||||
# sure to include leading whitespace must be used for the following lines). For example:
|
|
||||||
#
|
|
||||||
# [hbase/hbase-env.sh]
|
|
||||||
# body:
|
|
||||||
# COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+UseG1GC"
|
|
||||||
# COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+PrintGCDetails"
|
|
||||||
#
|
|
||||||
# would result in an hbase-env.sh file with:
|
|
||||||
#
|
|
||||||
# COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+UseG1GC"
|
|
||||||
# COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+PrintGCDetails"
|
|
||||||
#
|
|
||||||
# Two last notes:
|
|
||||||
# 1. Items starting with +++ will be eval'd with Python directly.
|
|
||||||
# 2. As defined in action.py, some wildcards are processed at cluster start time (e.g. {network}).
|
|
||||||
|
|
||||||
[hadoop/slaves]
|
|
||||||
+++ '\n'.join(["{{0}}.{network}".format(node) for node in {secondary_nodes}])
|
|
||||||
|
|
||||||
[hadoop/core-site.xml]
|
|
||||||
fs.default.name = hdfs://{primary_node[0]}.{network}:8020
|
|
||||||
|
|
||||||
[hadoop/mapred-site.xml]
|
|
||||||
mapreduce.framework.name = yarn
|
|
||||||
|
|
||||||
[hadoop/yarn-site.xml]
|
|
||||||
yarn.resourcemanager.hostname = {primary_node[0]}.{network}
|
|
||||||
yarn.nodemanager.aux-services = mapreduce_shuffle
|
|
||||||
yarn.nodemanager.aux-services.mapreduce_shuffle.class = org.apache.hadoop.mapred.ShuffleHandler
|
|
||||||
yarn.nodemanager.vmem-check-enabled = false
|
|
||||||
|
|
||||||
[hbase/regionservers]
|
|
||||||
+++ '\n'.join(["{{0}}.{network}".format(node) for node in {secondary_nodes}])
|
|
||||||
|
|
||||||
[hbase/backup-masters]
|
|
||||||
{secondary_nodes[0]}.{network}
|
|
||||||
|
|
||||||
[hbase/hbase-site.xml]
|
|
||||||
hbase.cluster.distributed = true
|
|
||||||
hbase.rootdir = hdfs://{primary_node[0]}.{network}/hbase
|
|
||||||
hbase.zookeeper.quorum = {primary_node[0]}.{network}
|
|
||||||
hbase.zookeeper.property.dataDir = /usr/local/zookeeper
|
|
||||||
|
|
||||||
# For now, set service users for Chaos Monkey to be root.
|
|
||||||
hbase.it.clustermanager.hadoop.hdfs.user = root
|
|
||||||
hbase.it.clustermanager.zookeeper.user = root
|
|
||||||
hbase.it.clustermanager.hbase.user = root
|
|
|
@ -1,82 +0,0 @@
|
||||||
# Licensed to the Apache Software Foundation (ASF) under one
|
|
||||||
# or more contributor license agreements. See the NOTICE file
|
|
||||||
# distributed with this work for additional information
|
|
||||||
# regarding copyright ownership. The ASF licenses this file
|
|
||||||
# to you under the Apache License, Version 2.0 (the
|
|
||||||
# "License"); you may not use this file except in compliance
|
|
||||||
# with the License. You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
#
|
|
||||||
# This configuration file is used to define some general properties of the apache-hbase clusterdock
|
|
||||||
# topology, as well as its command line.
|
|
||||||
|
|
||||||
[general]
|
|
||||||
name = Apache HBase
|
|
||||||
description = An Apache HBase cluster with 1 primary node and n-1 secondary nodes
|
|
||||||
|
|
||||||
|
|
||||||
[node_groups]
|
|
||||||
# Define node groups and specify which of each to start during the build process.
|
|
||||||
primary-node = node-1
|
|
||||||
secondary-nodes = node-2
|
|
||||||
|
|
||||||
|
|
||||||
[build]
|
|
||||||
arg.java-tarball = http://download.oracle.com/otn-pub/java/jdk/8u91-b14/jdk-8u91-linux-x64.tar.gz
|
|
||||||
arg.java-tarball.help = The URL (or filename) of the Java tarball to install on the cluster
|
|
||||||
arg.java-tarball.metavar = tarball
|
|
||||||
|
|
||||||
arg.java-version = 8u91
|
|
||||||
arg.java-version.help = The label to use when identifying the version of Java
|
|
||||||
arg.java-version.metavar = ver
|
|
||||||
|
|
||||||
arg.hadoop-tarball = https://archive.apache.org/dist/hadoop/core/hadoop-2.7.2/hadoop-2.7.2.tar.gz
|
|
||||||
arg.hadoop-tarball.help = The URL (or filename) of the Hadoop tarball to install on the cluster
|
|
||||||
arg.hadoop-tarball.metavar = tarball
|
|
||||||
|
|
||||||
arg.hadoop-version = 2.7.2
|
|
||||||
arg.hadoop-version.help = The label to use when identifying the version of Hadoop
|
|
||||||
arg.hadoop-version.metavar = ver
|
|
||||||
|
|
||||||
arg.hbase-tarball
|
|
||||||
arg.hbase-tarball.help = The URL (or filename) of the HBase tarball to install on the cluster
|
|
||||||
arg.hbase-tarball.metavar = tarball
|
|
||||||
|
|
||||||
arg.hbase-version
|
|
||||||
arg.hbase-version.help = The label to use when identifying the version of HBase
|
|
||||||
arg.hbase-version.metavar = ver
|
|
||||||
|
|
||||||
arg.hbase-git-commit
|
|
||||||
arg.hbase-git-commit.help = The git commit to checkout when building the image
|
|
||||||
arg.hbase-git-commit.metavar = commit
|
|
||||||
|
|
||||||
[start]
|
|
||||||
arg.java-version = 8u91
|
|
||||||
arg.java-version.help = The Java version on the cluster
|
|
||||||
arg.java-version.metavar = ver
|
|
||||||
|
|
||||||
arg.hadoop-version = 2.7.2
|
|
||||||
arg.hadoop-version.help = The Hadoop version on the cluster
|
|
||||||
arg.hadoop-version.metavar = ver
|
|
||||||
|
|
||||||
arg.hbase-version = master
|
|
||||||
arg.hbase-version.help = The HBase version on the cluster
|
|
||||||
arg.hbase-version.metavar = ver
|
|
||||||
|
|
||||||
arg.configurations = /root/clusterdock/clusterdock/topologies/apache_hbase/configurations.cfg
|
|
||||||
arg.configurations.help = Location of a configurations.cfg file to apply to the cluster
|
|
||||||
arg.configurations.metavar = path
|
|
||||||
|
|
||||||
arg.data-directories
|
|
||||||
arg.data-directories.help = A comma-separated list of host directories on which to save data (i.e. for HDFS and YARN)
|
|
||||||
arg.data-directories.metavar = dir
|
|
||||||
|
|
||||||
arg.dont-start-services = False
|
|
||||||
arg.dont-start-services.help = Don't start Hadoop and HBase services as part of cluster start
|
|
|
@ -1,44 +0,0 @@
|
||||||
#/**
|
|
||||||
# * Licensed to the Apache Software Foundation (ASF) under one
|
|
||||||
# * or more contributor license agreements. See the NOTICE file
|
|
||||||
# * distributed with this work for additional information
|
|
||||||
# * regarding copyright ownership. The ASF licenses this file
|
|
||||||
# * to you under the Apache License, Version 2.0 (the
|
|
||||||
# * "License"); you may not use this file except in compliance
|
|
||||||
# * with the License. You may obtain a copy of the License at
|
|
||||||
# *
|
|
||||||
# * http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
# *
|
|
||||||
# * Unless required by applicable law or agreed to in writing, software
|
|
||||||
# * distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# * See the License for the specific language governing permissions and
|
|
||||||
# * limitations under the License.
|
|
||||||
# */
|
|
||||||
-----BEGIN RSA PRIVATE KEY-----
|
|
||||||
MIIEpAIBAAKCAQEAtj/yZZNF+bv26eqWqsx+vFehSlxBJp/QhIrFWKpjHcpQJ29o
|
|
||||||
6hJN9moU3Goft2C5w6FoZdjC1TWlzhxUnRS8xeksFnW3ItvkjySLA1Iq6jchYxNd
|
|
||||||
fZ3HwTdH0rubM1uJ/CnkaijoxBqGBmPSL0TxfqcteaJ8APhslwl0WWJ6b+tBCHDV
|
|
||||||
pTLATebtggCAfhKmSuAYmn3QIqJ7DFoSGkwhkxpUHuuVCZxUH3CIxLIw+6npArr/
|
|
||||||
S5gtFo50oi6FXPmvv6mJg6yLqq3VlKcQh6d/COaJopHn+nLed2kECESUlpTruMpr
|
|
||||||
6IcGESgz4hnkmhop8oTY42sQJhPPF2Ahq9a3aQIDAQABAoIBADSbxPb5SkvKrH3d
|
|
||||||
j9yB51uq2A5FDzF9FI4OGOV9WdsxmW2oxVo8KnElMhxmLf2bWERWhXJQ3fz53YDf
|
|
||||||
wLUPVWaz5lwdYt4XJ6UCYXZ185lkjKiy4FvwfccSlBMKwMRUekJmPV8/q+Ff3qxd
|
|
||||||
iEDI4AU1cPUZqD4HeCEpQ4LB4KIJhCdLkCgoWxxaCwwuB6DnwB4P4VLeAfsm2NEX
|
|
||||||
k9dld87q/miOmuw9QsmSv9wYiQqoPdV5Qj7KYqHBAa6syqUfFni3Ibmw1WBzMydp
|
|
||||||
8YyP9HvrzDBMnPPzkmp6od0fAgGafIlkIxz/9sCKOSISnuuqahbNAJK/rIiJzLY3
|
|
||||||
Pi49M+ECgYEA2vCFObmM/hpKUPNxG841nQScFfeMg1q1z1krEmfjqDTELXyq9AOS
|
|
||||||
fGiVTxfWagOGoAWIwg3ZfgGEmxsKrOSxkFvabWdhN1Wn98Zf8fJG8YAwLYg8JOgf
|
|
||||||
gZ5pkxFW4FwrAeFDyJyKvFJVrbDw1PM41yvTmRzf3NjcaqJrBE2fgKUCgYEA1RmF
|
|
||||||
XjfMlBoMLZ4pXS1zF91WgOF4SNdJJj9RCGzqqdy+DzPKNAZVa0HBhaIHCZXL3Hcv
|
|
||||||
zqgEb6RSMysyVYjZPwfSwevsSuxpfriVpYux5MN3AEXX5Ysv51WWStWgt9+iQlfo
|
|
||||||
xAdxxukOa++PZ4Z+TIIEDAFS47rnKEQUh+ZNfHUCgYEA0amTa3wtcQlsMalvn9kR
|
|
||||||
rpRDhSXTAddUVIRnovCqKuKdG5JPg+4H0eu1UFDbnBpUSdoC5RKuPOTnQEHdL0Sy
|
|
||||||
ZjQQMMTXbE4y1Cy8pM4G8i539KKKNi20PkSdhaENOT4KUXqPlwWSNlYChprzhnqE
|
|
||||||
7EmkEPR9zNg//D4djbloDaECgYANOJIfsFKO9ba/tcpXL5SubFsLj/GIg2LUbqU2
|
|
||||||
YpuEgl+ATfRDmgj+qIu7ILxTCeol+XcL2Ty9OHKpHgr3Z5Ai6vdWdK6qT1SUOht+
|
|
||||||
s9YLnVzqtWqZoTMNpS+34N0hy0wj1ZRpZRTYBGmSpMA+6gc38/EQVZyw6E2jH+Yu
|
|
||||||
MEmqaQKBgQDGh9uXCl/WjhBOF9VLrX/Aeaa2Mzrh21Ic1dw6aWrE4EW6k1LvSP36
|
|
||||||
evrvrs2jQuzRMGH6DKX8ImnVEWjK+gZfgf2MuyDSW7KYR5zxkdZtRkotF6X0fu6N
|
|
||||||
8uLa7CN8UmS4FiAMLwNbTJ6zA6ohny7r+AiOqNGlP9vBFMhpGs3NFg==
|
|
||||||
-----END RSA PRIVATE KEY-----
|
|
|
@ -1,18 +0,0 @@
|
||||||
#/**
|
|
||||||
# * Licensed to the Apache Software Foundation (ASF) under one
|
|
||||||
# * or more contributor license agreements. See the NOTICE file
|
|
||||||
# * distributed with this work for additional information
|
|
||||||
# * regarding copyright ownership. The ASF licenses this file
|
|
||||||
# * to you under the Apache License, Version 2.0 (the
|
|
||||||
# * "License"); you may not use this file except in compliance
|
|
||||||
# * with the License. You may obtain a copy of the License at
|
|
||||||
# *
|
|
||||||
# * http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
# *
|
|
||||||
# * Unless required by applicable law or agreed to in writing, software
|
|
||||||
# * distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# * See the License for the specific language governing permissions and
|
|
||||||
# * limitations under the License.
|
|
||||||
# */
|
|
||||||
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2P/Jlk0X5u/bp6paqzH68V6FKXEEmn9CEisVYqmMdylAnb2jqEk32ahTcah+3YLnDoWhl2MLVNaXOHFSdFLzF6SwWdbci2+SPJIsDUirqNyFjE119ncfBN0fSu5szW4n8KeRqKOjEGoYGY9IvRPF+py15onwA+GyXCXRZYnpv60EIcNWlMsBN5u2CAIB+EqZK4BiafdAionsMWhIaTCGTGlQe65UJnFQfcIjEsjD7qekCuv9LmC0WjnSiLoVc+a+/qYmDrIuqrdWUpxCHp38I5omikef6ct53aQQIRJSWlOu4ymvohwYRKDPiGeSaGinyhNjjaxAmE88XYCGr1rdp clusterdock
|
|
Loading…
Reference in New Issue