219 lines
7.7 KiB
Markdown
219 lines
7.7 KiB
Markdown
---
|
|
title: "Ozone Containers"
|
|
summary: Ozone uses containers extensively for testing. This page documents the usage and best practices of Ozone.
|
|
weight: 2
|
|
---
|
|
<!---
|
|
Licensed to the Apache Software Foundation (ASF) under one or more
|
|
contributor license agreements. See the NOTICE file distributed with
|
|
this work for additional information regarding copyright ownership.
|
|
The ASF licenses this file to You under the Apache License, Version 2.0
|
|
(the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
-->
|
|
|
|
Docker heavily is used at the ozone development with three principal use-cases:
|
|
|
|
* __dev__:
|
|
* We use docker to start local pseudo-clusters (docker provides unified environment, but no image creation is required)
|
|
* __test__:
|
|
* We create docker images from the dev branches to test ozone in kubernetes and other container orchestator system
|
|
* We provide _apache/ozone_ images for each release to make it easier the evaluation of Ozone. These images are __not__ created __for production__ usage.
|
|
|
|
<div class="alert alert-warning" role="alert">
|
|
We <b>strongly</b> recommend that you create your own custom images when you
|
|
deploy ozone into production using containers. Please treat all the standard
|
|
shipped container images and k8s resources as examples and guides to help you
|
|
customize your own deployment.
|
|
</div>
|
|
|
|
* __production__:
|
|
* We document how can you create your own docker image for your production cluster.
|
|
|
|
Let's check out each of the use-cases in more detail:
|
|
|
|
## Development
|
|
|
|
Ozone artifact contains example docker-compose directories to make it easier to start Ozone cluster in your local machine.
|
|
|
|
From distribution:
|
|
|
|
```
|
|
cd compose/ozone
|
|
docker-compose up -d
|
|
```
|
|
|
|
After a local build
|
|
|
|
```
|
|
cd hadoop-ozone/dist/target/ozone-*/compose
|
|
docker-compose up -d
|
|
```
|
|
|
|
These environments are very important tools to start different type of Ozone clusters at any time.
|
|
|
|
To be sure that the compose files are up-to-date, we also provide acceptance test suites which start the cluster and check the basic behaviour.
|
|
|
|
The acceptance tests are part of the distribution, and you can find the test definitions in `./smoketest` directory.
|
|
|
|
You can start the tests from any compose directory:
|
|
|
|
For example:
|
|
|
|
```
|
|
cd compose/ozone
|
|
./test.sh
|
|
```
|
|
|
|
### Implementation details
|
|
|
|
`./compose` tests are based on the apache/hadoop-runner docker image. The image itself doesn't contain any Ozone jar file or binary just the helper scripts to start ozone.
|
|
|
|
hadoop-runner provdes a fixed environment to run Ozone everywhere, but the ozone distribution itself is mounted from the including directory:
|
|
|
|
(Example docker-compose fragment)
|
|
|
|
```
|
|
scm:
|
|
image: apache/hadoop-runner:jdk11
|
|
volumes:
|
|
- ../..:/opt/hadoop
|
|
ports:
|
|
- 9876:9876
|
|
|
|
```
|
|
|
|
The containers are conigured based on environment variables, but because the same environment variables should be set for each containers we maintain the list of the environment variables in a separated file:
|
|
|
|
```
|
|
scm:
|
|
image: apache/hadoop-runner:jdk11
|
|
#...
|
|
env_file:
|
|
- ./docker-config
|
|
```
|
|
|
|
The docker-config file contains the list of the required environment variables:
|
|
|
|
```
|
|
OZONE-SITE.XML_ozone.om.address=om
|
|
OZONE-SITE.XML_ozone.om.http-address=om:9874
|
|
OZONE-SITE.XML_ozone.scm.names=scm
|
|
OZONE-SITE.XML_ozone.enabled=True
|
|
#...
|
|
```
|
|
|
|
As you can see we use naming convention. Based on the name of the environment variable, the appropariate hadoop config XML (`ozone-site.xml` in our case) will be generated by a [script](https://github.com/apache/hadoop/tree/docker-hadoop-runner-latest/scripts) which is included in the `hadoop-runner` base image.
|
|
|
|
The [entrypoint](https://github.com/apache/hadoop/blob/docker-hadoop-runner-latest/scripts/starter.sh) of the `hadoop-runner` image contains a helper shell script which triggers this transformation and cab do additional actions (eg. initialize scm/om storage, download required keytabs, etc.) based on environment variables.
|
|
|
|
## Test/Staging
|
|
|
|
The `docker-compose` based approach is recommended only for local test not for multi node cluster. To use containers on a multi-node cluster we need a Container Orchestrator like Kubernetes.
|
|
|
|
Kubernetes example files are included in the `kubernetes` folder.
|
|
|
|
*Please note*: all the provided images are based the `hadoop-runner` image which contains all the required tool for testing in staging environments. For production we recommend to create your own, hardened image with your own base image.
|
|
|
|
### Test the release
|
|
|
|
The release can be tested with deploying any of the example clusters:
|
|
|
|
```
|
|
cd kubernetes/examples/ozone
|
|
kubectl apply -f
|
|
```
|
|
|
|
Plese note that in this case the latest released container will be downloaded from the dockerhub.
|
|
|
|
### Test the development build
|
|
|
|
To test a development build you can create your own image and upload it to your own docker registry:
|
|
|
|
|
|
```
|
|
mvn clean install -f pom.ozone.xml -DskipTests -Pdocker-build,docker-push -Ddocker.image=myregistry:9000/name/ozone
|
|
```
|
|
|
|
The configured image will be used in all the generated kubernetes resources files (`image:` keys are adjusted during the build)
|
|
|
|
```
|
|
cd kubernetes/examples/ozone
|
|
kubectl apply -f
|
|
```
|
|
|
|
## Production
|
|
|
|
<div class="alert alert-danger" role="alert">
|
|
We <b>strongly</b> recommend to use your own image in your production cluster
|
|
and
|
|
adjust base image, umask, security settings, user settings according to your own requirements.
|
|
</div>
|
|
|
|
You can use the source of our development images as an example:
|
|
|
|
* Base image: https://github.com/apache/hadoop/blob/docker-hadoop-runner-jdk11/Dockerfile
|
|
* Docker image: https://github.com/apache/hadoop/blob/trunk/hadoop-ozone/dist/src/main/Dockerfile
|
|
|
|
Most of the elements are optional and just helper function but to use the provided example kubernetes resources you may need the scripts from [here](https://github.com/apache/hadoop/tree/docker-hadoop-runner-jdk11/scripts)
|
|
|
|
* The two python scripts convert environment variables to real hadoop XML config files
|
|
* The start.sh executes the python scripts (and other initialization) based on environment variables.
|
|
|
|
## Containers
|
|
|
|
Ozone related container images and source locations:
|
|
|
|
|
|
<table class="table table-dark">
|
|
<thead>
|
|
<tr>
|
|
<th scope="col">#</th>
|
|
<th scope="col">Container</th>
|
|
<th scope="col">Repository</th>
|
|
<th scope="col">Base</th>
|
|
<th scope="col">Branch</th>
|
|
<th scope="col">Tags</th>
|
|
<th scope="col">Comments</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<th scope="row">1</th>
|
|
<td>apache/ozone</td>
|
|
<td>https://github.com/apache/hadoop-docker-ozone</td>
|
|
<td>ozone-... </td>
|
|
<td>hadoop-runner</td>
|
|
<td>0.3.0,0.4.0,0.4.1</td>
|
|
<td>For each Ozone release we create new release tag.</td>
|
|
</tr>
|
|
<tr>
|
|
<th scope="row">2</th>
|
|
<td>apache/hadoop-runner </td>
|
|
<td>https://github.com/apache/hadoop</td>
|
|
<td>docker-hadoop-runner</td>
|
|
<td>centos</td>
|
|
<td>jdk11,jdk8,latest</td>
|
|
<td>This is the base image used for testing Hadoop Ozone.
|
|
This is a set of utilities that make it easy for us run ozone.</td>
|
|
</tr>
|
|
<tr>
|
|
<th scope="row">3</th>
|
|
<td>apache/ozone:build (WIP)</td>
|
|
<td>https://github.com/apache/hadoop-docker-ozone</td>
|
|
<td>ozone-build </td>
|
|
<td> </td>
|
|
<td> </td>
|
|
<td>TODO: Add more documentation here.</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|