Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

HDFS-13414. Ozone: Update existing Ozone documentation according to the recent changes. Contributed by Elek, Marton.

This commit is contained in:
Mukul Kumar Singh 2018-04-12 21:21:44 +05:30 committed by Owen O'Malley
parent 8cc13ff962
commit 51459d4351
3 changed files with 78 additions and 51 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

38
hadoop-ozone/ozone-manager/src/main/site/markdown/OzoneCommandShell.md
View File

@ -25,10 +25,10 @@ The Ozone commands take the following format.
<name> -root`
The *port* specified in command should match the port mentioned in the config
property `dfs.datanode.http.address`. This property can be set in `hdfs-site.xml`.
The default value for the port is `9864` and is used in below commands.
property `hdds.rest.http-address`. This property can be set in `ozone-site.xml`.
The default value for the port is `9880` and is used in below commands.
The *--root* option is a command line short cut that allows *ozone oz*
The *-root* option is a command line short cut that allows *ozone oz*
commands to be run as the user that started the cluster. This is useful to
indicate that you want the commands to be run as some admin user. The only
reason for this option is that it makes the life of a lazy developer more
@ -44,37 +44,37 @@ ozone cluster.
Volumes can be created only by Admins. Here is an example of creating a volume.
* `ozone oz -createVolume http://localhost:9864/hive -user bilbo -quota
* `ozone oz -createVolume http://localhost:9880/hive -user bilbo -quota
100TB -root`
The above command creates a volume called `hive` owned by user `bilbo`. The
`--root` option allows the command to be executed as user `hdfs` which is an
`-root` option allows the command to be executed as user `hdfs` which is an
admin in the cluster.
### Update Volume
Updates information like ownership and quota on an existing volume.
* `ozone oz -updateVolume http://localhost:9864/hive -quota 500TB -root`
* `ozone oz -updateVolume http://localhost:9880/hive -quota 500TB -root`
The above command changes the volume quota of hive from 100TB to 500TB.
### Delete Volume
Deletes a Volume if it is empty.
* `ozone oz -deleteVolume http://localhost:9864/hive -root`
* `ozone oz -deleteVolume http://localhost:9880/hive -root`
### Info Volume
Info volume command allows the owner or the administrator of the cluster to read meta-data about a specific volume.
* `ozone oz -infoVolume http://localhost:9864/hive -root`
* `ozone oz -infoVolume http://localhost:9880/hive -root`
### List Volumes
List volume command can be used by administrator to list volumes of any user. It can also be used by a user to list volumes owned by him.
* `ozone oz -listVolume http://localhost:9864/ -user bilbo -root`
* `ozone oz -listVolume http://localhost:9880/ -user bilbo -root`
The above command lists all volumes owned by user bilbo.
@ -89,7 +89,7 @@ Following examples assume that these commands are run by the owner of the volume
Create bucket call allows the owner of a volume to create a bucket.
* `ozone oz -createBucket http://localhost:9864/hive/january`
* `ozone oz -createBucket http://localhost:9880/hive/january`
This call creates a bucket called `january` in the volume called `hive`. If
the volume does not exist, then this call will fail.
@ -98,23 +98,23 @@ the volume does not exist, then this call will fail.
### Update Bucket
Updates bucket meta-data, like ACLs.
* `ozone oz -updateBucket http://localhost:9864/hive/january -addAcl
* `ozone oz -updateBucket http://localhost:9880/hive/january -addAcl
user:spark:rw`
### Delete Bucket
Deletes a bucket if it is empty.
* `ozone oz -deleteBucket http://localhost:9864/hive/january`
* `ozone oz -deleteBucket http://localhost:9880/hive/january`
### Info Bucket
Returns information about a given bucket.
* `ozone oz -infoBucket http://localhost:9864/hive/january`
* `ozone oz -infoBucket http://localhost:9880/hive/january`
### List Buckets
List buckets on a given volume.
* `ozone oz -listBucket http://localhost:9864/hive`
* `ozone oz -listBucket http://localhost:9880/hive`
Ozone Key Commands
------------------
@ -125,26 +125,26 @@ Ozone key commands allows users to put, delete and get keys from ozone buckets.
Creates or overwrites a key in ozone store, -file points to the file you want
to upload.
* `ozone oz -putKey http://localhost:9864/hive/january/processed.orc -file
* `ozone oz -putKey http://localhost:9880/hive/january/processed.orc -file
processed.orc`
### Get Key
Downloads a file from the ozone bucket.
* `ozone oz -getKey http://localhost:9864/hive/january/processed.orc -file
* `ozone oz -getKey http://localhost:9880/hive/january/processed.orc -file
processed.orc.copy`
### Delete Key
Deletes a key from the ozone store.
* `ozone oz -deleteKey http://localhost:9864/hive/january/processed.orc`
* `ozone oz -deleteKey http://localhost:9880/hive/january/processed.orc`
### Info Key
Reads key metadata from the ozone store.
* `ozone oz -infoKey http://localhost:9864/hive/january/processed.orc`
* `ozone oz -infoKey http://localhost:9880/hive/january/processed.orc`
### List Keys
List all keys in an ozone bucket.
* `ozone oz -listKey http://localhost:9864/hive/january`
* `ozone oz -listKey http://localhost:9880/hive/january`

59
hadoop-ozone/ozone-manager/src/main/site/markdown/OzoneGettingStarted.md.vm
View File

@ -22,16 +22,17 @@ API for accessing the store.
Getting Started
---------------
Ozone is a work in progress and currently lives in its own branch. To
Ozone is a work in progress and currently lives in the hadoop source tree.
The subprojects (ozone/hdds) are part of the hadoop source tree but by default
not compiled and not part of the official releases. To
use it, you have to build a package by yourself and deploy a cluster.
### Building Ozone
To build Ozone, please checkout the hadoop sources from github. Then
checkout the ozone branch, HDFS-7240 and build it.
checkout the trunk branch and build it.
- `git checkout HDFS-7240`
- `mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true -Pdist -Phdsl -Dtar -DskipShade`
`mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true -Pdist -Phdds -Dtar -DskipShade`
skipShade is just to make compilation faster and not really required.
@ -66,13 +67,12 @@ Now let us run some work load against ozone, to do that we will run freon.
This will log into the datanode and run bash.
- `docker-compose exec datanode bash`
- `cd /opt/hadoop/bin`
Now you can run the oz command shell or freon, the ozone load generator.
Now you can run the `ozone` command shell or freon, the ozone load generator.
This is the command to run freon.
- `./oz freon -mode offline -validateWrites -numOfVolumes 1 -numOfBuckets 10 -numOfKeys 100`
- `ozone freon -mode offline -validateWrites -numOfVolumes 1 -numOfBuckets 10 -numOfKeys 100`
You can checkout the KSM UI to see the requests information.
@ -122,9 +122,9 @@ place and not mingled with HDFS settings.
</property>
```
* _*ozone.scm.names*_ Ozone is build on top of container framework (See Ozone
Architecture TODO). Storage container manager(SCM) is a distributed block
service which is used by ozone and other storage services.
* _*ozone.scm.names*_ Ozone is build on top of container framework. Storage
container manager(SCM) is a distributed block service which is used by ozone
and other storage services.
This property allows datanodes to discover where SCM is, so that
datanodes can send heartbeat to SCM. SCM is designed to be highly available
and datanodes assume there are multiple instances of SCM which form a highly
@ -175,6 +175,19 @@ Ozone File System.
</property>
```
* _*dfs.datanode.plugin*_ Datanode service plugins: the container manager part
of ozone is running inside the datanode as a service plugin. To activate ozone
you should define the service plugin implementation class. **Important**
It should be added to the **hdfs-site.xml** as the plugin should be activated
as part of the normal HDFS Datanode bootstrap.
```
<property>
<name>dfs.datanode.plugins</name>
<value>org.apache.hadoop.ozone.HddsDatanodeService</value>
</property>
```
Here is a quick summary of settings needed by Ozone.
| Setting | Value | Comment |
@ -230,6 +243,16 @@ Here is a quick summary of settings needed by Ozone.
</configuration>
```
And don't forget to enable the datanode component with adding the
following configuration to the hdfs-site.xml:
```
<property>
<name>dfs.datanode.plugins</name>
<value>org.apache.hadoop.ozone.HddsDatanodeService</value>
</property>
```
### Starting Ozone
Ozone is designed to run concurrently with HDFS. The simplest way to [start
@ -246,19 +269,19 @@ is running, please verify it is fully functional by running some commands like
The first time you bring up Ozone, SCM must be initialized.
- `./hdfs scm -init`
- `./ozone scm -init`
Start SCM.
- `./hdfs --daemon start scm`
- `./ozone --daemon start scm`
Once SCM gets started, KSM must be initialized.
- `./hdfs ksm -createObjectStore`
- `./ozone ksm -createObjectStore`
Start KSM.
- `./hdfs --daemon start ksm`
- `./ozone --daemon start ksm`
if you would like to start HDFS and Ozone together, you can do that by running
a single command.
@ -316,5 +339,9 @@ On the SCM/KSM side, you will be able to see
- `hadoop-hdfs-ksm-hostname.log`
- `hadoop-hdfs-scm-hostname.log`
Please file any issues you see under [Object store in HDFS (HDFS-7240)](https://issues.apache.org/jira/browse/HDFS-7240)
as this is still a work in progress.
Please file any issues you see under the related issues:
- [Object store in HDFS: HDFS-7240](https://issues.apache.org/jira/browse/HDFS-7240)
- [Ozone File System: HDFS-13074](https://issues.apache.org/jira/browse/HDFS-13074)
- [Building HDFS on top of new storage layer (HDDS): HDFS-10419](https://issues.apache.org/jira/browse/HDFS-10419)

32
hadoop-ozone/ozone-manager/src/main/site/markdown/OzoneRest.md
View File

@ -84,7 +84,7 @@ Query Parameter:
Sample HTTP POST request:
curl -i -X POST -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE root" "http://localhost:9864/volume-to-create"
curl -i -X POST -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE root" "http://localhost:9880/volume-to-create"
this request creates a volume as user *bilbo*, the authorization field is set to *OZONE root* because this call requires administration privilege. The client receives a response with zero content length.
@ -112,7 +112,7 @@ Query Parameter:
Sample HTTP PUT request:
curl -X PUT -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user: john" http://localhost:9864/volume-to-update
curl -X PUT -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user: john" http://localhost:9880/volume-to-update
this request modifies the owner of */volume-to-update* to *john*.
@ -126,7 +126,7 @@ Schema:
Sample HTTP DELETE request:
curl -i -X DELETE -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user: bilbo" http://localhost:9864/volume-to-delete
curl -i -X DELETE -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user: bilbo" http://localhost:9880/volume-to-delete
this request deletes an empty volume */volume-to-delete*. The client receives a zero length content.
@ -154,7 +154,7 @@ Query Parameter:
Sample HTTP GET request:
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/volume-of-bilbo?info=volume"
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/volume-of-bilbo?info=volume"
this request gets the info of volume */volume-of-bilbo*, the client receives a response with a JSON object of volume info.
@ -193,7 +193,7 @@ Query Parameter:
Sample HTTP GET request:
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/?max-keys=100&prefix=Jan"
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/?max-keys=100&prefix=Jan"
this request gets all volumes owned by *bilbo* and each volume's name contains prefix *Jan*, the result at most contains *100* entries. The client receives a list of SON objects, each of them describes the info of a volume.
@ -239,7 +239,7 @@ Additional HTTP Headers:
Sample HTTP POST request:
curl -i -X POST -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" http://localhost:9864/volume-of-bilbo/bucket-0
curl -i -X POST -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" http://localhost:9880/volume-of-bilbo/bucket-0
this request creates a bucket *bucket-0* under volume *volume-of-bilbo*.
@ -268,7 +268,7 @@ Additional HTTP Headers:
Sample HTTP PUT request:
curl -i -X PUT -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" -H "x-ozone-acl: ADD user:peregrin:rw" http://localhost:9864/volume-of-bilbo/bucket-to-update
curl -i -X PUT -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" -H "x-ozone-acl: ADD user:peregrin:rw" http://localhost:9880/volume-of-bilbo/bucket-to-update
this request adds an ACL policy specified by HTTP header *x-ozone-acl* to bucket */volume-of-bilbo/bucket-to-update*, the ACL field *ADD user:peregrin:rw* gives add additional read/write permission to user *peregrin* to this bucket.
@ -290,7 +290,7 @@ Schema:
Sample HTTP DELETE request:
curl -i -X DELETE -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user:bilbo" "http://localhost:9864/volume-of-bilbo/bucket-0"
curl -i -X DELETE -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user:bilbo" "http://localhost:9880/volume-of-bilbo/bucket-0"
this request deletes bucket */volume-of-bilbo/bucket-0*. The client receives a zero length content response.
@ -319,7 +319,7 @@ Query Parameters:
Sample HTTP GET request:
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/volume-of-bilbo/bucket-0?info=bucket"
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/volume-of-bilbo/bucket-0?info=bucket"
this request gets the info of bucket */volume-of-bilbo/bucket-0*. The client receives a response of JSON object contains bucket info.
@ -358,7 +358,7 @@ Query Parameters:
Sample HTTP GET request:
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/volume-of-bilbo?max-keys=10"
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/volume-of-bilbo?max-keys=10"
this request lists all the buckets under volume *volume-of-bilbo*, and the result at most contains 10 entries. The client receives response of a array of JSON objects, each of them represents for a bucket info.
@ -404,7 +404,7 @@ Additional HTTP headers:
Sample PUT HTTP request:
curl -X PUT -T /path/to/localfile -H "Authorization:OZONE" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user:bilbo" "http://localhost:9864/volume-of-bilbo/bucket-0/file-0"
curl -X PUT -T /path/to/localfile -H "Authorization:OZONE" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user:bilbo" "http://localhost:9880/volume-of-bilbo/bucket-0/file-0"
this request uploads a local file */path/to/localfile* specified by option *-T* to ozone as user *bilbo*, mapped to ozone key */volume-of-bilbo/bucket-0/file-0*. The client receives a zero length content response.
@ -418,7 +418,7 @@ Schema:
Sample HTTP GET request:
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/volume-of-bilbo/bucket-0/file-0"
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/volume-of-bilbo/bucket-0/file-0"
this request reads the content of key */volume-of-bilbo/bucket-0/file-0*. If the content of the file is plain text, it can be directly dumped onto stdout.
@ -434,7 +434,7 @@ this request reads the content of key */volume-of-bilbo/bucket-0/file-0*. If the
if the file is not plain text, specify *-O* option in curl command and the file *file-0* will be downloaded into current working directory, file name will be same as the key. A sample request like following:
curl -O -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/volume-of-bilbo/bucket-0/file-1"
curl -O -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/volume-of-bilbo/bucket-0/file-1"
response looks like following:
@ -452,7 +452,7 @@ Schema:
Sample HTTP DELETE request:
curl -i -X DELETE -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user:bilbo" "http://localhost:9864/volume-of-bilbo/bucket-0/file-0"
curl -i -X DELETE -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user:bilbo" "http://localhost:9880/volume-of-bilbo/bucket-0/file-0"
this request deletes key */volume-of-bilbo/bucket-0/file-0*. The client receives a zero length content result:
@ -480,7 +480,7 @@ Query Parameter:
Sample HTTP DELETE request:
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/volume-of-bilbo/buket-0/file-0?info=key"
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/volume-of-bilbo/buket-0/file-0?info=key"
this request returns information of the key */volume-of-bilbo/bucket-0/file-0*. The client receives a JSON object listed attributes of the key.
@ -519,7 +519,7 @@ Query Parameters:
Sample HTTP GET request:
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http:/localhost:9864/volume-of-bilbo/bucket-0/?max-keys=100&prefix=file"
curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http:/localhost:9880/volume-of-bilbo/bucket-0/?max-keys=100&prefix=file"
this request list keys under bucket */volume-of-bilbo/bucket-0*, the listing result is filtered by prefix *file*. The client receives an array of JSON objects, each of them represents the info of a matched key.