151 lines
4.9 KiB
Markdown
151 lines
4.9 KiB
Markdown
---
|
|
title: S3 Protocol
|
|
weight: 3
|
|
summary: Ozone supports Amazon's Simple Storage Service (S3) protocol. In fact, You can use S3 clients and S3 SDK based applications without any modifications with Ozone.
|
|
---
|
|
|
|
<!---
|
|
Licensed to the Apache Software Foundation (ASF) under one or more
|
|
contributor license agreements. See the NOTICE file distributed with
|
|
this work for additional information regarding copyright ownership.
|
|
The ASF licenses this file to You under the Apache License, Version 2.0
|
|
(the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
-->
|
|
|
|
|
|
Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools.
|
|
|
|
## Getting started
|
|
|
|
S3 Gateway is a separated component which provides the S3 compatible APIs. It should be started additional to the regular Ozone components.
|
|
|
|
You can start a docker based cluster, including the S3 gateway from the release package.
|
|
|
|
Go to the `compose/ozones3` directory, and start the server:
|
|
|
|
```bash
|
|
docker-compose up -d
|
|
```
|
|
|
|
You can access the S3 gateway at `http://localhost:9878`
|
|
|
|
## URL Schema
|
|
|
|
Ozone S3 gateway supports both the virtual-host-style URL s3 bucket addresses (eg. http://bucketname.host:9878) and the path-style addresses (eg. http://host:9878/bucketname)
|
|
|
|
By default it uses the path-style addressing. To use virtual host style URLs set your main domain name in your `ozone-site.xml`:
|
|
|
|
```xml
|
|
<property>
|
|
<name>ozone.s3g.domain.name</name>
|
|
<value>s3g.internal</value>
|
|
</property>
|
|
```
|
|
|
|
## Bucket browser
|
|
|
|
Buckets could be browsed from the browser by adding `?browser=true` to the bucket URL.
|
|
|
|
For example the content of the 'testbucket' could be checked from the browser using the URL http://localhost:9878/testbucket?browser=true
|
|
|
|
|
|
## Implemented REST endpoints
|
|
|
|
Operations on S3Gateway service:
|
|
|
|
Endpoint | Status |
|
|
------------|-------------|
|
|
GET service | implemented |
|
|
|
|
Operations on Bucket:
|
|
|
|
Endpoint | Status | Notes
|
|
------------------------------------|-------------|---------------
|
|
GET Bucket (List Objects) Version 2 | implemented |
|
|
HEAD Bucket | implemented |
|
|
DELETE Bucket | implemented |
|
|
PUT Bucket (Create bucket) | implemented |
|
|
Delete Multiple Objects (POST) | implemented |
|
|
|
|
Operation on Objects:
|
|
|
|
Endpoint | Status | Notes
|
|
------------------------------------|-----------------|---------------
|
|
PUT Object | implemented |
|
|
GET Object | implemented |
|
|
Multipart Upload | implemented | Except the listing of the current MultiPartUploads.
|
|
DELETE Object | implemented |
|
|
HEAD Object | implemented |
|
|
|
|
|
|
## Security
|
|
|
|
If security is not enabled, you can *use* **any** AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
|
|
|
|
If security is enabled, you can get the key and the secret with the `ozone s3 getsecret` command (*kerberos based authentication is required).
|
|
|
|
```bash
|
|
/etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
|
|
ozone s3 getsecret
|
|
awsAccessKey=testuser/scm@EXAMPLE.COM
|
|
awsSecret=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
|
|
|
|
```
|
|
|
|
Now, you can use the key and the secret to access the S3 endpoint:
|
|
|
|
```bash
|
|
export AWS_ACCESS_KEY_ID=testuser/scm@EXAMPLE.COM
|
|
export AWS_SECRET_ACCESS_KEY=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
|
|
aws s3api --endpoint http://localhost:9878 create-bucket --bucket bucket1
|
|
```
|
|
|
|
|
|
## S3 bucket name mapping to Ozone buckets
|
|
|
|
**Note**: Ozone has a notion for 'volumes' which is missing from the S3 Rest endpoint. Under the hood S3 bucket names are mapped to Ozone 'volume/bucket' locations (depending on the given authentication information).
|
|
|
|
To show the storage location of a S3 bucket, use the `ozone s3 path <bucketname>` command.
|
|
|
|
```bash
|
|
aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket=bucket1
|
|
|
|
ozone s3 path bucket1
|
|
Volume name for S3Bucket is : s3thisisakey
|
|
Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey
|
|
```
|
|
|
|
## Clients
|
|
|
|
### AWS Cli
|
|
|
|
`aws` CLI could be used by specifying the custom REST endpoint.
|
|
|
|
```bash
|
|
aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest
|
|
```
|
|
|
|
Or
|
|
|
|
```bash
|
|
aws s3 ls --endpoint http://localhost:9878 s3://buckettest
|
|
```
|
|
|
|
### S3 Fuse driver (goofys)
|
|
|
|
Goofys is a S3 FUSE driver. It could be used to mount any Ozone bucket as posix file system.
|
|
|
|
|
|
```bash
|
|
goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1
|
|
```
|