hadoop/hadoop-hdds/docs/content/S3.md

title	menu
S3	main parent weight Client 1

Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools.

Getting started

S3 Gateway is a separated component which provides the S3 compatible. It should be started additional to the regular Ozone components.

You can start a docker based cluster, including the S3 gateway from the release package.

Go to the compose/ozones3 directory, and start the server:

docker-compose up -d

You can access the S3 gateway at http://localhost:9878

URL Schema

Ozone S3 gateway supports both the virtual-host-style URL s3 bucket addresses (eg. http://bucketname.host:9878) and the path-style addresses (eg. http://host:9878/bucketname)

By default it uses the path-style addressing. To use virtual host style URLs set your main domain name in your ozone-site.xml:

<property>
   <name>ozone.s3g.domain.name</name>
   <value>s3g.internal</value>
</property>

Bucket browser

Buckets could be browsed from the browser by adding ?browser=true to the bucket URL.

For example the content of the 'testbucket' could be checked from the browser using the URL http://localhost:9878/testbucket?browser=true

Implemented REST endpoints

Operations on S3Gateway service:

Endpoint	Status
GET service	implemented

Operations on Bucket:

Endpoint	Status	Notes
GET Bucket (List Objects) Version 2	implemented
HEAD Bucket	implemented
DELETE Bucket	implemented
PUT Bucket (Create bucket)	implemented
Delete Multiple Objects (POST)	implemented

Operation on Objects:

Endpoint	Status	Notes
PUT Object	implemented
GET Object	implemented
Multipart Upload	implemented	Except the listing of the current MultiPartUploads.
DELETE Object	implemented
HEAD Object	implemented

Security

If security is not enabled, you can use any AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

If security is enabled, you can get the key and the secret with the ozone s3 getsecret command (*kerberos based authentication is required).

/etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
ozone s3 getsecret
awsAccessKey=testuser/scm@EXAMPLE.COM
awsSecret=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999

Now, you can use the key and the secret to access the S3 endpoint:

export AWS_ACCESS_KEY_ID=testuser/scm@EXAMPLE.COM
export AWS_SECRET_ACCESS_KEY=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
aws s3api --endpoint http://localhost:9878 create-bucket --bucket bucket1

S3 bucket name mapping to Ozone buckets

Note: Ozone has a notion for 'volumes' which is missing from the S3 Rest endpoint. Under the hood S3 bucket names are mapped to Ozone 'volume/bucket' locations (depending on the given authentication information).

To show the storage location of a S3 bucket, use the ozone s3 path <bucketname> command.

aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket=bucket1

ozone sh bucket path bucket1
Volume name for S3Bucket is : s3thisisakey
Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey

Clients

AWS Cli

aws CLI could be used with specifying the custom REST endpoint.

aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest

Or

aws s3 ls --endpoint http://localhost:9878 s3://buckettest

S3 Fuse driver (goofys)

Goofys is a S3 FUSE driver. It could be used to mount any Ozone bucket as posix file system:

goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1