Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
druid/docs/design/deep-storage.md

77 lines
3.3 KiB
Markdown
Raw Normal View History

Front Matter header needs to be on the first line for md to be rendered properly by jekyll (#6733)
2018-12-13 14:47:20 -05:00
---
Docusaurus build framework + ingestion doc refresh. (#8311) * Docusaurus build framework + ingestion doc refresh. * stick to npm instead of yarn * fix typos * restore some _bin * Adjustments. * detect and fix redirect anchors * update anchor lint * Web-console: remove specific column filters (#8343) * add clear filter * update tool kit * remove usless check * auto run * add % * Fix resource leak (#8337) * Fix resource leak * Patch comments * Enable Spotbugs NP_NONNULL_RETURN_VIOLATION (#8234) * Fixes from PR review. * Fix more anchors. * Preamble nix. * Fix more anchors, headers * clean up placeholder page * add to website lint to travis config * better broken link checking * travis fix * Fixed more broken links * better redirects * unfancy catch * fix LGTM error * link fixes * fix md issues * Addl fixes
2019-08-21 00:48:59 -04:00
id: deep-storage
title: "Deep storage"
Front Matter header needs to be on the first line for md to be rendered properly by jekyll (#6733)
2018-12-13 14:47:20 -05:00
---
add missing license headers, in particular to MD files; clean up RAT … (#6563) * add missing license headers, in particular to MD files; clean up RAT exclusions * revert inadvertent doc changes * docs * cr changes * fix modified druid-production.svg
2018-11-13 12:38:37 -05:00
<!--
~ Licensed to the Apache Software Foundation (ASF) under one
~ or more contributor license agreements. See the NOTICE file
~ distributed with this work for additional information
~ regarding copyright ownership. The ASF licenses this file
~ to you under the Apache License, Version 2.0 (the
~ "License"); you may not use this file except in compliance
~ with the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing,
~ software distributed under the License is distributed on an
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
~ KIND, either express or implied. See the License for the
~ specific language governing permissions and limitations
~ under the License.
-->
refactor extensions into their own docs
2016-03-22 16:54:49 -04:00
De-incubation cleanup in code, docs, packaging (#9108) * De-incubation cleanup in code, docs, packaging * remove unused docs script
2020-01-03 12:33:19 -05:00
Deep storage is where segments are stored. It is a storage mechanism that Apache Druid does not provide. This deep storage infrastructure defines the level of durability of your data, as long as Druid processes can see this storage infrastructure and get at the segments stored on it, you will not lose data no matter how many Druid nodes you lose. If segments disappear from this storage layer, then you will lose whatever data those segments represented.
renaming all *.md filenames to only have lowercase and dashes so that they are editable on case-insensitive os as well
2015-05-05 17:07:32 -04:00
Zero-copy local deep storage. (#13394) * Zero-copy local deep storage. This is useful for local deep storage, since it reduces disk usage and makes Historicals able to load segments instantaneously. Two changes: 1) Introduce "druid.storage.zip" parameter for local storage, which defaults to false. This changes default behavior from writing an index.zip to writing a regular directory. This is safe to do even during a rolling update, because the older code actually already handled unzipped directories being present on local deep storage. 2) In LocalDataSegmentPuller and LocalDataSegmentPusher, use hard links instead of copies when possible. (Generally this is possible when the source and destination directory are on the same filesystem.)
2022-12-12 20:28:24 -05:00
## Local
renaming all *.md filenames to only have lowercase and dashes so that they are editable on case-insensitive os as well
2015-05-05 17:07:32 -04:00
Zero-copy local deep storage. (#13394) * Zero-copy local deep storage. This is useful for local deep storage, since it reduces disk usage and makes Historicals able to load segments instantaneously. Two changes: 1) Introduce "druid.storage.zip" parameter for local storage, which defaults to false. This changes default behavior from writing an index.zip to writing a regular directory. This is safe to do even during a rolling update, because the older code actually already handled unzipped directories being present on local deep storage. 2) In LocalDataSegmentPuller and LocalDataSegmentPusher, use hard links instead of copies when possible. (Generally this is possible when the source and destination directory are on the same filesystem.)
2022-12-12 20:28:24 -05:00
Local storage is intended for use in the following situations:
renaming all *.md filenames to only have lowercase and dashes so that they are editable on case-insensitive os as well
2015-05-05 17:07:32 -04:00
Zero-copy local deep storage. (#13394) * Zero-copy local deep storage. This is useful for local deep storage, since it reduces disk usage and makes Historicals able to load segments instantaneously. Two changes: 1) Introduce "druid.storage.zip" parameter for local storage, which defaults to false. This changes default behavior from writing an index.zip to writing a regular directory. This is safe to do even during a rolling update, because the older code actually already handled unzipped directories being present on local deep storage. 2) In LocalDataSegmentPuller and LocalDataSegmentPusher, use hard links instead of copies when possible. (Generally this is possible when the source and destination directory are on the same filesystem.)
2022-12-12 20:28:24 -05:00
- You have just one server.
- Or, you have multiple servers, and they all have access to a shared filesystem (for example: NFS).
In multi-server production clusters, rather than local storage with a shared filesystem, it is instead recommended to
use cloud-based deep storage ([Amazon S3](#amazon-s3-or-s3-compatible), [Google Cloud Storage](#google-cloud-storage),
or [Azure Blob Storage](#azure-blob-storage)), S3-compatible storage (like Minio), or [HDFS](#hdfs). These options are
generally more convenient, more scalable, and more robust than setting up a shared filesystem.
The following configurations in `common.runtime.properties` apply to local storage:
renaming all *.md filenames to only have lowercase and dashes so that they are editable on case-insensitive os as well
2015-05-05 17:07:32 -04:00
|Property|Possible Values|Description|Default|
|--------|---------------|-----------|-------|
Zero-copy local deep storage. (#13394) * Zero-copy local deep storage. This is useful for local deep storage, since it reduces disk usage and makes Historicals able to load segments instantaneously. Two changes: 1) Introduce "druid.storage.zip" parameter for local storage, which defaults to false. This changes default behavior from writing an index.zip to writing a regular directory. This is safe to do even during a rolling update, because the older code actually already handled unzipped directories being present on local deep storage. 2) In LocalDataSegmentPuller and LocalDataSegmentPusher, use hard links instead of copies when possible. (Generally this is possible when the source and destination directory are on the same filesystem.)
2022-12-12 20:28:24 -05:00
|`druid.storage.type`|`local`||Must be set.|
|`druid.storage.storageDirectory`|any local directory|Directory for storing segments. Must be different from `druid.segmentCache.locations` and `druid.segmentCache.infoDir`.|`/tmp/druid/localStorage`|
|`druid.storage.zip`|`true`, `false`|Whether segments in `druid.storage.storageDirectory` are written as directories (`false`) or zip files (`true`).|`false`|
For example:
```
druid.storage.type=local
druid.storage.storageDirectory=/tmp/druid/localStorage
```
The `druid.storage.storageDirectory` must be set to a different path than `druid.segmentCache.locations` or
`druid.segmentCache.infoDir`.
## Amazon S3 or S3-compatible
See [`druid-s3-extensions`](../development/extensions-core/s3.md).
renaming all *.md filenames to only have lowercase and dashes so that they are editable on case-insensitive os as well
2015-05-05 17:07:32 -04:00
Zero-copy local deep storage. (#13394) * Zero-copy local deep storage. This is useful for local deep storage, since it reduces disk usage and makes Historicals able to load segments instantaneously. Two changes: 1) Introduce "druid.storage.zip" parameter for local storage, which defaults to false. This changes default behavior from writing an index.zip to writing a regular directory. This is safe to do even during a rolling update, because the older code actually already handled unzipped directories being present on local deep storage. 2) In LocalDataSegmentPuller and LocalDataSegmentPusher, use hard links instead of copies when possible. (Generally this is possible when the source and destination directory are on the same filesystem.)
2022-12-12 20:28:24 -05:00
## Google Cloud Storage
renaming all *.md filenames to only have lowercase and dashes so that they are editable on case-insensitive os as well
2015-05-05 17:07:32 -04:00
Zero-copy local deep storage. (#13394) * Zero-copy local deep storage. This is useful for local deep storage, since it reduces disk usage and makes Historicals able to load segments instantaneously. Two changes: 1) Introduce "druid.storage.zip" parameter for local storage, which defaults to false. This changes default behavior from writing an index.zip to writing a regular directory. This is safe to do even during a rolling update, because the older code actually already handled unzipped directories being present on local deep storage. 2) In LocalDataSegmentPuller and LocalDataSegmentPusher, use hard links instead of copies when possible. (Generally this is possible when the source and destination directory are on the same filesystem.)
2022-12-12 20:28:24 -05:00
See [`druid-google-extensions`](../development/extensions-core/google.md).
renaming all *.md filenames to only have lowercase and dashes so that they are editable on case-insensitive os as well
2015-05-05 17:07:32 -04:00
Zero-copy local deep storage. (#13394) * Zero-copy local deep storage. This is useful for local deep storage, since it reduces disk usage and makes Historicals able to load segments instantaneously. Two changes: 1) Introduce "druid.storage.zip" parameter for local storage, which defaults to false. This changes default behavior from writing an index.zip to writing a regular directory. This is safe to do even during a rolling update, because the older code actually already handled unzipped directories being present on local deep storage. 2) In LocalDataSegmentPuller and LocalDataSegmentPusher, use hard links instead of copies when possible. (Generally this is possible when the source and destination directory are on the same filesystem.)
2022-12-12 20:28:24 -05:00
## Azure Blob Storage
renaming all *.md filenames to only have lowercase and dashes so that they are editable on case-insensitive os as well
2015-05-05 17:07:32 -04:00
Zero-copy local deep storage. (#13394) * Zero-copy local deep storage. This is useful for local deep storage, since it reduces disk usage and makes Historicals able to load segments instantaneously. Two changes: 1) Introduce "druid.storage.zip" parameter for local storage, which defaults to false. This changes default behavior from writing an index.zip to writing a regular directory. This is safe to do even during a rolling update, because the older code actually already handled unzipped directories being present on local deep storage. 2) In LocalDataSegmentPuller and LocalDataSegmentPusher, use hard links instead of copies when possible. (Generally this is possible when the source and destination directory are on the same filesystem.)
2022-12-12 20:28:24 -05:00
See [`druid-azure-extensions`](../development/extensions-core/azure.md).
renaming all *.md filenames to only have lowercase and dashes so that they are editable on case-insensitive os as well
2015-05-05 17:07:32 -04:00
refactor extensions into core and contrib
2016-02-29 20:43:35 -05:00
## HDFS
renaming all *.md filenames to only have lowercase and dashes so that they are editable on case-insensitive os as well
2015-05-05 17:07:32 -04:00
Docusaurus build framework + ingestion doc refresh. (#8311) * Docusaurus build framework + ingestion doc refresh. * stick to npm instead of yarn * fix typos * restore some _bin * Adjustments. * detect and fix redirect anchors * update anchor lint * Web-console: remove specific column filters (#8343) * add clear filter * update tool kit * remove usless check * auto run * add % * Fix resource leak (#8337) * Fix resource leak * Patch comments * Enable Spotbugs NP_NONNULL_RETURN_VIOLATION (#8234) * Fixes from PR review. * Fix more anchors. * Preamble nix. * Fix more anchors, headers * clean up placeholder page * add to website lint to travis config * better broken link checking * travis fix * Fixed more broken links * better redirects * unfancy catch * fix LGTM error * link fixes * fix md issues * Addl fixes
2019-08-21 00:48:59 -04:00
See [druid-hdfs-storage extension documentation](../development/extensions-core/hdfs.md).
renaming all *.md filenames to only have lowercase and dashes so that they are editable on case-insensitive os as well
2015-05-05 17:07:32 -04:00
Zero-copy local deep storage. (#13394) * Zero-copy local deep storage. This is useful for local deep storage, since it reduces disk usage and makes Historicals able to load segments instantaneously. Two changes: 1) Introduce "druid.storage.zip" parameter for local storage, which defaults to false. This changes default behavior from writing an index.zip to writing a regular directory. This is safe to do even during a rolling update, because the older code actually already handled unzipped directories being present on local deep storage. 2) In LocalDataSegmentPuller and LocalDataSegmentPusher, use hard links instead of copies when possible. (Generally this is possible when the source and destination directory are on the same filesystem.)
2022-12-12 20:28:24 -05:00
## Additional options
renaming all *.md filenames to only have lowercase and dashes so that they are editable on case-insensitive os as well
2015-05-05 17:07:32 -04:00
Update Ingestion section (#14023) Co-authored-by: Charles Smith <techdocsmith@gmail.com> Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> Co-authored-by: Victoria Lim <lim.t.victoria@gmail.com>
2023-05-19 12:42:27 -04:00
For additional deep storage options, please see our [extensions list](../configuration/extensions.md).