mirror of
https://github.com/apache/druid.git
synced 2025-02-11 12:35:00 +00:00
d007477742
* Docusaurus build framework + ingestion doc refresh. * stick to npm instead of yarn * fix typos * restore some _bin * Adjustments. * detect and fix redirect anchors * update anchor lint * Web-console: remove specific column filters (#8343) * add clear filter * update tool kit * remove usless check * auto run * add % * Fix resource leak (#8337) * Fix resource leak * Patch comments * Enable Spotbugs NP_NONNULL_RETURN_VIOLATION (#8234) * Fixes from PR review. * Fix more anchors. * Preamble nix. * Fix more anchors, headers * clean up placeholder page * add to website lint to travis config * better broken link checking * travis fix * Fixed more broken links * better redirects * unfancy catch * fix LGTM error * link fixes * fix md issues * Addl fixes
49 lines
2.7 KiB
Markdown
49 lines
2.7 KiB
Markdown
---
|
|
id: insert-segment-to-db
|
|
title: "insert-segment-to-db tool"
|
|
---
|
|
|
|
<!--
|
|
~ Licensed to the Apache Software Foundation (ASF) under one
|
|
~ or more contributor license agreements. See the NOTICE file
|
|
~ distributed with this work for additional information
|
|
~ regarding copyright ownership. The ASF licenses this file
|
|
~ to you under the Apache License, Version 2.0 (the
|
|
~ "License"); you may not use this file except in compliance
|
|
~ with the License. You may obtain a copy of the License at
|
|
~
|
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
|
~
|
|
~ Unless required by applicable law or agreed to in writing,
|
|
~ software distributed under the License is distributed on an
|
|
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
~ KIND, either express or implied. See the License for the
|
|
~ specific language governing permissions and limitations
|
|
~ under the License.
|
|
-->
|
|
|
|
|
|
In older versions of Apache Druid (incubating), `insert-segment-to-db` was a tool that could scan deep storage and
|
|
insert data from there into Druid metadata storage. It was intended to be used to update the segment table in the
|
|
metadata storage after manually migrating segments from one place to another, or even to recover lost metadata storage
|
|
by telling it where the segments are stored.
|
|
|
|
In Druid 0.14.x and earlier, Druid wrote segment metadata to two places: the metadata store's `druid_segments` table, and
|
|
`descriptor.json` files in deep storage. This practice was stopped in Druid 0.15.0 as part of
|
|
[consolidated metadata management](https://github.com/apache/druid/issues/6849), for the following reasons:
|
|
|
|
1. If any segments are manually dropped or re-enabled by cluster operators, this information is not reflected in
|
|
deep storage. Restoring metadata from deep storage would undo any such drops or re-enables.
|
|
2. Ingestion methods that allocate segments optimistically (such as native Kafka or Kinesis stream ingestion, or native
|
|
batch ingestion in 'append' mode) can write segments to deep storage that are not meant to actually be used by the
|
|
Druid cluster. There is no way, while purely looking at deep storage, to differentiate the segments that made it into
|
|
the metadata store originally (and therefore _should_ be used) from the segments that did not (and therefore
|
|
_should not_ be used).
|
|
3. Nothing in Druid other than the `insert-segment-to-db` tool read the `descriptor.json` files.
|
|
|
|
After this change, Druid stopped writing `descriptor.json` files to deep storage, and now only writes segment metadata
|
|
to the metadata store. This meant the `insert-segment-to-db` tool is no longer useful, so it was removed in Druid 0.15.0.
|
|
|
|
It is highly recommended that you take regular backups of your metadata store, since it is difficult to recover Druid
|
|
clusters properly without it.
|