mirror of https://github.com/apache/druid.git
188 lines
8.6 KiB
Markdown
188 lines
8.6 KiB
Markdown
---
|
|
id: tutorial-delete-data
|
|
title: "Tutorial: Deleting data"
|
|
sidebar_label: "Deleting data"
|
|
---
|
|
|
|
<!--
|
|
~ Licensed to the Apache Software Foundation (ASF) under one
|
|
~ or more contributor license agreements. See the NOTICE file
|
|
~ distributed with this work for additional information
|
|
~ regarding copyright ownership. The ASF licenses this file
|
|
~ to you under the Apache License, Version 2.0 (the
|
|
~ "License"); you may not use this file except in compliance
|
|
~ with the License. You may obtain a copy of the License at
|
|
~
|
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
|
~
|
|
~ Unless required by applicable law or agreed to in writing,
|
|
~ software distributed under the License is distributed on an
|
|
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
~ KIND, either express or implied. See the License for the
|
|
~ specific language governing permissions and limitations
|
|
~ under the License.
|
|
-->
|
|
|
|
|
|
This tutorial demonstrates how to delete existing data.
|
|
|
|
This tutorial requires the following:
|
|
* A running Apache Druid instance. If you don't have Druid, see the [single-machine quickstart](index.md) to get started.
|
|
* The command-line JSON processor, [jq](https://stedolan.github.io/jq/download/).
|
|
|
|
## Load initial data
|
|
|
|
In this tutorial, we will use the Wikipedia edits data, with an indexing spec that creates hourly segments. This spec is located at `quickstart/tutorial/deletion-index.json`, and it creates a datasource called `deletion-tutorial`.
|
|
|
|
Let's load this initial data:
|
|
|
|
```bash
|
|
bin/post-index-task --file quickstart/tutorial/deletion-index.json --url http://localhost:8081
|
|
```
|
|
|
|
When the load finishes, open [http://localhost:8888/unified-console.md#datasources](http://localhost:8888/unified-console.html#datasources) in a browser.
|
|
|
|
## How to permanently delete data
|
|
|
|
Permanent deletion of a Druid segment has two steps:
|
|
|
|
1. The segment must first be marked as "unused". This occurs when a user manually disables a segment through the Coordinator API.
|
|
2. After segments have been marked as "unused", a Kill Task will delete any "unused" segments from Druid's metadata store as well as deep storage.
|
|
|
|
Let's drop some segments now, by using the coordinator API to drop data by interval and segmentIds.
|
|
|
|
## Disable segments by interval
|
|
|
|
Let's disable segments in a specified interval. This will mark all segments in the interval as "unused", but not remove them from deep storage.
|
|
Let's disable segments in interval `2015-09-12T18:00:00.000Z/2015-09-12T20:00:00.000Z` i.e. between hour 18 and 20.
|
|
|
|
```bash
|
|
curl -X 'POST' -H 'Content-Type:application/json' -d '{ "interval" : "2015-09-12T18:00:00.000Z/2015-09-12T20:00:00.000Z" }' http://localhost:8081/druid/coordinator/v1/datasources/deletion-tutorial/markUnused
|
|
```
|
|
|
|
When the request completes, the Segments view of the web console no longer displays the segments for hours 18 and 19.
|
|
|
|
|
|
Note that the hour 18 and 19 segments are still present in deep storage:
|
|
|
|
```bash
|
|
$ ls -l1 var/druid/segments/deletion-tutorial/
|
|
2015-09-12T00:00:00.000Z_2015-09-12T01:00:00.000Z
|
|
2015-09-12T01:00:00.000Z_2015-09-12T02:00:00.000Z
|
|
2015-09-12T02:00:00.000Z_2015-09-12T03:00:00.000Z
|
|
2015-09-12T03:00:00.000Z_2015-09-12T04:00:00.000Z
|
|
2015-09-12T04:00:00.000Z_2015-09-12T05:00:00.000Z
|
|
2015-09-12T05:00:00.000Z_2015-09-12T06:00:00.000Z
|
|
2015-09-12T06:00:00.000Z_2015-09-12T07:00:00.000Z
|
|
2015-09-12T07:00:00.000Z_2015-09-12T08:00:00.000Z
|
|
2015-09-12T08:00:00.000Z_2015-09-12T09:00:00.000Z
|
|
2015-09-12T09:00:00.000Z_2015-09-12T10:00:00.000Z
|
|
2015-09-12T10:00:00.000Z_2015-09-12T11:00:00.000Z
|
|
2015-09-12T11:00:00.000Z_2015-09-12T12:00:00.000Z
|
|
2015-09-12T12:00:00.000Z_2015-09-12T13:00:00.000Z
|
|
2015-09-12T13:00:00.000Z_2015-09-12T14:00:00.000Z
|
|
2015-09-12T14:00:00.000Z_2015-09-12T15:00:00.000Z
|
|
2015-09-12T15:00:00.000Z_2015-09-12T16:00:00.000Z
|
|
2015-09-12T16:00:00.000Z_2015-09-12T17:00:00.000Z
|
|
2015-09-12T17:00:00.000Z_2015-09-12T18:00:00.000Z
|
|
2015-09-12T18:00:00.000Z_2015-09-12T19:00:00.000Z
|
|
2015-09-12T19:00:00.000Z_2015-09-12T20:00:00.000Z
|
|
2015-09-12T20:00:00.000Z_2015-09-12T21:00:00.000Z
|
|
2015-09-12T21:00:00.000Z_2015-09-12T22:00:00.000Z
|
|
2015-09-12T22:00:00.000Z_2015-09-12T23:00:00.000Z
|
|
2015-09-12T23:00:00.000Z_2015-09-13T00:00:00.000Z
|
|
```
|
|
|
|
## Disable segments by segment IDs
|
|
|
|
Let's disable some segments by their segmentID. This will again mark the segments as "unused", but not remove them from deep storage. You can see the full segmentID for a segment using the web console.
|
|
|
|
In the [segments view](http://localhost:8888/unified-console.html#segments), click one of the segment rows to open the segment metadata dialog:
|
|
|
|
|
|
|
|
The `identifier` field in the metadata dialog shows the full segment ID. For example, the hour 23 segment has segment ID `deletion-tutorial_2015-09-12T23:00:00.000Z_2015-09-13T00:00:00.000Z_2023-05-16T00:04:12.091Z`.
|
|
|
|
Disable the last two segments, hour 22 and 23 segments, by sending a POST request to the Coordinator with the corresponding segment IDs.
|
|
The following command queries the Coordinator for segment IDs and uses `jq` to parse and extract the IDs of the last two segments.
|
|
The segment IDs are stored in an environment variable named `unusedSegmentIds`.
|
|
```bash
|
|
unusedSegmentIds=$(curl -X 'GET' -H 'Content-Type:application/json' http://localhost:8081/druid/coordinator/v1/datasources/deletion-tutorial/segments | jq '.[-2:]')
|
|
```
|
|
|
|
The following request marks the segments unused:
|
|
```bash
|
|
curl -X 'POST' -H 'Content-Type:application/json' -d "{\"segmentIds\": $unusedSegmentIds}" http://localhost:8081/druid/coordinator/v1/datasources/deletion-tutorial/markUnused
|
|
```
|
|
|
|
When the request completes, the Segments view of the web console no longer displays the segments for hours 22 and 23.
|
|
|
|
|
|
|
|
Note that the hour 22 and 23 segments are still in deep storage:
|
|
|
|
```bash
|
|
$ ls -l1 var/druid/segments/deletion-tutorial/
|
|
2015-09-12T00:00:00.000Z_2015-09-12T01:00:00.000Z
|
|
2015-09-12T01:00:00.000Z_2015-09-12T02:00:00.000Z
|
|
2015-09-12T02:00:00.000Z_2015-09-12T03:00:00.000Z
|
|
2015-09-12T03:00:00.000Z_2015-09-12T04:00:00.000Z
|
|
2015-09-12T04:00:00.000Z_2015-09-12T05:00:00.000Z
|
|
2015-09-12T05:00:00.000Z_2015-09-12T06:00:00.000Z
|
|
2015-09-12T06:00:00.000Z_2015-09-12T07:00:00.000Z
|
|
2015-09-12T07:00:00.000Z_2015-09-12T08:00:00.000Z
|
|
2015-09-12T08:00:00.000Z_2015-09-12T09:00:00.000Z
|
|
2015-09-12T09:00:00.000Z_2015-09-12T10:00:00.000Z
|
|
2015-09-12T10:00:00.000Z_2015-09-12T11:00:00.000Z
|
|
2015-09-12T11:00:00.000Z_2015-09-12T12:00:00.000Z
|
|
2015-09-12T12:00:00.000Z_2015-09-12T13:00:00.000Z
|
|
2015-09-12T13:00:00.000Z_2015-09-12T14:00:00.000Z
|
|
2015-09-12T14:00:00.000Z_2015-09-12T15:00:00.000Z
|
|
2015-09-12T15:00:00.000Z_2015-09-12T16:00:00.000Z
|
|
2015-09-12T16:00:00.000Z_2015-09-12T17:00:00.000Z
|
|
2015-09-12T17:00:00.000Z_2015-09-12T18:00:00.000Z
|
|
2015-09-12T18:00:00.000Z_2015-09-12T19:00:00.000Z
|
|
2015-09-12T19:00:00.000Z_2015-09-12T20:00:00.000Z
|
|
2015-09-12T20:00:00.000Z_2015-09-12T21:00:00.000Z
|
|
2015-09-12T21:00:00.000Z_2015-09-12T22:00:00.000Z
|
|
2015-09-12T22:00:00.000Z_2015-09-12T23:00:00.000Z
|
|
2015-09-12T23:00:00.000Z_2015-09-13T00:00:00.000Z
|
|
```
|
|
|
|
## Run a kill task
|
|
|
|
Now that we have disabled some segments, we can submit a Kill Task, which will delete the disabled segments from metadata and deep storage.
|
|
|
|
A Kill Task spec has been provided at `quickstart/tutorial/deletion-kill.json`. Submit this task to the Overlord with the following command:
|
|
|
|
```bash
|
|
curl -X 'POST' -H 'Content-Type:application/json' -d @quickstart/tutorial/deletion-kill.json http://localhost:8081/druid/indexer/v1/task
|
|
```
|
|
|
|
When the task finishes, note that Druid deleted the disabled segments from deep storage.
|
|
|
|
|
|
```bash
|
|
$ ls -l1 var/druid/segments/deletion-tutorial/
|
|
2015-09-12T00:00:00.000Z_2015-09-12T01:00:00.000Z
|
|
2015-09-12T01:00:00.000Z_2015-09-12T02:00:00.000Z
|
|
2015-09-12T02:00:00.000Z_2015-09-12T03:00:00.000Z
|
|
2015-09-12T03:00:00.000Z_2015-09-12T04:00:00.000Z
|
|
2015-09-12T04:00:00.000Z_2015-09-12T05:00:00.000Z
|
|
2015-09-12T05:00:00.000Z_2015-09-12T06:00:00.000Z
|
|
2015-09-12T06:00:00.000Z_2015-09-12T07:00:00.000Z
|
|
2015-09-12T07:00:00.000Z_2015-09-12T08:00:00.000Z
|
|
2015-09-12T08:00:00.000Z_2015-09-12T09:00:00.000Z
|
|
2015-09-12T09:00:00.000Z_2015-09-12T10:00:00.000Z
|
|
2015-09-12T10:00:00.000Z_2015-09-12T11:00:00.000Z
|
|
2015-09-12T11:00:00.000Z_2015-09-12T12:00:00.000Z
|
|
2015-09-12T12:00:00.000Z_2015-09-12T13:00:00.000Z
|
|
2015-09-12T13:00:00.000Z_2015-09-12T14:00:00.000Z
|
|
2015-09-12T14:00:00.000Z_2015-09-12T15:00:00.000Z
|
|
2015-09-12T15:00:00.000Z_2015-09-12T16:00:00.000Z
|
|
2015-09-12T16:00:00.000Z_2015-09-12T17:00:00.000Z
|
|
2015-09-12T17:00:00.000Z_2015-09-12T18:00:00.000Z
|
|
2015-09-12T20:00:00.000Z_2015-09-12T21:00:00.000Z
|
|
2015-09-12T21:00:00.000Z_2015-09-12T22:00:00.000Z
|
|
```
|