druid/docs
Agustin Gonzalez abe76ccb90
Batch ingestion replace (#12137)
* Tombstone support for replace functionality

* A used segment interval is the interval of a current used segment that overlaps any of the input intervals for the spec

* Update compaction test to match replace behavior

* Adapt ITAutoCompactionTest to work with tombstones rather than dropping segments. Add support for tombstones in the broker.

* Style plus simple queriableindex test

* Add segment cache loader tombstone test

* Add more tests

* Add a method to the LogicalSegment to test whether it has any data

* Test filter with some empty logical segments

* Refactor more compaction/dropexisting tests

* Code coverage

* Support for all empty segments

* Skip tombstones when looking-up broker's timeline. Discard changes made to tool chest to avoid empty segments since they will no longer have empty segments after lookup because we are skipping over them.

* Fix null ptr when segment does not have a queriable index

* Add support for empty replace interval (all input data has been filtered out)

* Fixed coverage & style

* Find tombstone versions from lock versions

* Test failures & style

* Interner was making this fail since the two segments were consider equal due to their id's being equal

* Cleanup tombstone version code

* Force timeChunkLock whenever replace (i.e. dropExisting=true) is being used

* Reject replace spec when input intervals are empty

* Documentation

* Style and unit test

* Restore test code deleted by mistake

* Allocate forces TIME_CHUNK locking and uses lock versions. TombstoneShardSpec added.

* Unused imports. Dead code. Test coverage.

* Coverage.

* Prevent killer from throwing an exception for tombstones. This is the killer used in the peon for killing segments.

* Fix OmniKiller + more test coverage.

* Tombstones are now marked using a shard spec

* Drop a segment factory.json in the segment cache for tombstones

* Style

* Style + coverage

* style

* Add TombstoneLoadSpec.class to mapper in test

* Update core/src/main/java/org/apache/druid/segment/loading/TombstoneLoadSpec.java

Typo

Co-authored-by: Jonathan Wei <jon-wei@users.noreply.github.com>

* Update docs/configuration/index.md

Missing

Co-authored-by: Jonathan Wei <jon-wei@users.noreply.github.com>

* Typo

* Integrated replace with an existing test since the replace part was redundant and more importantly, the test file was very close or exceeding the 10 min default "no output" CI Travis threshold.

* Range does not work with multi-dim

Co-authored-by: Jonathan Wei <jon-wei@users.noreply.github.com>
2022-03-08 20:07:02 -07:00
..
_bin De-incubation cleanup in code, docs, packaging (#9108) 2020-01-03 12:33:19 -05:00
assets Design/architecture doc touchups (#11762) 2021-10-04 11:09:35 -07:00
comparisons Update druid-vs-kudu.md (#11470) 2021-07-21 22:58:14 +08:00
configuration Batch ingestion replace (#12137) 2022-03-08 20:07:02 -07:00
dependencies Doc updates for metadata cleanup and storage (#12190) 2022-01-27 11:40:54 -08:00
design Allow coordinator run auto compaction duty period to be configured separately from other indexing duties (#12263) 2022-02-18 23:02:57 -08:00
development Adding Shared Access resource support for azure (#12266) 2022-02-22 18:27:43 +05:30
ingestion Batch ingestion replace (#12137) 2022-03-08 20:07:02 -07:00
misc Refactor SQL docs (#12239) 2022-02-11 14:43:30 -08:00
operations add a new query laning metrics to visualize lane assignment (#12111) 2022-03-04 15:21:17 +08:00
querying GroupBy: Cap dictionary-building selector memory usage. (#12309) 2022-03-08 13:13:11 -08:00
tutorials Officially support Java 11. (#12232) 2022-03-04 14:15:45 -08:00