8fd17fe0af
fix a few typos in Kinesis doc ( #11776 )
2021-10-06 19:43:20 -07:00
104c9a07f0
Fix broken anchor and heading levels in Kafka/Kinesis ingestion ( #11748 )
...
* Fix broken anchor and heading levels
* Fix CI
2021-10-05 19:30:50 -07:00
3c4bba1478
Update kinesis-ingestion.md ( #11767 )
...
* Update kinesis-ingestion.md
It seems that we are declaring (a final int) recordsPerFetch as 400 and fetchDelayMillis as 0 in https://github.com/implydata/druid/blob/imply-2021.09/extensions-core/kinesis-indexing-service/src/main/java/org/apache/druid/indexing/kinesis/KinesisIndexTaskIOConfig.java#L36
```
public static final int DEFAULT_RECORDS_PER_FETCH = 4000;
public static final int DEFAULT_FETCH_DELAY_MILLIS = 0;
```
updating `recordsPerFetch` and `fetchDelayMillis` to actual default values as hardcoded above .
* Update docs/development/extensions-core/kinesis-ingestion.md
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2021-10-04 11:26:53 -07:00
6d14ea2d14
Dynamic auto scale Kinesis-Stream ingest tasks ( #10985 )
...
* ready to test
* revert misc.xml
* document kinesis md
* Update docs/development/extensions-core/kafka-ingestion.md
* Update docs/development/extensions-core/kinesis-ingestion.md
* Update docs/development/extensions-core/kinesis-ingestion.md
* Update docs/development/extensions-core/kinesis-ingestion.md
* Update docs/development/extensions-core/kinesis-ingestion.md
* Update docs/development/extensions-core/kinesis-ingestion.md
* Update docs/development/extensions-core/kinesis-ingestion.md
* Update docs/development/extensions-core/kinesis-ingestion.md
* Update docs/development/extensions-core/kinesis-ingestion.md
* Update docs/development/extensions-core/kinesis-ingestion.md
* Update docs/development/extensions-core/kinesis-ingestion.md
* Update kafka-ingestion.md
remove leading `
* Update kinesis-ingestion.md
add missing `
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2021-08-30 15:44:29 -07:00
9032a0b079
updates Kafka and Kinesis to use . Fixes some typos and other style i… ( #11624 )
...
* updates Kafka and Kinesis to use . Fixes some typos and other style issues for Kafka.
* fix spelling
* Update docs/development/extensions-core/kafka-ingestion.md
Co-authored-by: Jihoon Son <jihoonson@apache.org>
* Update docs/development/extensions-core/kafka-ingestion.md
Co-authored-by: Jihoon Son <jihoonson@apache.org>
* Update docs/development/extensions-core/kafka-ingestion.md
Co-authored-by: Jihoon Son <jihoonson@apache.org>
* Update docs/development/extensions-core/kafka-ingestion.md
Co-authored-by: Jihoon Son <jihoonson@apache.org>
* Update docs/development/extensions-core/kafka-ingestion.md
Co-authored-by: Jihoon Son <jihoonson@apache.org>
* Update docs/development/extensions-core/kinesis-ingestion.md
Co-authored-by: Jihoon Son <jihoonson@apache.org>
* Update docs/development/extensions-core/kinesis-ingestion.md
Co-authored-by: Jihoon Son <jihoonson@apache.org>
* Update docs/development/extensions-core/kafka-ingestion.md
Co-authored-by: Jihoon Son <jihoonson@apache.org>
* address comments
Co-authored-by: Jihoon Son <jihoonson@apache.org>
2021-08-26 13:22:30 -07:00
6524d838d7
Docs refactor of ingestion. Carries #11541 ( #11576 )
...
* Docs refactor of ingestion. Carries #11541
* Update docs/misc/math-expr.md
* add Apache license
* fix header, add topics to sidebar
* Update docs/ingestion/partitioning.md
* pick up changes to and md from c7fdf1d#11479
Co-authored-by: Suneet Saldanha <suneet@apache.org>
Co-authored-by: Jihoon Son <jihoonson@apache.org>
2021-08-13 08:42:03 -07:00
145cf9e5c3
fix document about input format ( #11342 )
...
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-06-08 23:44:54 +08:00
8544d29bc7
remove experimental from Kinesis with caveats ( #10998 )
...
* remove experimental from Kinesis with caveats
* add suggested known issue
* spelling fixes
2021-03-29 13:57:58 -07:00
96d26e5338
Fix kinesis ingestion bugs ( #10761 )
...
* add offsetFetchPeriod to kinesis ingestion doc
* Remove jackson dependencies from extensions
* Use fixed delay for lag collection
* Metrics reset after finishing processing
* comments
* Broaden the list of exceptions to retry for
* Unit tests
* Add more tests
* Refactoring
* re-order metrics
* Doc suggestions
Co-authored-by: Charles Smith <38529548+techdocsmith@users.noreply.github.com>
* Add tests
Co-authored-by: Charles Smith <38529548+techdocsmith@users.noreply.github.com>
2021-02-05 02:49:58 -08:00
1c1b396eaa
AWS Web Identity / IRSA Support ( #10541 )
...
* AWS Web Identity Support
required for AWS IRSA
* Update kinesis-ingestion.md
* disabling coverage tests
https://github.com/apache/druid/pull/10541#issuecomment-737558213
* exclude coverage
* Update licenses.yaml
2021-01-25 18:44:02 +05:30
f66fdbfa5d
add offsetFetchPeriod to kinesis ingestion doc ( #10734 )
2021-01-08 14:19:26 -08:00
6ae8059c09
cleaning up and fixing links ( #10528 )
...
* cleaning up and fixing links
* reverting local link
* Update indexer.md
* link checking
* Fixing one more stale link for PostgreSQL
2020-12-17 13:37:43 -08:00
b95bf444b2
add docs for kinesis lag metrics ( #10435 )
2020-09-28 13:13:53 -07:00
9587fc0b84
Fix documentation for Kinesis fetchThreads. ( #10156 )
...
* Fix documentation for Kinesis fetchThreads
The default was changed in #9819 , but the documentation wasn't updated.
* Add 'procs' to spelling.
2020-07-08 19:47:09 -07:00
ce03f31a73
Clarifying workerThreads and a few other nits ( #9804 )
...
* Update data-formats.md
Per Suneet, "Since you're editing this file can you also fix the json on line 177 please - it's missing a comma after the }"
* Light text cleanup
* Removing discussion of sample data, since it's repeated in the data loading tutorial, and not immediately relevant here.
* Clarifying accepted values for URI lookup
* Update index.md
* original quickstart full first pass
* original quickstart full first pass
* first pass all the way through
* straggler
* image touchups and finished old tutorial
* a bit of finishing up
* druid-caffeine-cache ext previously removed
* Sample MaxDirectMemorySize value unrealistic
* Review comments
* fixing links
* spell checking gymnastics
* workerThreads desc slightly expanded
* typo
* Typo
* Reversing Kafka config order
* Changing order of configs for Kinesis
* Trying this again: ioConfig then tuningConfig
2020-05-06 09:05:18 -07:00
85dfbb64cb
Update documention for metricCompression ( #9811 )
2020-05-03 12:56:48 -07:00
1852bf33ea
Add Integration Test for functionality of kinesis ingestion ( #9576 )
...
* kinesis IT
* Kinesis IT
* Kinesis IT
* Kinesis IT
* Kinesis IT
* Kinesis IT
* Kinesis IT
* Kinesis IT
* Kinesis IT
* Kinesis IT
* Kinesis IT
* Kinesis IT
* Kinesis IT
* Kinesis IT
* Kinesis IT
* fix kinesis timeout
* Kinesis IT
* Kinesis IT
* fix checkstyle
* Kinesis IT
* address comments
* fix checkstyle
2020-04-03 09:45:22 -07:00
bf85ea19b2
roaring bitmaps by default ( #9548 )
...
* it is finally time
* fix it
* more docs
* fix doc
2020-03-23 18:15:57 -07:00
556a3861ed
Make docs on reset supervisor operation scarier ( #9288 )
...
* Update kafka-ingestion.md
Companion doc update to #9253 , intended to make a supervisor reset scarier
* Update kinesis-ingestion.md
2020-02-04 15:30:31 -08:00
153495068b
Doc update for the new input source and the new input format ( #9171 )
...
* Doc update for new input source and input format.
- The input source and input format are promoted in all docs under docs/ingestion
- All input sources including core extension ones are located in docs/ingestion/native-batch.md
- All input formats and parsers including core extension ones are localted in docs/ingestion/data-formats.md
- New behavior of the parallel task with different partitionsSpecs are documented in docs/ingestion/native-batch.md
* parquet
* add warning for range partitioning with sequential mode
* hdfs + s3, gs
* add fs impl for gs
* address comments
* address comments
* gcs
2020-01-17 15:52:05 -08:00
d1500c1328
Update Kinesis resharding information about task failures ( #9104 )
2020-01-07 15:44:48 -08:00
aa539177ec
De-incubation cleanup in code, docs, packaging ( #9108 )
...
* De-incubation cleanup in code, docs, packaging
* remove unused docs script
2020-01-03 12:33:19 -05:00
00ce18a0ea
Additional Kinesis resharding fixes ( #8870 )
...
* Additional Kinesis resharding fixes
* Address PR comments
* Remove unused method
* Adjust SegmentTransactionalInsertAction null handling
* Check for unchanged metadata on empty publish
* Add logs for empty publish
* Fix javadoc
* Clear offset when invalid endOffsets are seen
* Fix LGTM alert
* Fix build
* Add resharding note to Kinesis docs
* Checkstyle
* Spelling
* Address PR comments
* Checkstyle
2019-11-28 12:59:01 -08:00
7605c23354
Remove Tranquility configs and certain doc references. ( #8793 )
...
Since it hasn't received updates or community interest in a while, it makes sense
to de-emphasize it in the distribution and most documentation (outside of simple
mentions of its existence).
2019-10-30 16:30:16 -07:00
89ce6384f5
More Kinesis resharding adjustments ( #8671 )
...
* More Kinesis resharding adjustments
* Fix TC inspection
* Fix comment'
* Adjust comment, small refactor
* Make repartition transition time configurable
* Add spellcheck exclusion
* Spelling fix
2019-10-15 23:19:17 -07:00
7dcbaca658
Spellcheck docs ( #8548 )
...
* Spellcheck docs
Fix spelling mistakes in docs and add CI job for running spellcheck on
docs.
* Add missing license header
2019-09-17 12:47:30 -07:00
d007477742
Docusaurus build framework + ingestion doc refresh. ( #8311 )
...
* Docusaurus build framework + ingestion doc refresh.
* stick to npm instead of yarn
* fix typos
* restore some _bin
* Adjustments.
* detect and fix redirect anchors
* update anchor lint
* Web-console: remove specific column filters (#8343 )
* add clear filter
* update tool kit
* remove usless check
* auto run
* add %
* Fix resource leak (#8337 )
* Fix resource leak
* Patch comments
* Enable Spotbugs NP_NONNULL_RETURN_VIOLATION (#8234 )
* Fixes from PR review.
* Fix more anchors.
* Preamble nix.
* Fix more anchors, headers
* clean up placeholder page
* add to website lint to travis config
* better broken link checking
* travis fix
* Fixed more broken links
* better redirects
* unfancy catch
* fix LGTM error
* link fixes
* fix md issues
* Addl fixes
2019-08-20 21:48:59 -07:00
Charles Smith
Frank Chen
Vaibhav
zhangyue19921010
Yi Yuan
Charles Smith
Abhishek Agarwal
Himadri Singh
sthetland
Clint Wylie
Gian Merlino
Jian Wang
Maytas Monsereenusorn
Jihoon Son
Jonathan Wei
Chi Cao Minh