Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
druid/website
History
Kashif Faraz 9dc2569f22
Track and emit segment loading rate for HttpLoadQueuePeon on Coordinator (#16691) 
Design:
The loading rate is computed as a moving average of at least the last 10 GiB of successful segment loads.
To account for multiple loading threads on a server, we use the concept of a batch to track load times.
A batch is a set of segments added by the coordinator to the load queue of a server in one go.

Computation:
batchDurationMillis = t(load queue becomes empty) - t(first load request in batch is sent to server)
batchBytes = total bytes successfully loaded in batch
avg loading rate in batch (kbps) = (8 * batchBytes) / batchDurationMillis
overall avg loading rate (kbps) = (8 * sumOverWindow(batchBytes)) / sumOverWindow(batchDurationMillis)

Changes:
- Add `LoadingRateTracker` which computes a moving average load rate based on
the last few GBs of successful segment loads.
- Emit metric `segment/loading/rateKbps` from the Coordinator. In the future, we may
also consider emitting this metric from the historicals themselves.
- Add `expectedLoadTimeMillis` to response of API `/druid/coordinator/v1/loadQueue?simple`
 		2024-08-03 13:14:21 +05:30
..
script docs: Anchor link checker (#15624) 2024-01-08 15:19:05 -08:00
src Docusaurus2 upgrade for master (#14411) 2023-08-16 19:01:21 -07:00
static fix: Creating span label not closed (#15323) 2023-11-07 11:01:28 +08:00
.spelling Track and emit segment loading rate for HttpLoadQueuePeon on Coordinator (#16691) 2024-08-03 13:14:21 +05:30
README.md Docusaurus2 upgrade for master (#14411) 2023-08-16 19:01:21 -07:00
docusaurus.config.js docs: add mermaid diagram support (#15771) 2024-01-30 11:24:15 -08:00
package-lock.json docs: Update `caniuse-lite` (#16137) 2024-03-15 15:58:16 -07:00
package.json docs: update package and regenerate package-lock (#16038) 2024-03-05 14:56:43 +05:30
redirects.js docs: add redirect for kafka lookups (#16668) 2024-06-27 10:56:51 -07:00
sidebars.json doc: minor fixes to migration guides (#16784) 2024-07-23 13:09:51 -07:00

README.md

Druid doc builder

This website was created with Docusaurus.

To view documentation run:

npm install

Then run:

npm start

The current version of the web site appears in your browser. Edit pages with your favorite editor. Refresh the web page after each edit to review your changes.

Dependencies

  • NodeJS. Use the version Docusaurus specifies, not a newer one. (For example, if 12.x is requested, don't install 16.x.) Docusaurus may require a version newer than that available in your Linux package repository, but older than the latest version. See this page to find the version required by Docusaurus.
  • The Yarn dependency from Docusaurus is optional. (This Yarn is not the Hadoop resource manager, it is a package manager for Node.js).
  • Docusaurus. Installed automatically as part of the the above npm commands.

Variables

Documentation pages can refer to a number of special variables using the {{var}} syntax:

  • DRUIDVERSION - the version of Druid in which the page appears. Allows creating links to files of the same version on GitHub.

The variables are not replaced when running the web site locally using the start command above. They're replaced as part of the process that copies the docs to apache/druid-website-src/.

Spellcheck

Please run a spellcheck before issuing a pull request to avoid a build failure due to spelling issues. Run:

npm run link-lint
npm run spellcheck

If you introduce new (correctly spelled) project names or technical terms, add them to the dictionary in the .spelling file in this directory. Also, terms enclosed in backticks are not spell checked. Example: `symbolName`