94 lines
3.8 KiB
Plaintext
94 lines
3.8 KiB
Plaintext
[[index-modules-translog]]
|
|
== Translog
|
|
|
|
Changes to Lucene are only persisted to disk during a Lucene commit,
|
|
which is a relatively heavy operation and so cannot be performed after every
|
|
index or delete operation. Changes that happen after one commit and before another
|
|
will be lost in the event of process exit or HW failure.
|
|
|
|
To prevent this data loss, each shard has a _transaction log_ or write ahead
|
|
log associated with it. Any index or delete operation is written to the
|
|
translog after being processed by the internal Lucene index.
|
|
|
|
In the event of a crash, recent transactions can be replayed from the
|
|
transaction log when the shard recovers.
|
|
|
|
An Elasticsearch flush is the process of performing a Lucene commit and
|
|
starting a new translog. It is done automatically in the background in order
|
|
to make sure the transaction log doesn't grow too large, which would make
|
|
replaying its operations take a considerable amount of time during recovery.
|
|
It is also exposed through an API, though its rarely needed to be performed
|
|
manually.
|
|
|
|
|
|
[float]
|
|
=== Flush settings
|
|
|
|
The following <<indices-update-settings,dynamically updatable>> settings
|
|
control how often the in-memory buffer is flushed to disk:
|
|
|
|
`index.translog.flush_threshold_size`::
|
|
|
|
Once the translog hits this size, a flush will happen. Defaults to `512mb`.
|
|
|
|
`index.translog.flush_threshold_ops`::
|
|
|
|
After how many operations to flush. Defaults to `unlimited`.
|
|
|
|
`index.translog.flush_threshold_period`::
|
|
|
|
How long to wait before triggering a flush regardless of translog size. Defaults to `30m`.
|
|
|
|
`index.translog.interval`::
|
|
|
|
How often to check if a flush is needed, randomized between the interval value
|
|
and 2x the interval value. Defaults to `5s`.
|
|
|
|
[float]
|
|
=== Translog settings
|
|
|
|
The translog itself is only persisted to disk when it is ++fsync++ed. Until
|
|
then, data recently written to the translog may only exist in the file system
|
|
cache and could potentially be lost in the event of hardware failure.
|
|
|
|
The following <<indices-update-settings,dynamically updatable>> settings
|
|
control the behaviour of the transaction log:
|
|
|
|
`index.translog.sync_interval`::
|
|
|
|
How often the translog is ++fsync++ed to disk. Defaults to `5s`. Can be set to
|
|
`0` to sync after each operation.
|
|
|
|
`index.translog.fs.type`::
|
|
|
|
Either a `buffered` translog (default) which buffers 64kB in memory before
|
|
writing to disk, or a `simple` translog which writes every entry to disk
|
|
immediately. Whichever is used, these writes are only ++fsync++ed according
|
|
to the `sync_interval`.
|
|
|
|
The `buffered` translog is written to disk when it reaches 64kB in size, or
|
|
whenever a `sync` is triggered by the `sync_interval`.
|
|
|
|
.Why don't we `fsync` the translog after every write?
|
|
******************************************************
|
|
|
|
The disk is the slowest part of any server. An `fsync` ensures that data in
|
|
the file system buffer has been physically written to disk, but this
|
|
persistence comes with a performance cost.
|
|
|
|
However, the translog is not the only persistence mechanism in Elasticsearch.
|
|
Any index or update request is first written to the primary shard, then
|
|
forwarded in parallel to any replica shards. The primary waits for the action
|
|
to be completed on the replicas before returning success to the client.
|
|
|
|
If the node holding the primary shard dies for some reason, its transaction
|
|
log could be missing the last 5 seconds of data. However, that data should
|
|
already be available on a replica shard on a different node. Of course, if
|
|
the whole data centre loses power at the same time, then it is possible that
|
|
you could lose the last 5 seconds (or `sync_interval`) of data.
|
|
|
|
We are constantly monitoring the perfromance implications of better default
|
|
translog sync semantics, so the default might change as time passes and HW,
|
|
virtualization, and other aspects improve.
|
|
|
|
****************************************************** |