OpenSearch/docs/reference/index-modules/translog.asciidoc

[[index-modules-translog]]
== Translog

Each shard has a transaction log or write ahead log associated with it.
It allows to guarantee that when an index/delete operation occurs, it is
applied atomically, while not "committing" the internal Lucene index for
each request. A flush ("commit") still happens based on several
parameters:

`index.translog.flush_threshold_ops`::

After how many operations to flush. Defaults to `unlimited`.

`index.translog.flush_threshold_size`:: 

Once the translog hits this size, a flush will happen. Defaults to `512mb`.

`index.translog.flush_threshold_period`:: 

The period with no flush happening to force a flush. Defaults to `30m`.

`index.translog.interval`:: 

How often to check if a flush is needed, randomized
between the interval value and 2x the interval value. Defaults to `5s`.

`index.translog.sync_interval`::

How often the translog is ++fsync++ed to disk. Defaults to `5s`.


Note: these parameters can be updated at runtime using the Index
Settings Update API (for example, these number can be increased when
executing bulk updates to support higher TPS)
Migrated documentation into the main repo 2013-08-28 19:24:34 -04:00			`[[index-modules-translog]]`
			`== Translog`

			`Each shard has a transaction log or write ahead log associated with it.`
			`It allows to guarantee that when an index/delete operation occurs, it is`
			`applied atomically, while not "committing" the internal Lucene index for`
			`each request. A flush ("commit") still happens based on several`
			`parameters:`

Update translog.asciidoc Documented `index.gateway.local.sync` 2014-07-31 08:06:06 -04:00			`index.translog.flush_threshold_ops`::
Migrated documentation into the main repo 2013-08-28 19:24:34 -04:00
Update translog.asciidoc Documented `index.gateway.local.sync` 2014-07-31 08:06:06 -04:00			After how many operations to flush. Defaults to `unlimited`.
Migrated documentation into the main repo 2013-08-28 19:24:34 -04:00
Update translog.asciidoc Documented `index.gateway.local.sync` 2014-07-31 08:06:06 -04:00			`index.translog.flush_threshold_size`::
randomize flush interval so multiple shards won't flush at the sam time - also, allow to update interval using update settings on an index 2014-01-07 13:58:16 -05:00
Core: change default xlog size from 200 MB to 512 MB Closes #9341 2015-01-19 15:52:29 -05:00			Once the translog hits this size, a flush will happen. Defaults to `512mb`.
Update translog.asciidoc Documented `index.gateway.local.sync` 2014-07-31 08:06:06 -04:00
			`index.translog.flush_threshold_period`::

			The period with no flush happening to force a flush. Defaults to `30m`.

			`index.translog.interval`::

			`How often to check if a flush is needed, randomized`
randomize flush interval so multiple shards won't flush at the sam time - also, allow to update interval using update settings on an index 2014-01-07 13:58:16 -05:00			between the interval value and 2x the interval value. Defaults to `5s`.
Update translog.asciidoc Documented `index.gateway.local.sync` 2014-07-31 08:06:06 -04:00
Decouple recoveries from engine flush In order to safely complete recoveries / relocations we have to keep all operation done since the recovery start at available for replay. At the moment we do so by preventing the engine from flushing and thus making sure that the operations are kept in the translog. A side effect of this is that the translog keeps on growing until the recovery is done. This is not a problem as we do need these operations but if the another recovery starts concurrently it may have an unneededly long translog to replay. Also, if we shutdown the engine for some reason at this point (like when a node is restarted) we have to recover a long translog when we come back. To void this, the translog is changed to be based on multiple files instead of a single one. This allows recoveries to keep hold to the files they need while allowing the engine to flush and do a lucene commit (which will create a new translog files bellow the hood). Change highlights: - Refactor Translog file management to allow for multiple files. - Translog maintains a list of referenced files, both by outstanding recoveries and files containing operations not yet committed to Lucene. - A new Translog.View concept is introduced, allowing recoveries to get a reference to all currently uncommitted translog files plus all future translog files created until the view is closed. They can use this view to iterate over operations. - Recovery phase3 is removed. That phase was replaying operations while preventing new writes to the engine. This is unneeded as standard indexing also send all operations from the start of the recovery to the recovering shard. Replay all ops in the view acquired in recovery start is enough to guarantee no operation is lost. - IndexShard now creates the translog together with the engine. The translog is closed by the engine on close. ShadowIndexShards do not open the translog. - Moved the ownership of translog fsyncing to the translog it self, changing the responsible setting to `index.translog.sync_interval` (was `index.gateway.local.sync`) Closes #10624 2015-03-27 05:18:09 -04:00			`index.translog.sync_interval`::
Update translog.asciidoc Documented `index.gateway.local.sync` 2014-07-31 08:06:06 -04:00
			How often the translog is ++fsync++ed to disk. Defaults to `5s`.

Migrated documentation into the main repo 2013-08-28 19:24:34 -04:00
			`Note: these parameters can be updated at runtime using the Index`
			`Settings Update API (for example, these number can be increased when`
			`executing bulk updates to support higher TPS)`