The Life of a Write: From RAM to Disk

[!NOTE] This module explores the core principles of The Life of a Write: From RAM to Disk, deriving solutions from first principles and hardware constraints to build world-class, production-ready expertise.

1. The Hook: “Near Real-Time” (NRT)

You write a document. You search for it immediately. It’s not there. 1 second later, it appears. Why? Because Disk I/O is expensive, so Elasticsearch cheats.

2. The Write Path (Step-by-Step)

Step 1: The Memory Buffer

When you POST /index/_doc/1, the document is written to the In-Memory Buffer.

  • It is NOT yet searchable.
  • It is NOT yet safe (if power fails, it’s gone).

Step 2: The Translog (Safety)

Simultaneously, the document is appended to the Translog (Transaction Log) on disk.

  • fsync: Happens every 5 seconds (by default).
  • Purpose: Recovery after crash.

Step 3: Refresh (Searchability)

Every 1 second (default), the Memory Buffer is cleared and written to a new Lucene Segment in the Filesystem Cache.

  • Now it is searchable.
  • Creating a segment is cheaper than fsync, so we do it often.

Step 4: Flush (Persistence)

Every 30 minutes (or when Translog is full), a Flush happens:

  1. All data in Filesystem Cache is fsynced to disk.
  2. Translog is cleared.

3. Visualizing the Path

Client
→
Memory Buffer
→
Segment (In Cache)
↓
Translog (Disk)
State: Idle

4. Tuning for Performance

  • Heavy Indexing? Increase refresh_interval from 1s to 30s. You lose 30s of “real-time”, but gain massive CPU throughput (fewer segments created).
  • Data Safety? Change index.translog.durability to async for faster writes (risk losing 5s of data).