\documentclass{article}

\usepackage{imakeidx}
\usepackage[pdfborder={0 0 0},bookmarks=true,bookmarksnumbered=true]{hyperref}
\usepackage{bookmark}
\usepackage{longtable}
\usepackage{upquote}

% Include auto-generated fragments from bch-docgen.
% Run: cargo run -p bch-docgen
\newcommand{\bchdoc}[1]{\input{doc/generated/#1}}

\title{bcachefs: Principles of Operation}
\author{Kent Overstreet (and others)}

\date{\today}

\begin{document}

\maketitle
\tableofcontents

\section{Introduction and overview}

Bcachefs is a modern, general purpose, copy on write filesystem descended from
bcache, a block layer cache.

The internal architecture is very different from most existing filesystems where
the inode is central and many data structures hang off of the inode. Instead,
bcachefs is architected more like a filesystem on top of a relational database,
with tables for the different filesystem data types - extents, inodes, dirents,
xattrs, et cetera.

The entire filesystem is built on two primitives: btrees and buckets. All
filesystem state is a key-value pair in a btree. A handful of btrees store core
filesystem data (extents, inodes, dirents, xattrs, subvolumes), while others
handle allocation tracking, reverse mappings, background maintenance, and crash
recovery. There are no separate inode tables, bitmap allocators, or per-inode
extent trees. On the storage side, all disk space is divided into buckets with
generation numbers, described in the allocation section below. The btrees map
logical filesystem objects to physical bucket locations, and backpointers
provide the reverse mapping. Everything else in bcachefs is built on this
transactional key-value store over generational bucket storage.

bcachefs supports almost all of the same features as other modern COW
filesystems, such as ZFS and btrfs, but in general with a cleaner, simpler,
higher performance design.

\subsection{Performance overview}

The core of the architecture is a very high performance and very low latency b+
tree, which also is not a conventional b+ tree but more of hybrid, taking
concepts from compacting data structures: btree nodes are very large, log
structured, and compacted (resorted) as necessary in memory. This means our b+
trees are very shallow compared to other filesystems. Other COW filesystems use
the Linux page cache for metadata, which limits them to 4K btree nodes; bcachefs
manages its own btree node cache, so we can use 128K--256K nodes and tune
reclaim independently of the page cache. At petabyte scale with spinning disks,
deep btrees with 4K nodes mean more seeks per lookup and level-2 nodes too large
to stay resident under page cache pressure.

What this means for the end user is that since we require very few seeks or disk
reads, filesystem latency is extremely good - especially cache cold filesystem
latency, which does not show up in most benchmarks but has a huge impact on real
world performance, as well as how fast the system "feels" in normal interactive
usage. Latency has been a major focus throughout the codebase - notably, we have
assertions that we never hold b+ tree locks while doing IO, and the btree
transaction layer makes it easy to aggressively drop and retake locks as
needed - one major goal of bcachefs is to be the first general purpose soft
realtime filesystem.

Additionally, unlike other COW btrees, btree updates are journalled. This
greatly improves our write efficiency on random update workloads, as it means
btree writes are only done when we have a large block of updates, or when
required by memory reclaim or journal reclaim.

\subsection{Transactions}

All btree operations go through a transaction layer (\texttt{btree\_trans}) that
provides atomic multi-key updates with optimistic concurrency control.
Transactions collect updates in memory, then commit them atomically: first
acquiring a journal reservation, then taking write locks on affected btree nodes
in sorted order, applying updates, and releasing locks after the journal has
accepted the change (but before the journal write is persisted to disk). This
means btree locks are held only for in-memory operations, never across IO.

Btree nodes use six-locks (shared/intent/exclusive): the intent state allows an
operation that modifies multiple nodes to hold intent locks for its duration,
upgrading to write only for each individual in-memory update, avoiding the need
to hold write locks across entire split or merge operations.

Rather than blocking on lock contention, when a transaction encounters a
potential deadlock it drops all locks and restarts from the beginning. Deadlock
detection is integrated into the lock acquisition path: before sleeping on a
contended lock, the transaction checks for lock cycles and aborts if one is
found. This restart-based approach avoids deadlocks entirely and keeps
worst-case latency bounded, at the cost of
requiring all transaction code to be idempotent and prepared for restarts at any
point. This idempotency requirement has a profound consequence for reliability:
since every operation can be safely interrupted and restarted, the filesystem is
inherently resilient to interruption at any point - including during recovery
itself. A crash during journal replay or an on-disk format upgrade simply
results in the operation continuing from a safe point on the next mount.
Recovery passes generally restart from the beginning and re-check everything;
only passes that maintain persistent work queues can resume mid-pass.

Sequence numbers on btree nodes enable efficient relocking after a restart: if a
node's sequence number hasn't changed, the lock can be reacquired without
re-traversing the btree.

Transactions also support triggers: callbacks that fire on key insertion or
deletion. Transactional triggers run before commit and may generate additional
updates (e.g. updating backpointers when an extent changes). Atomic triggers
run with the journal lock held and are used for accounting updates that must be
exactly synchronized with the commit.

For operations that cannot be made atomic within a single transaction (such as
truncate or file insert/collapse, which may need to modify an unbounded number
of extents), bcachefs uses logged operations: the intent is written to the
\texttt{logged\_ops} btree before starting, and on recovery any incomplete
logged operations are replayed to completion.

\subsection{Recovery}

On unclean shutdown, the journal is replayed to bring the btrees up to date.
Journal entries record btree updates in commit order; replay applies them
sequentially to reconstruct the state at the last completed commit.

To guarantee ordering of btree updates after a crash, we need to detect when a
btree node was flushed to disk but its corresponding journal entry was not;
otherwise, recovery could see later updates without the earlier updates they
depended on.

During btree node reads, bcachefs detects this situation: bsets (sorted key
arrays within a node) whose journal sequence number is newer than the last
successfully written journal entry indicate the btree node was written to disk
but the corresponding journal entry was not. These sequences are blacklisted
in the superblock (\texttt{bch\_sb\_field\_journal\_seq\_blacklist}), and
bsets referencing them are ignored until the btree node is next rewritten.
After an unclean shutdown, a padding of 64 sequence numbers past the last
journal entry read is also blacklisted, to cover in-flight btree writes.
Blacklisted sequences are never reused---the 64-bit counter simply skips
past them. Blacklist entries are garbage collected once no btree node on disk
references them.

\subsubsection{Recovery passes}

Beyond journal replay, bcachefs has approximately 48 recovery passes with an
explicit dependency graph. Not all passes run on every mount - which passes run
depends on context:

\begin{description}
	\item[Every mount] A default set of passes is scheduled: reading
		accounting, bucket generations, and snapshots first, checking
		snapshot consistency, then resuming logged operations and
		cleaning up dead inodes.

	\item[Unclean shutdown] Adds journal replay. No extra repair passes
		are needed because the filesystem is always consistent after
		journal replay (absent bugs or data corruption).

	\item[Version upgrade/downgrade] Each version transition defines
		specific passes to run for format migration.

	\item[Fsck] A defined subset of validation passes runs: checking inodes,
		extents, dirents, xattrs, link counts, and directory structure.
		Note that \texttt{check\_backpointers\_to\_extents} (the reverse
		direction check) is too expensive to run during normal fsck.
\end{description}

Recovery is split into two phases around the transition to read-write mode.
Early passes run in-memory only, accumulating delayed metadata writes: reading
btrees into memory, verifying btree topology, and reconstructing allocation
state. Once the allocator and journal are initialized, the filesystem
transitions to read-write and journal replay runs - followed by the heavier
validation and repair passes that need to write.

The filesystem can schedule new recovery passes at any time, including during
other recovery passes or normal operation, as soon as it notices something needs
repair. In early recovery, this can rewind the recovery sequence to run earlier
passes again. Most validation passes (those marked \texttt{PASS\_ONLINE}) can
also run after the filesystem is fully mounted, either triggered explicitly or
scheduled automatically when an inconsistency is detected at runtime.

For catastrophic damage, heavier passes exist that reconstruct entire btrees
from scratch. These are triggered automatically when normal recovery detects
structural damage: \texttt{scan\_for\_btree\_nodes} is scheduled when
\texttt{check\_topology} finds missing or unreachable btree nodes - it
physically scans the raw device for valid btree node headers and recovers what
it can. \texttt{reconstruct\_snapshots} is scheduled when the snapshots btree
itself is lost, or when references to missing snapshots cannot be resolved
by deletion alone and recovering those snapshots is feasible.
\texttt{check\_allocations} rebuilds all allocation accounting from the extents
and backpointers btrees. These passes are not part of normal recovery and may
take considerable time on large filesystems.

When a pass is scheduled at runtime (post-mount), its persistence depends on
severity. Passes scheduled without the \texttt{nopersistent} flag are written
to the superblock's \texttt{recovery\_passes\_required} field and survive
reboots: the pass will run automatically at the next mount even if the
filesystem is cleanly unmounted. Ephemeral scheduling (used when a prior pass
must complete first before the repair makes sense) is remembered only until
the current recovery sequence completes and does not touch the superblock.
Once a pass runs successfully, it clears its own bit from
\texttt{recovery\_passes\_required}.

Some passes are scheduled with a ratelimit flag, used for errors that may be
detected in large numbers during a single I/O operation (for example, EC
accounting inconsistencies during a stripe write). A ratelimited pass is not
re-scheduled unless the same error is detected again via a non-ratelimited
request, preventing repair-loop flooding. The ratelimiting state is in-memory
only and does not affect persistence.

The complete list of recovery passes, in execution order:

\bchdoc{recovery-passes}

\subsection{Bucket based allocation}

As mentioned bcachefs is descended from bcache, where the ability to efficiently
invalidate cached data and reuse disk space was a core design requirement. To
make this possible the allocator divides the disk up into buckets, typically
512k to 2M but possibly larger or smaller. Buckets and data pointers have
generation numbers: we can reuse a bucket with cached data in it without finding
and deleting all the data pointers by incrementing the generation number. The
generation number mechanism was originally created for invalidating cached data,
but it turned out to be valuable well beyond that: it makes bootstrapping
allocation information during journal replay much more tractable, and it makes
it possible to repair certain kinds of corruption that would be unrecoverable
otherwise.

In keeping with the copy-on-write theme of avoiding update in place wherever
possible, we never rewrite or overwrite data within a bucket - when we allocate
a bucket, we write to it sequentially and then we don't write to it again until
the bucket has been invalidated and the generation number incremented.

This means we require a copying garbage collector to deal with internal
fragmentation, when patterns of random writes leave us with many buckets that
are partially empty (because the data they contained was overwritten) - copy GC
evacuates buckets that are mostly empty by writing the data they contain to new
buckets. This also means that we need to reserve space on the device for the
copy GC reserve when formatting - 8\% by default (configurable from 5\% to
21\%). A fragmentation LRU btree tracks bucket fill levels so that copygc can
find the most fragmented buckets directly instead of scanning the entire alloc
btree.

The allocator originally used in-memory bucket arrays and dedicated scanning
threads to maintain free lists, discard lists, and eviction heaps. These were
replaced by btree-based data structures: a freespace btree, a discard btree,
and an LRU btree. The scanning threads are gone entirely; the code that
replaces them is transactional btree code, much of it trigger-based, which is
far easier to debug and reason about. The old threads were also prone to
excessive CPU usage when the filesystem was nearly full; the btree approach
eliminated those corner cases.

There are some advantages to structuring the allocator this way, besides being
able to support cached data:
\begin{itemize}
	\item By maintaining multiple write points that are writing to different buckets,
		we're able to easily and naturally segregate unrelated IO from different
		processes, which helps greatly with fragmentation.

	\item The fast path of the allocator is essentially a simple bump allocator - the
		disk space allocation is extremely fast

	\item Fragmentation is generally a non issue unless copygc has to kick
		in, and it usually doesn't under typical usage patterns. The
		allocator and copygc are doing essentially the same things as
		the flash translation layer in SSDs, but within the filesystem
		we have much greater visibility into where writes are coming
		from and how to segregate them, as well as which data is
		actually live - performance is generally more predictable than
		with SSDs under similar usage patterns.

	\item The same algorithms are intended to be used for managing SMR
		(shingled magnetic recording) and zoned hard drives directly,
		avoiding the translation layer in the drive. Buckets map
		naturally to zones: when we allocate a bucket, we write to it
		once in an append-only fashion, then never write to it again
		until we discard and reuse the whole thing --- exactly the
		semantics a zoned device requires.
\end{itemize}

\subsection{Backpointers}

Every sector range on disk that contains data or metadata has a corresponding
backpointer: a reverse reference from the physical location back to the logical
btree entry that owns it. Backpointers are stored in a dedicated write-buffered
btree (ID 13), keyed by (device, starting offset, discriminator), with the value
recording the btree ID, level, and position of the owning key.

The discriminator field exists because compressed extents complicate ownership:
a single highly-compressed block can map to a large extent, and if that extent
is partially overwritten, multiple smaller extents may share ownership of the
same physical block. Erasure code stripes also get backpointers because they
co-own data with the extents they protect --- even after an extent is deleted,
the stripe may still need the data for EC reconstruction until the stripe itself
is reclaimed. Stripe backpointers are flagged with
\texttt{BACKPOINTER\_STRIPE\_PTR} and routed to the separate
\texttt{stripe\_backpointers} btree.

Backpointers enable efficient answers to the question ``what data lives in this
bucket?'' without scanning the entire extents btree. This is essential for device
evacuation (moving all data off a device), scrub (verifying data integrity in
physical order), and the reconcile subsystem (tracking what needs to be moved on
rotational devices in LBA order). Three recovery passes validate backpointers:
\texttt{check\_extents\_to\_backpointers}, \texttt{check\_backpointers\_to\_extents},
and \texttt{check\_btree\_backpointers}.

\section{Feature overview}

\subsection{IO path options}
\label{sec:io-path-options}

Most options that control the IO path can be set at either the filesystem level
or on individual inodes (files and directories). When set on a directory via the
\texttt{bcachefs set-file-option} command, they will be automatically applied
recursively.

\subsubsection{Checksumming}

bcachefs supports both metadata and data checksumming - crc32c by default, but
stronger checksums are available as well. Enabling data checksumming incurs some
performance overhead - besides the checksum calculation, writes have to be
bounced for checksum stability (Linux generally cannot guarantee that the buffer
being written is not modified in flight), but reads generally do not have to be
bounced.

Checksum granularity in bcachefs is at the level of individual extents, which
results in smaller metadata but means we have to read entire extents in order to
verify the checksum. By default, checksummed and compressed extents are capped
at 256 KB (controlled by \texttt{encoded\_extent\_max}).
For most applications and usage scenarios this is an ideal trade off, but
small random \texttt{O\_DIRECT} reads will incur significant overhead. In the
future, checksum granularity will be a per-inode option.

\subsubsection{Encryption}

bcachefs supports authenticated (AEAD) encryption using ChaCha20/Poly1305.
Unlike typical block layer or filesystem encryption (usually AES-XTS), which
operates on fixed blocks with no room for nonces or MACs, bcachefs stores a
nonce and cryptographic MAC alongside every data pointer. This creates a chain
of trust from the superblock (or journal) down to individual extents: any
modification, deletion, reordering, or rollback of metadata is detectable.

Encryption is all-or-nothing at the filesystem level: all data and metadata
except the superblock is encrypted, and all data and metadata is authenticated.
Per-file or per-directory encryption is not supported because btree nodes are
shared structures that do not belong to any individual file.

\textbf{Critical exception}: Data written with the \texttt{nocow} option is
stored \textbf{unencrypted}, even on an encrypted filesystem. The nocow write
path bypasses the normal data path transformations including encryption. This
is because the current AEAD encryption scheme is fundamentally incompatible
with mutable data. Encryption can only be enabled at format time; it cannot
be added to an existing filesystem.

\paragraph{Key hierarchy}

The key hierarchy has three levels:
\begin{enumerate}
	\item \textbf{Passphrase}: User-supplied, never stored on disk. Fed to
		the scrypt KDF (parameters stored in the superblock's
		\texttt{bch\_sb\_field\_crypt}) to derive a 256-bit
		passphrase key. The KDF runs entirely in userspace, so
		alternative key sources (hardware tokens, key files) can be
		integrated without kernel changes.

	\item \textbf{Master key}: A random 256-bit key generated at format
		time, stored in the superblock encrypted by the passphrase
		key. A magic value (\texttt{BCH\_KEY\_MAGIC}) stored
		alongside the encrypted master key allows verification of a
		correct passphrase without trial decryption of filesystem
		data. Changing the passphrase re-encrypts only the master key,
		not any filesystem data.

	\item \textbf{Per-extent nonces}: Each extent is encrypted with the
		master key and a 128-bit nonce composed of the extent's
		96-bit version number, compression type, and uncompressed
		size, combined with a per-CRC nonce offset. Data encryption
		uses the \texttt{BCH\_NONCE\_EXTENT} domain separator; the
		Poly1305 MAC key uses \texttt{BCH\_NONCE\_POLY}.
\end{enumerate}

There is currently no key rotation mechanism: the master key is fixed for the
lifetime of the filesystem. Key escrow, multi-passphrase unlock, and hardware
key (TPM, FIDO2) support are not implemented. Losing the passphrase means
permanent data loss.

\paragraph{Kernel keyring integration}

The kernel never sees the passphrase. Instead, userspace derives the passphrase
key via scrypt and adds it to the Linux kernel keyring as a \texttt{user} type
key with description \texttt{bcachefs:<UUID>}. At mount time, the kernel calls
\texttt{request\_key()} to find this key, uses it to decrypt the master key
from the superblock, and caches the decrypted master key in kernel memory for
the lifetime of the mount.

This design inherits the well-known pain points of the Linux keyring subsystem:

\begin{itemize}
	\item \textbf{Session isolation}: Keys added to a session keyring are
		not visible from other sessions of the same user. An
		\texttt{ssh} session that runs \texttt{bcachefs unlock} does
		not make the key available to a different terminal, to
		systemd mount units, or to cron jobs. The key must be added
		to \texttt{KEY\_SPEC\_USER\_KEYRING} (the per-UID keyring) to
		be visible across sessions, but this is not always the
		default.

	\item \textbf{Privilege boundaries}: \texttt{sudo mount} uses root's
		keyring, not the calling user's. Systemd units run in
		isolated session contexts. The key must be explicitly placed
		in a keyring that the mounting process can access.
\end{itemize}

\paragraph{MAC storage}

The Poly1305 MAC is stored in the extent's CRC entry. By default, the MAC is
truncated to 80 bits (\texttt{chacha20\_poly1305\_80}), which is sufficient for
most threat models. The \texttt{wide\_macs} option stores the full 128-bit MAC
at the cost of 8 bytes per extent, and is recommended when the storage device
itself is untrusted (e.g. USB drives, network storage) and an attacker can make
repeated forgery attempts or perform rollback attacks (taking a snapshot without
your knowledge, then tricking you into mounting the older snapshot read-write).
Metadata always uses 128-bit MACs regardless of the \texttt{wide\_macs} setting.

\paragraph{Nonce reuse vulnerability with external snapshots}

AEAD algorithms require that a (key, nonce) pair is never reused for different
plaintexts. bcachefs derives extent nonces from the extent's version number
(\texttt{struct bversion}), which is part of the on-disk key and is unique
within a single filesystem instance. However, if the underlying storage is
snapshotted externally (LVM, ZFS zvol, VM snapshot, loop device on a reflinked
file) and the snapshot is mounted read-write, both instances share the same
master key and will derive the same nonces for new writes to the same logical
locations. This breaks ChaCha20's semantic security: an attacker observing
both ciphertexts can XOR them to obtain the XOR of the plaintexts.

bcachefs's own snapshot mechanism does not have this problem: internal snapshots
share extents via reflinks with COW semantics, and new writes get new version
numbers and therefore new nonces.

\textbf{Mitigation}: Never mount a snapshot of an externally-snapshotted
encrypted volume read-write --- keep external snapshots linear, not branching.
The most recent (non-snapshot) version is safe to mount read-write; only the
older snapshots must remain read-only (use \texttt{-o nochanges}). Alternatively,
place LUKS between the snapshot layer and bcachefs --- the LUKS ciphertext
hides the ChaCha20 nonce reuse from observation. For example, on LVM: LVM
$\to$ LUKS $\to$ bcachefs.

\subsubsection{Compression}

bcachefs supports gzip, lz4 and zstd compression. As with data checksumming, we
compress entire extents, not individual disk blocks - this gives us better
compression ratios than other filesystems, at the cost of reduced small random
read performance. Compression algorithms are fundamentally able to find more
redundancy when fed more data at once; compressing at 4K page granularity
also creates painful alignment issues, because rounding compressed output
back up to block alignment loses much of the compression ratio. Per-extent
granularity also simplifies the IO path: everything works in terms of
extents, and there is nothing smaller than an extent to write code to handle.

Data can also be compressed or recompressed with a different algorithm in the
background by the reconcile subsystem, if the \texttt{background\_compression}
option is set.

\subsection{Multiple devices}

bcachefs is a multi-device filesystem. Devices need not be the same size: by
default, the allocator will stripe across all available devices but biasing in
favor of the devices with more free space, so that all devices in the filesystem
fill up at the same rate. Devices need not have the same performance
characteristics: we track device IO latency and direct reads to the device that
is currently fastest.

\subsubsection{Replication}

bcachefs supports standard RAID1/10 style redundancy with the
\texttt{data\_replicas} and \texttt{metadata\_replicas} options. Layout is not
fixed as with RAID10: a given extent can be replicated across any set of
devices; the \texttt{bcachefs fs usage} command shows how data is replicated
within a filesystem.

\subsubsection{Erasure coding}

bcachefs supports Reed-Solomon erasure coding, the same algorithm used by
most RAID5/6 implementations. Erasure coding is a per-inode option
(\texttt{erasure\_code}), so it can be enabled on specific directories via
\texttt{set-file-option} without affecting the rest of the filesystem. The
desired redundancy is taken from the \texttt{data\_replicas} option:
\texttt{data\_replicas=2} yields one parity block (RAID-5 style),
\texttt{data\_replicas=3} yields two (RAID-6 style), and
\texttt{data\_replicas=1} disables erasure coding entirely. Parity is limited
to at most two blocks. Erasure coding of metadata is not supported.

Each block in a stripe is one bucket on one device. Stripe width is determined
dynamically: all eligible devices in the target group are used, up to a maximum
of 16 blocks per stripe. Eligible devices must be read-write, have nonzero
durability, and share the same bucket size (the most common bucket size among
candidates is chosen; devices with a different bucket size are excluded). The
minimum is \texttt{redundancy + 2} devices (3 for RAID-5, 4 for RAID-6). With
$n$ eligible devices, a stripe has $n - \mathrm{redundancy}$ data blocks and
\texttt{redundancy} parity blocks, maximizing storage efficiency. There is no
configuration to limit stripe width to a subset of available devices.

Stripe fragmentation is tracked in the LRU btree. When all data blocks in a
stripe become empty (sector count zero), the stripe is automatically deleted.
Partially empty stripes are candidates for reuse: new stripe creation scans
the fragmentation LRU for a stripe with matching parameters (same disk label,
algorithm, and redundancy), copies the non-empty blocks into the new stripe,
and fills the remaining slots with fresh data. This consolidation recovers
space without a full copygc pass. Changing the \texttt{erasure\_code} option
at runtime triggers reconcile to add or remove EC protection on existing data.

In conventional RAID, the ``write hole'' is a significant problem: a small write
within a stripe requires updating the P and Q (parity) blocks, and since those
writes cannot be done atomically, a crash can leave parity inconsistent,
corrupting reconstruct reads for unrelated data in the same stripe. ZFS avoids
this by making every write a new stripe, but the resulting fragmentation hurts
performance: reads of fragmented data are bounded by the latency of the slowest
fragment, driving median latency toward tail latency.

bcachefs takes advantage of copy-on-write to avoid both problems. Since
updating stripes in place is the root cause, we simply never do it. And since
per-extent stripes would cause the same fragmentation as ZFS, we erasure code
entire buckets, taking advantage of bucket-based allocation and copying
garbage collection.

Writes with erasure coding enabled are initially replicated, but one replica is
allocated from a bucket queued for a new stripe. When the stripe is full, parity
buckets are written, the stripe key is committed to the stripes btree, and then
the extra replicas are dropped from each extent individually. The effect is
similar to full data journalling, producing an ideal on-disk layout. Since
device write caches are only flushed on journal commit, the allocator can
immediately return the extra-replica buckets to the write point for reuse,
so the overhead of this full data journalling approach is negligible in
practice.

Erasure coding involves several btrees: the stripes btree (ID 6) stores stripe
metadata and parity checksums, the \texttt{bucket\_to\_stripe} btree (ID 26)
maps each stripe-member bucket to the stripes referencing it, the
\texttt{stripe\_backpointers} btree (ID 27) stores backpointers indexed by
stripe pointers for data on invalid or removed devices (enabling stripe repair),
the alloc btree tracks per-bucket \texttt{stripe\_refcount} and
\texttt{stripe\_sectors} separately from \texttt{dirty\_sectors}, and after
stripe creation each data extent receives an inline \texttt{stripe\_ptr} entry.
Backpointers for stripe blocks point back to the stripes btree rather than the
extents btree. Stripe fragmentation is tracked in the LRU btree so that copygc
and stripe reuse can reclaim mostly-empty stripes.

The allocator segregates EC and non-EC writes at the open-bucket level: a write
requesting EC will only be placed in a bucket already tagged for stripe
membership, and non-EC writes will never use such buckets. This means a bug in
stripe creation, parity computation, or extent updating is structurally scoped
to EC-enabled data: non-EC extents never carry \texttt{stripe\_ptr} entries
and are never read through EC reconstruction paths. Btree nodes are never
placed in EC buckets; this is explicitly checked and flagged as a filesystem
inconsistency. If stripe creation fails partway (e.g.\ a crash between
writing parity and updating extents), the extra replicas from the staging
phase remain valid; the reconcile subsystem detects the incomplete state and
retries.

\subsubsection{Device labels and targets}

By default, writes are striped across all devices in a filesystem, but they may
be directed to a specific device or set of devices with the various target
options. The allocator only prefers to allocate from devices matching the
specified target; if those devices are full, it will fall back to allocating
from any device in the filesystem.

Target options may refer to a device directly, e.g.
\texttt{foreground\_target=/dev/sda1}, or they may refer to a device label. A
device label is a path delimited by periods - e.g. ssd.ssd1 (and labels need not
be unique). This gives us ways of referring to multiple devices in target
options: If we specify ssd in a target option, that will refer to all devices
with the label ssd or labels that start with ssd. (e.g. ssd.ssd1, ssd.ssd2).

Four target options exist. These options all may be set at the filesystem level
(at format time, at mount time, or at runtime via sysfs), or on a particular
file or directory:

\begin{description}
	\item \texttt{foreground\_target}: normal foreground data writes, and
		metadata if \\ \texttt{metadata\_target} is not set
	\item \texttt{metadata\_target}: btree writes
	\item \texttt{background\_target}: If set, user data (not metadata) will
		be moved to this target in the background
	\item\texttt{promote\_target}: If set, a cached copy will be added to
		this target on read, if none exists
\end{description}

\subsubsection{Caching}

When an extent has multiple copies on different devices, some of those copies
may be marked as cached. Buckets containing only cached data are discarded as
needed by the allocator in LRU order.

When data is moved from one device to another according to the \\
\texttt{background\_target} option, the original copy is left in place but
marked as cached. With the \texttt{promote\_target} option, the original copy is
left unchanged and the new copy on the \texttt{promote\_target} device is marked
as cached.

To do writeback caching, set \texttt{foreground\_target} and
\texttt{promote\_target} to the cache device, and \texttt{background\_target} to
the backing device. To do writearound caching, set \texttt{foreground\_target}
to the backing device and \texttt{promote\_target} to the cache device.

\subsubsection{Durability}

Some devices may be considered to be more reliable than others. For example, we
might have a filesystem composed of a hardware RAID array and several NVME flash
devices, to be used as cache. We can set replicas=2 so that losing any of the
NVME flash devices will not cause us to lose data, and then additionally we can
set durability=2 for the hardware RAID device to tell bcachefs that we don't
need extra replicas for data on that device - data on that device will count as
two replicas, not just one.

The durability option can also be used for writethrough caching: by setting
durability=0 for a device, it can be used as a cache and only as a cache -
bcachefs won't consider copies on that device to count towards the number of
replicas we're supposed to keep.

\subsubsection{Splitbrain detection}

When assembling a multi-device filesystem, bcachefs detects devices that have
diverged - been mounted and modified independently. Every superblock records a
sequence number (\texttt{seq}) and write timestamp (\texttt{write\_time});
devices with the same sequence but different write times have diverged. Since
version 1.4, superblocks also track the last known sequence of every other
member device: if a device's actual sequence exceeds what another member
expected, it was modified independently while that member was offline.

The two common causes of splitbrain are:

\begin{itemize}
	\item \textbf{Partial mount of replicated filesystem}: Mounting a subset
		of devices from a replicated filesystem (\texttt{replicas >= 2})
		with \texttt{-o degraded=very} allows writes to proceed despite
		missing devices. If the ``missing'' device is later reconnected,
		it now has stale metadata while the mounted devices have advanced.

	\item \textbf{External snapshots}: LVM, SAN, or VM snapshots create
		block devices with valid bcachefs superblocks sharing the same
		filesystem UUID. If both original and snapshot are visible
		during assembly, or if the snapshot is mounted read-write and
		later reassembled with the original, the devices have diverged.
\end{itemize}

When divergence is detected, bcachefs excludes the stale device. Once devices
have genuinely diverged with writes on both sides, recovery is impractical -
resolving differences key-by-key across all btrees is not feasible. The
\texttt{-o no\_splitbrain\_check} option includes divergent devices anyway,
but is only safe when no actual modifications occurred on the excluded device.

\subsection{Reflink}

bcachefs supports reflink, similarly to other filesystems with the same feature.
\texttt{cp --reflink} will create a copy that shares the underlying storage.
Reading from that file will become slightly slower - the extent pointing to that
data is moved to the reflink btree (with a refcount added) and in the extents
btree we leave a key that points to the indirect extent in the reflink btree,
meaning that we now have to do two btree lookups to read from that data instead
of just one.

\subsubsection{On-disk representation}

In the extents btree, a \texttt{KEY\_TYPE\_reflink\_p} replaces the original
extent. It contains an index (\texttt{REFLINK\_P\_IDX}, 56 bits) pointing
into the reflink btree (ID 7), plus \texttt{front\_pad} and
\texttt{back\_pad} fields. In the reflink btree, a
\texttt{KEY\_TYPE\_reflink\_v} stores the actual data pointers, CRCs, and
compression metadata (identical to a regular \texttt{KEY\_TYPE\_extent})
preceded by a 64-bit reference count.

The pad fields exist because copygc or reconcile may split an indirect extent
into fragments. Without the pads, fragments outside the pointer's nominal
range would have their refcounts leaked. The pads remember the full range
originally referenced so that triggers walk all fragments when updating
refcounts. If the indirect extent is missing in the live data range (e.g.
due to corruption), fsck sets the \texttt{REFLINK\_P\_ERROR} flag on the
pointer; gaps only in the padded region adjust the pad instead.

The \texttt{reflink\_p} also carries a
\texttt{REFLINK\_P\_MAY\_UPDATE\_OPTIONS} flag that controls whether IO path
options (compression, checksum type, replicas, targets) may propagate from the
referencing file to the shared indirect extent. This is a security boundary: a
reflink copy of data owned by another user must not allow the copier to
decrease replicas or change checksum settings on data they do not own. At
creation time, the source file's \texttt{reflink\_p} gets this flag set, but
the destination's does not (the VFS layer does not yet pass down the
permission context needed to determine whether the copier has write access to
the source).

\subsubsection{Creation and lifecycle}

When \texttt{cp --reflink} (or the \texttt{FICLONE} ioctl) creates a reflink,
the source extent is converted in place. A new \texttt{KEY\_TYPE\_reflink\_v}
is allocated at the end of the reflink btree (by seeking to
\texttt{POS\_MAX}), containing the original data pointers and a refcount
initialized to zero. The source extent is replaced with a
\texttt{KEY\_TYPE\_reflink\_p}. If the source is already a
\texttt{reflink\_p}, no conversion is needed. A new \texttt{reflink\_p} is
then created in the destination file; btree triggers on the inserts increment
the refcount.

On insertion or deletion of a \texttt{reflink\_p}, the trigger walks the full
referenced range (including pad) in the reflink btree and increments or
decrements the refcount on each overlapping \texttt{reflink\_v} fragment,
expanding the pads if the indirect extent is larger than expected (due to a
prior split). Writing new data over a \texttt{reflink\_p} requires no special
logic: the normal btree update inserts a regular \texttt{KEY\_TYPE\_extent},
the overwrite trigger decrements the refcount, and when a
\texttt{reflink\_v}'s refcount reaches zero, its trigger converts the key to
\texttt{KEY\_TYPE\_deleted}, cascading through the normal extent trigger to
free disk space and remove backpointers.

Reflink is currently a one-way transformation: once an extent becomes
indirect, it never converts back, even when the refcount drops to 1. The
\texttt{reflink\_v} trigger fires at refcount 0 to delete the indirect
extent, but does not de-indirect at refcount 1 because the trigger would
need to walk transaction updates to find the sole remaining
\texttt{reflink\_p}, and operations like \texttt{fcollapse} and
\texttt{finsert} can cause transient refcount fluctuations (1 $\to$ 0
$\to$ 1) within a single transaction as extents are moved around. With IO
option propagation, de-indirecting at refcount 1 is becoming a more
pressing concern, since a lone indirect extent with one reference still
pays the cost of an extra btree lookup on every read.
Additionally, \texttt{reflink\_p} keys are not merged during btree
compaction because a merged pointer could span an unbounded number of
\texttt{reflink\_v} fragments; merging requires triggers to walk pending
transaction updates and diff overlapping \texttt{reflink\_p} ranges.

\subsubsection{IO option propagation}

A \texttt{reflink\_v} has no backpointer to its owning inode, so it cannot
look up per-inode IO options at read time. Instead, the indirect extent
embeds a \texttt{bch\_extent\_reconcile} entry that stores the desired IO
options (compression, checksum type, replicas, targets) alongside
\texttt{*\_from\_inode} flags recording which options came from a per-inode
setting rather than the filesystem default. At creation time no reconcile
entry is added; the data is simply copied verbatim from the source extent.

When reconcile scans the extents btree and encounters a \texttt{reflink\_p}
with \texttt{REFLINK\_P\_MAY\_UPDATE\_OPTIONS} set, it follows through to
the corresponding \texttt{reflink\_v} keys in the reflink btree and updates
their embedded reconcile entries with the referencing inode's current
options. If the on-disk data does not match (e.g. the inode now requests
zstd compression but the data is uncompressed), the reconcile entry's
\texttt{need\_rb} bits are set and the data is scheduled for background
rewrite. Without the flag, reconcile does not propagate that
\texttt{reflink\_p}'s inode options to the indirect extent.

The \texttt{reflink\_v} can only hold one set of IO options at a time. Since
only the source file's \texttt{reflink\_p} currently gets the
\texttt{MAY\_UPDATE\_OPTIONS} flag, there is no conflict when multiple files
reference the same indirect extent: the source file's options take
precedence, and other referencing files cannot influence the indirect
extent's IO path behavior. The read path always uses the CRC and compression
metadata stored in the \texttt{reflink\_v}'s extent entries (reflecting how
the data was actually written), regardless of the referencing file's current
options; the referencing inode's options only affect promote decisions.

\subsubsection{Interaction with snapshots}

The reflink btree is not snapshot-aware: \texttt{reflink\_v} keys are shared
across all snapshots. The \texttt{reflink\_p} keys in the extents btree are
snapshot-aware, so when a file is snapshotted both subvolumes see the same
\texttt{reflink\_p} keys through normal snapshot visibility. Writing to
either subvolume creates a new extent in that snapshot and decrements the
shared refcount; the other snapshot's \texttt{reflink\_p} is unchanged.

\subsection{Inline data extents}

bcachefs supports inline data extents, controlled by the \texttt{inline\_data}
option (on by default). When the end of a file is being written and is smaller
than \texttt{min(blocksize/2, 1024)} bytes, it will be written as an inline data
extent. Inline data extents can also be reflinked: the inline data is moved to
the reflink btree as a \texttt{KEY\_TYPE\_indirect\_inline\_data} (which carries
a refcount and the inline data bytes) and a \texttt{KEY\_TYPE\_reflink\_p} is
left in the extents btree, following the same mechanics as regular extent
reflinks.

\subsection{Subvolumes and snapshots}

bcachefs supports subvolumes and snapshots with a similar userspace interface as
btrfs. A new subvolume may be created empty, or it may be created as a snapshot
of another subvolume. Snapshots are writeable by default and may be snapshotted
again, creating a tree of snapshots; they can also be created as read-only with
the \texttt{--read-only} (or \texttt{-r}) flag.

Snapshots are very cheap to create: they're not based on cloning of COW btrees
as with btrfs, but instead are based on versioning of individual keys in the
btrees. Many thousands or millions of snapshots can be created, with the only
limitation being disk space.

\subsubsection{Subvolume and snapshot architecture}

A subvolume is a named entry point into the filesystem: it holds a root inode
number and a snapshot ID. The snapshot ID links the subvolume to a leaf node in
the snapshots btree, which records the snapshot tree structure: parent, children
(up to two), depth, and a skiplist for fast ancestor queries. Only leaf snapshot
nodes are associated with subvolumes; interior nodes exist purely for tree
structure. Snapshot trees are grouped under \texttt{snapshot\_tree} entries,
each recording the root snapshot and a master subvolume. Each subvolume carries
a \texttt{BCH\_SUBVOLUME\_SNAP} flag that records whether it was created as a
snapshot of another subvolume; exactly one subvolume per snapshot tree (the
master) does not have this flag set.

Four btrees are snapshot-aware: extents, inodes, dirents, and xattrs. Every key
in these btrees includes a snapshot ID in its position
(\texttt{bpos.snapshot}), so keys from different snapshots coexist in the same
btree ordered by (inode, offset, snapshot). When reading from a snapshot, the
iterator walks up the snapshot tree: a key is visible if its snapshot ID is an
ancestor of (or equal to) the requested snapshot, and no closer ancestor has
overwritten it. Deletion within a snapshot inserts a whiteout key that blocks
visibility of the ancestor's version without affecting other snapshots. To
avoid a linear parent-pointer walk on every lookup, each snapshot node stores a
128-bit ancestor bitmap for O(1) checks when ancestor and descendant are
within 128 snapshot IDs of each other, plus a randomized skiplist of three
ancestor IDs for O(log $n$) convergence on deeper trees. During early
recovery, before this data is validated, queries fall back to a simple parent
walk.

When a snapshot is created, two new snapshot nodes are allocated as children of
the source subvolume's current snapshot node. One child becomes the new
snapshot's ID; the other replaces the source subvolume's snapshot ID. No keys
are copied: both children inherit visibility of all ancestor keys through the
snapshot tree. Subsequent writes to either subvolume create new keys tagged with
that subvolume's snapshot ID, diverging only where modifications occur.

Deleting a snapshot marks it for asynchronous cleanup by a background thread
that must walk every snapshot-aware btree to remove or relocate affected keys.
For leaf snapshots, keys are deleted or converted to whiteouts where ancestor
visibility must be preserved. For interior nodes that have lost all but one
child, keys are moved to the surviving child before the node is removed. The
eytzinger-layout search trees within each btree node make this scan efficient:
they allow cache-friendly binary search over the packed keys, so the deletion
thread can seek through large btree nodes without excessive cache misses even
when processing millions of keys. Interior node removal is deferred to the next
mount because it requires updating depth and skiplist fields across the
subtree. Progress can be monitored via
\texttt{/sys/fs/bcachefs/<uuid>/snapshot\_delete\_status}.

The following subcommands exist for managing subvolumes and snapshots:
\begin{itemize}
	\item \texttt{bcachefs subvolume create}: Create a new, empty subvolume
	\item \texttt{bcachefs subvolume delete}: Delete an existing subvolume
		or snapshot
	\item \texttt{bcachefs subvolume snapshot}: Create a snapshot of an
		existing subvolume
\end{itemize}

A subvolume can also be deleted with a normal rmdir after deleting all the
contents, as with \texttt{rm -rf}. Still to be implemented: recursive snapshot
creation and a method for recursively listing subvolumes.

\subsection{Quotas}

bcachefs supports conventional user/group/project quotas. Quotas do not
currently apply to snapshot subvolumes, because if a file changes ownership in
the snapshot it would be ambiguous as to what quota data within that file
should be charged to. Quota accounting resolves every inode through the master
subvolume only, charging sectors and inode counts to the uid/gid/project
recorded there. Writes to snapshot subvolumes bypass quota enforcement entirely.
If there is no master subvolume for a snapshot tree, quota accounting for that
tree is skipped.

When a directory has a project ID set it is inherited automatically by
descendants on creation and rename. When renaming a directory would cause the
project ID to change we return -EXDEV so that the move is done file by file, so
that the project ID is propagated correctly to descendants - thus, project
quotas can be used as subdirectory quotas.

\subsection{32-bit inodes}

The \texttt{inodes\_32bit} option restricts new inode numbers to 32 bits. This
is needed for legacy NFS compatibility (NFSv2 and some older NFSv3
implementations use 32-bit inode numbers), for 32-bit userspace that may
truncate 64-bit inode numbers, and for Wine/Proton/Steam which run 32-bit
Windows applications. This option can be set per-directory, affecting only
files created within that directory tree.

Note that \texttt{inodes\_32bit} silently disables
\texttt{shard\_inode\_numbers\_bits}: there are too few bits in the 32-bit
space to make CPU-based sharding practical.

\subsection{Casefolding}

bcachefs supports case-insensitive filenames via the Linux kernel's Unicode
subsystem (\texttt{CONFIG\_UNICODE}). The \texttt{casefold} option is a
per-directory property that is inherited by newly created subdirectories.

\textbf{Important}: Casefolding can only be enabled on an empty directory.
Existing entries cannot be converted because they would need to be rehashed
with case-folded names. On disk, casefolded directories store both the original
filename and its case-folded form; the case-folded form is used for lookups
while the original is preserved for display.

The Unicode version used for case folding is hardcoded to Unicode 12.1.0. The
kernel will fail to mount (or use an older version correctly) if built with
different Unicode tables.

\section{Management}

\subsection{Formatting}

To format a new bcachefs filesystem use the subcommand \texttt{bcachefs
format}, or \texttt{mkfs.bcachefs}. All persistent filesystem-wide options can
be specified at format time. For an example of a multi device filesystem with
compression, encryption, replication and writeback caching:
\begin{quote} \begin{verbatim}
bcachefs format --compression=lz4               \
                --encrypted                     \
                --replicas=2                    \
                --label=ssd.ssd1 /dev/sda       \
                --label=ssd.ssd2 /dev/sdb       \
                --label=hdd.hdd1 /dev/sdc       \
                --label=hdd.hdd2 /dev/sdd       \
                --label=hdd.hdd3 /dev/sde       \
                --label=hdd.hdd4 /dev/sdf       \
                --foreground_target=ssd	        \
                --promote_target=ssd            \
                --background_target=hdd
\end{verbatim} \end{quote}

\subsection{Mounting}

To mount a multi device filesystem, there are two options. You can specify all
component devices, separated by colons, e.g.
\begin{quote} \begin{verbatim}
mount -t bcachefs /dev/sda:/dev/sdb:/dev/sdc /mnt
\end{verbatim} \end{quote}
Or, use the mount.bcachefs tool to mount by filesystem UUID or label:
\begin{quote} \begin{verbatim}
mount -t bcachefs UUID=<uuid> /mnt
mount -t bcachefs LABEL=<label> /mnt
\end{verbatim} \end{quote}

No special handling is needed for recovering from unclean shutdown. Journal
replay happens automatically, and diagnostic messages in the dmesg log will
indicate whether recovery was from clean or unclean shutdown.

The \texttt{-o degraded} option will allow a filesystem to be mounted without
all the devices, but will fail if data would be missing. The
\texttt{-o degraded=very} option can be used to attempt mounting when data
would be missing.

The \texttt{-o verbose} option enables additional log output during the mount
process.

\subsection{Fsck}

The \texttt{bcachefs fsck} subcommand (also available as \texttt{fsck.bcachefs})
can run fsck in userspace (\texttt{-K}), in the kernel at mount time
(\texttt{-k}), or online on a mounted filesystem, selecting the mode
automatically or via command line options. The in-kernel offline mode is useful
when an older bcachefs-tools binary cannot handle the filesystem version. In
all cases the exact same fsck implementation runs - the recovery passes
described above. Running in userspace allows stopping with ctrl-c and prompting
for fixes; \texttt{bcachefs fsck -y} auto-fixes, \texttt{-n} runs read-only.

The \texttt{-o nochanges} mount option disallows any writes to the underlying
devices, pinning dirty data in memory if journal replay is necessary. This is
the primary tool for safe inspection of a damaged filesystem: it allows
recovery to run and metadata to be read without risking further damage. It is
also used for testing on-disk format upgrades (\texttt{-o version\_upgrade})
before committing to them. \texttt{bcachefs fsck -n} implies
\texttt{-o nochanges}; \texttt{bcachefs fsck -ny} can be used to test
filesystem repair in dry-run mode.

When running fsck at mount time via \texttt{-o fsck}, use
\texttt{-o fix\_errors} to auto-fix. Before committing to a repair, test it
with \texttt{-o ro,nochanges,fsck,fix\_errors} first: this runs the full repair
in memory without writing to disk, allowing the filesystem to be mounted and
its contents verified before repeating without \texttt{nochanges}.

Online fsck runs the validation passes that are marked as safe for concurrent
use (\texttt{PASS\_ONLINE}) on a mounted, read-write filesystem. Since version
1.19, the default error action is \texttt{errors=fix\_safe}: when the filesystem
detects a metadata inconsistency during normal operation - for example, a dirent
pointing to a nonexistent inode - it automatically schedules the relevant repair
pass to run online in the background, without requiring user intervention, a
remount, or taking the filesystem offline. Errors that are not safe to
auto-repair cause the filesystem to go emergency read-only, with a message
suggesting to run fsck and report the issue to developers so it can be marked
for self-healing in future versions.

\subsection{Self-healing}

Bcachefs classifies metadata errors into three tiers based on severity and
whether a safe automatic repair is known:

\begin{description}
\item[Autofix errors (\texttt{FSCK\_AUTOFIX})] Inconsistencies with a
  well-understood, always-safe repair that can be applied immediately inline
  during normal operation, without interrupting I/O or requiring a recovery
  pass. An example is a missing journal entry replica marker: the repair is
  simply to write the correct marker, with no data at risk.

\item[Fixable errors (\texttt{FSCK\_CAN\_FIX})] Inconsistencies that have a
  known repair procedure but require a dedicated recovery pass to execute
  safely. Under \texttt{errors=fix\_safe} (the default since version 1.19),
  detecting such an error schedules the relevant recovery pass to run online
  in the background. The pass runs concurrently with normal filesystem
  operation without taking the filesystem offline, provided it carries the
  \texttt{PASS\_ONLINE} flag. Errors detected during recovery that would
  rewind the recovery sequence are handled specially: recovery restarts from
  the affected pass rather than aborting.

\item[Inconsistent errors] Structural corruption for which no safe automatic
  repair is known, or where continuing operation risks further data loss. These
  trigger emergency read-only regardless of the \texttt{errors} setting (except
  \texttt{errors=continue}, which logs the error and proceeds). The filesystem
  logs a message directing the administrator to run fsck. If a repair procedure
  is subsequently developed and deemed safe, the error can be reclassified to
  \texttt{FSCK\_CAN\_FIX} in a future kernel release.
\end{description}

The \texttt{errors} mount option controls the policy for the second and third
tiers; see \S\ref{sec:error-actions}. The default \texttt{fix\_safe} enables
background self-repair for fixable errors while taking the conservative
read-only path for inconsistent ones.

Every error detection event is recorded as a persistent count in the
superblock's errors section. These counts survive across reboots and accumulate
over the filesystem's lifetime, providing an audit trail of what has been
detected and repaired. The counts are visible in \texttt{bcachefs show-super}
output (included by default) and via the sysfs \texttt{errors} file. A count
of zero for an error type means it has never been detected; a nonzero count
means it was detected and, if the error type has \texttt{FSCK\_CAN\_FIX},
automatically scheduled for repair.

\subsection{Status of data}

The \texttt{bcachefs fs usage} may be used to display filesystem usage broken
out in various ways. Data usage is broken out by type: superblock, journal,
btree, data, cached data, and parity, and by which sets of devices extents are
replicated across. We also give per-device usage which includes fragmentation
due to partially used buckets.

\subsection{Journal}

The journal has a number of tunables that affect filesystem performance. Journal
commits are fairly expensive operations as they require issuing FLUSH and FUA
operations to the underlying devices. By default, we issue a journal flush one
second after a filesystem update has been done; this is controlled with the
\texttt{journal\_flush\_delay} option, which takes a parameter in milliseconds.

Filesystem sync and fsync operations issue journal flushes; this can be disabled
with the \texttt{journal\_flush\_disabled} option - the
\texttt{journal\_flush\_delay} option will still apply, and in the event of a
system crash we will never lose more than (by default) one second of work. This
option may be useful on a personal workstation or laptop, and perhaps less
appropriate on a server.

The journal reclaim thread runs in the background, kicking off btree node writes
and btree key cache flushes to free up space in the journal. Even in the absence
of space pressure it will run slowly in the background: this is controlled by
the \texttt{journal\_reclaim\_delay} parameter, with a default of 100
milliseconds.

The journal should be sized sufficiently that bursts of activity do not fill up
the journal too quickly; also, a larger journal means that we can queue up
larger btree writes. The \texttt{bcachefs device resize-journal} can be used for
resizing the journal on disk on a particular device - it can be used on a
mounted or unmounted filesystem. Note that post-format journal resizing
inherently fragments the journal (allocating new buckets that may not be
contiguous), so it is best done on a relatively empty device and as few times
as possible; reducing journal size is not currently implemented. Current journal
utilization can be inspected via
\texttt{/sys/fs/bcachefs/<uuid>/internal/journal\_debug}, which reports dirty
entries out of total capacity, sequence numbers, watermark level, and reclaim
statistics.

\subsubsection{Journal rewind}

The \texttt{journal\_rewind} mount option accepts a journal sequence number and
rolls back the most important filesystem metadata to that point, discarding
subsequent updates and fixing up inconsistencies afterwards. This is a tool
specifically for reverting changes from a bug in recovery that irreversibly
corrupted or deleted metadata, or (combined with \texttt{-o nochanges}) for
attempting to recover recently deleted files. It drops all leaf-level btree
updates to non-allocator btrees at or above the specified sequence number, then
automatically triggers a full fsck to rebuild allocation metadata.

Journal rewind is used in conjunction with \texttt{bcachefs list\_journal} to
identify the sequence number to rewind to. The
\texttt{journal\_transaction\_names} option (on by default) is required for
journal rewind: it controls whether \texttt{BCH\_JSET\_ENTRY\_overwrite}
entries (the old values being replaced) are stored in the journal, which is
what rewind replays to undo changes. It also records which function created
each journal entry, making it easier to identify which transactions caused a
problem. Since rewind operates on metadata only, data in
buckets that have been reused (generation incremented and overwritten with new
data) is unrecoverable, and \texttt{nocow} data cannot be rolled back at all.
The journal records allocator updates, so \texttt{list\_journal -b alloc} can
identify when specific buckets were recycled. The rewind window is bounded by
the journal size: only sequences between \texttt{last\_seq} (oldest entry still
on disk) and the current sequence are available, so a larger journal provides a
deeper rewind history.

\subsection{Device management}

\subsubsection{Filesystem resize}

A filesystem can be resized on a particular device with the
\texttt{bcachefs device resize} subcommand. Currently only growing is supported,
not shrinking.

\subsubsection{Device add/removal}

The following subcommands exist for adding and removing devices from a mounted
filesystem:
\begin{itemize}
	\item \texttt{bcachefs device add}: Formats and adds a new device to an
		existing filesystem.
	\item \texttt{bcachefs device remove}: Permanently removes a device from
		an existing filesystem. Offline devices can be removed by index
		(found in the superblock via \texttt{show-super}).
	\item \texttt{bcachefs device online}: Connects a device to a running
		filesystem that was mounted without it (i.e. in degraded mode)
	\item \texttt{bcachefs device offline}: Disconnects a device from a
		mounted filesystem without removing it.
	\item \texttt{bcachefs device evacuate}: Migrates data off of a
		particular device to prepare for removal, setting it read-only
		if necessary.
	\item \texttt{bcachefs device set-state}: Changes the state of a member
		device: one of rw (readwrite), ro (readonly), evacuating, or spare.

		An evacuating device is considered to have 0 durability, and
		replicas on that device won't be counted towards the number of
		replicas an extent should have - as a result, reconcile will
		write new replicas elsewhere and drop the old ones, evacuating
		all data from the device. bcachefs will still attempt to read
		from evacuating devices.
\end{itemize}

The \texttt{bcachefs device remove}, \texttt{bcachefs device offline} and
\texttt{bcachefs device set-state} commands take force options for when they
would leave the filesystem degraded or with data missing.

\subsubsection{Device labeling}
Devices can be labeled after formatting by setting \\
\texttt{/sys/fs/bcachefs/<uuid>/dev-<device-id>/label}

\subsection{Data management}

\subsubsection{Reconcile}

The reconcile subsystem (formerly ``rebalance'') runs in the background to
ensure all data and metadata matches configured IO path options: replication
count (\texttt{data\_replicas}, \texttt{metadata\_replicas}), compression,
checksum type, erasure coding, and device targets (\texttt{foreground\_target},
\texttt{metadata\_target}, etc.). When options change or devices are added or
removed, reconcile propagates these changes to existing data and metadata by
moving, rewriting, or recompressing extents and btree nodes.

Reconcile uses 6 dedicated btrees for work tracking:
\texttt{reconcile\_work} and \texttt{reconcile\_hipri} for normal and
high-priority logical work, \texttt{reconcile\_pending} for work that failed
due to insufficient space or devices, \texttt{reconcile\_scan} for option
change propagation via scan cookies, and physical variants
(\texttt{reconcile\_work\_phys}, \texttt{reconcile\_hipri\_phys}) that track
work by device LBA for efficient processing on rotational devices.

Work enters the system when filesystem or inode options change (triggering a
scan), or when extent triggers detect a mismatch between current data placement
and desired options. The reconcile thread processes work in priority order:
high-priority metadata first (under-replicated or evacuating), then high-priority
data, then normal metadata (moving stray metadata to \texttt{metadata\_target}),
then normal data, then pending retries (only after device configuration changes).
The \texttt{reconcile\_pending} btree solves the long-standing ``rebalance
spinning'' problem, where the old rebalance thread would burn CPU retrying
moves that could not complete (e.g.\ because the target was full). A practical
consequence: you can create a single-device filesystem with
\texttt{replicas=2}, and all data will be marked as degraded with 1x
availability; when a second device is added, reconcile automatically
replicates everything to 2x without further user intervention.

The \texttt{bcachefs reconcile status} command shows current reconcile
progress, and \texttt{bcachefs reconcile wait} blocks until reconcile
completes for specified work types. The \texttt{reconcile\_enabled} and
\texttt{reconcile\_on\_ac\_only} options control when reconcile runs.

\subsubsection{Scrub}

The \texttt{bcachefs data scrub} command validates all data with checksums,
fixing bitrot when a valid copy can be found. Affected file paths are currently
logged to the kernel log only.

\section{Command reference}

The \texttt{bcachefs} tool provides filesystem management through grouped
subcommands. Run \texttt{bcachefs} with no arguments for a command summary,
or \texttt{bcachefs <command> --help} for detailed usage.

\bchdoc{cli-reference}

\section{Options}
\label{sec:options}

Most bcachefs options can be set filesystem wide, and a significant subset can
also be set on inodes (files and directories), overriding the global defaults.
Filesystem wide options may be set when formatting, when mounting, or at runtime
via \texttt{/sys/fs/bcachefs/<uuid>/options/}. When set at runtime via sysfs,
the persistent options in the superblock are updated as well; when options are
passed as mount parameters the persistent options are unmodified. Additionally
some of the filesystem wide options can be set via \texttt{bcachefs set-fs-option}

\subsection{Options reference}

\bchdoc{opts-table}

\subsection{File and directory options}

A subset of filesystem options can be set on individual files and directories,
overriding the filesystem-wide defaults. The per-inode options are:
\texttt{data\_checksum}, \texttt{compression}, \texttt{background\_compression},
\texttt{data\_replicas}, \texttt{foreground\_target},
\texttt{background\_target}, \texttt{promote\_target}, \texttt{metadata\_target},
\texttt{erasure\_code}, \texttt{nocow}, \texttt{casefold}, and
\texttt{inodes\_32bit}. Note that \texttt{casefold} and \texttt{inodes\_32bit}
can only be toggled on empty directories.

Each inode stores its own copy of these options so that reads never need to
walk parent directories. Each option has a separate ``defined'' flag that
records whether the option was explicitly set on this inode; options without
this flag are propagated from the parent directory on inode creation and rename.
When renaming a directory would cause inherited attributes to change we fail the
rename with -EXDEV, causing userspace to do the rename file by file so that
inherited attributes stay consistent.

Inode options are available as extended attributes. The options that have been
explicitly set are available under the \texttt{bcachefs} namespace, and the
effective options (explicitly set and inherited options) are available under the
\texttt{bcachefs\_effective} namespace. Examples of listing options with the
getfattr command:

\begin{quote} \begin{verbatim}
$ getfattr -d -m '^bcachefs\.' filename
$ getfattr -d -m '^bcachefs_effective\.' filename
\end{verbatim} \end{quote}

Options may be set via the extended attribute interface, but it is preferable to
use the \texttt{bcachefs set-file-option} command as it will correctly
propagate options recursively.

\subsection{Inode number sharding}

On systems with many CPUs, inode number allocation can become a contention
point: every file creation needs a new inode number, and a single global cursor
serializes all CPUs. The \texttt{shard\_inode\_numbers\_bits} option (0-8,
set at format time to the log2 of the number of online CPUs if not specified)
partitions the inode number space by CPU ID, giving each CPU its own
allocation range. This eliminates cross-CPU contention at the cost of
non-sequential inode numbers.

\subsection{Error actions}
\label{sec:error-actions}
The \texttt{errors} option controls how the filesystem responds to metadata
inconsistencies. Valid error actions are:
\bchdoc{error-actions}

\subsection{Checksum types}
Valid checksum types are:
\bchdoc{csum-opts}

\subsection{Compression types}
Valid compression types are:
\bchdoc{compression-opts}

\subsection{String hash types}
Valid hash types for string hash tables are:
\bchdoc{str-hash-opts}

\section{Debugging tools}

\subsection{Sysfs interface}

Mounted filesystems are available in sysfs at \texttt{/sys/fs/bcachefs/<uuid>/}
with various options, performance counters and internal debugging aids.

\subsubsection{Options}

Filesystem options may be viewed and changed via \\
\texttt{/sys/fs/bcachefs/<uuid>/options/}, and settings changed via sysfs will
be persistently changed in the superblock as well.

\subsubsection{Time stats}
\label{sec:timestats}

bcachefs tracks the latency and frequency of various operations and events, with
quantiles for latency/duration in the
\texttt{/sys/fs/bcachefs/<uuid>/time\_stats/} directory.

\bchdoc{time-stats}

\subsubsection{Persistent counters}
\label{sec:counters}

bcachefs tracks persistent counters in the superblock that survive across mounts.
These are available in \texttt{/sys/fs/bcachefs/<uuid>/counters/} and count
either events or sectors depending on type.

\bchdoc{counters}

\subsubsection{Internals}

\begin{description}
	\item \texttt{btree\_cache} \\
		Shows information on the btree node cache: number of cached
		nodes, number of dirty nodes, and whether the cannibalize lock
		(for reclaiming cached nodes to allocate new nodes) is held.

	\item \texttt{btree\_key\_cache} \\
		Prints information on the btree key cache: number of freed keys
		(which must wait for an SRCU barrier to complete before being
		freed), number of cached keys, and number of dirty keys.

	\item \texttt{journal\_debug} \\
		Prints a variety of internal journal state.

	\item \texttt{new\_stripes} \\
		Lists new erasure-coded stripes being created.

	\item \texttt{open\_buckets} \\
		Lists buckets currently being written to, along with data type
		and refcount.

	\item \texttt{io\_timers\_read} \\
	\item \texttt{io\_timers\_write} \\
		Lists outstanding IO timers - timers that wait on total reads or
		writes to the filesystem.

	\item \texttt{trigger\_journal\_commit} \\
		Echoing to this file triggers a journal commit.

	\item \texttt{trigger\_journal\_flush} \\
		Echoing to this file flushes all journal pins (forcing dirty
		btree nodes and key cache entries to be written) and then
		issues a journal meta write.

	\item \texttt{trigger\_gc} \\
		Echoing to this file causes the GC code to recalculate each
		bucket's oldest\_gen field.

	\item \texttt{trigger\_btree\_cache\_shrink} \\
		Echoing to this file prunes the btree node cache.

\end{description}

\subsubsection{Unit and performance tests}

Echoing into \texttt{/sys/fs/bcachefs/<uuid>/perf\_test} runs various low level
btree tests, some intended as unit tests and others as performance tests. The
syntax is
\begin{quote} \begin{verbatim}
	echo <test_name> <nr_iterations> <nr_threads> > perf_test
\end{verbatim} \end{quote}

When complete, the elapsed time will be printed in the dmesg log. The full list
of tests that can be run can be found near the bottom of
\texttt{fs/bcachefs/tests.c}.

\subsection{Debugfs interface}

Various internal debugging files are available under
\texttt{/sys/kernel/debug/bcachefs/<uuid>/}.

The \texttt{btrees/} subdirectory contains one directory per btree (e.g.
\texttt{btrees/extents/}, \texttt{btrees/inodes/}), each with three files:

\begin{description}
	\item \texttt{keys} \\
		Entire btree contents, one key per line.

	\item \texttt{formats} \\
		Information about each btree node: the size of the packed bkey
		format, how full each btree node is, number of packed and
		unpacked keys, and number of nodes and failed nodes in the
		in-memory search trees.

	\item \texttt{bfloat-failed} \\
		For each sorted set of keys in a btree node, we construct a
		binary search tree in eytzinger layout with compressed keys.
		Sometimes we aren't able to construct a correct compressed
		search key, which results in slower lookups; this file lists the
		keys that resulted in these failed nodes.
\end{description}

The remaining files in the debugfs root are:

\begin{description}
	\item \texttt{btree\_transactions} \\
		Lists each running btree transaction that has locks held,
		listing which nodes they have locked and what type of lock, what
		node (if any) the process is blocked attempting to lock, and
		where the btree transaction was invoked from.

	\item \texttt{btree\_updates} \\
		Lists outstanding interior btree updates: the mode (nothing
		updated yet, or updated a btree node, or wrote a new btree root,
		or was reparented by another btree update), whether its new
		btree nodes have finished writing, its embedded closure's
		refcount (while nonzero, the btree update is still waiting), and
		the pinned journal sequence number.

	\item \texttt{journal\_pins} \\
		Lists what is preventing each journal entry from being
		reclaimed: dirty btree nodes that still reference that
		sequence number, key cache entries waiting to be flushed,
		or in-progress operations that need the entry to remain
		available for crash recovery. Useful for diagnosing journal
		space exhaustion.

	\item \texttt{btree\_deadlock} \\
		Checks for and reports btree lock deadlock cycles among
		transactions.

	\item \texttt{cached\_btree\_nodes} \\
		Lists btree nodes currently in the btree node cache.

	\item \texttt{btree\_transaction\_stats} \\
		Per-transaction-function statistics: lock hold times, restart
		counts, and maximum memory usage.

	\item \texttt{btree\_node\_scan} \\
		Results from the last scan for btree nodes on raw devices,
		used during catastrophic recovery.

	\item \texttt{write\_points} \\
		Lists active write points and the buckets they are currently
		writing to.
\end{description}

\subsection{Listing and dumping filesystem metadata}

\subsubsection{bcachefs show-super}

This subcommand is used for examining and printing bcachefs superblocks. It
takes two optional parameters:
\begin{description}
	\item \texttt{-l}: Print superblock layout, which records the amount of
		space reserved for the superblock and the locations of the
		backup superblocks.
	\item \texttt{-f, --fields=(fields)}: List of superblock sections to
		print, \texttt{all} to print all sections.
\end{description}

\subsubsection{bcachefs list}

This subcommand gives access to the same functionality as the debugfs interface,
listing btree nodes and contents, but for offline filesystems.

\subsubsection{bcachefs list\_journal}

This subcommand lists the contents of the journal for offline filesystems. The
journal is a complete record of every btree update, accounting delta, log
message, and timestamp, ordered by sequence number. Each entry can be traced
back to the transaction function that created it (when
\texttt{journal\_transaction\_names} is enabled). Filtering by btree, sequence
range, or transaction name makes it possible to reconstruct the exact sequence
of events leading to a problem - and to identify the right sequence number for
journal rewind.

\subsubsection{bcachefs dump}

This subcommand can dump all metadata in a filesystem (including multi device
filesystems) as qcow2 images: when encountering issues that \texttt{fsck} can
not recover from and need attention from the developers, this makes it possible
to send the developers only the required metadata. Encrypted filesystems must
first be unlocked with \texttt{bcachefs remove-passphrase}.

\section{ioctl interface}

This section documents bcachefs-specific ioctls:

\begin{description}
	\item \texttt{BCH\_IOCTL\_QUERY\_UUID} \\
		Returns the UUID of the filesystem: used to find the sysfs
		directory given a path to a mounted filesystem.

	\item \texttt{BCH\_IOCTL\_FS\_USAGE} \\
		Obsolete. Formerly queried filesystem usage; replaced by
		\texttt{BCH\_IOCTL\_QUERY\_ACCOUNTING}.

	\item \texttt{BCH\_IOCTL\_DEV\_USAGE} \\
		Obsolete. Formerly queried per-device usage; replaced by
		\texttt{BCH\_IOCTL\_QUERY\_ACCOUNTING}.

	\item \texttt{BCH\_IOCTL\_READ\_SUPER} \\
		Returns the filesystem superblock, and optionally the superblock
		for a particular device given that device's index.

	\item \texttt{BCH\_IOCTL\_DISK\_ADD} \\
		Given a path to a device, adds it to a mounted and running
		filesystem. The device must already have a bcachefs superblock;
		options and parameters are read from the new device's superblock
		and added to the member info section of the existing
		filesystem's superblock.

	\item \texttt{BCH\_IOCTL\_DISK\_REMOVE} \\
		Given a path to a device or a device index, attempts to remove
		it from a mounted and running filesystem. This operation
		requires walking the btree to remove all references to this
		device, and may fail if data would become degraded or lost,
		unless appropriate force flags are set.

	\item \texttt{BCH\_IOCTL\_DISK\_ONLINE} \\
		Given a path to a device that is a member of a running
		filesystem (in degraded mode), brings it back online.

	\item \texttt{BCH\_IOCTL\_DISK\_OFFLINE} \\
		Given a path or device index of a device in a multi device
		filesystem, attempts to close it without removing it, so that
		the device may be re-added later and the contents will still be
		available.

	\item \texttt{BCH\_IOCTL\_DISK\_SET\_STATE} \\
		Given a path or device index of a device in a multi device
		filesystem, attempts to set its state to one of read-write,
		read-only, evacuating, or spare. Takes flags to force if the
		filesystem would become degraded.

	\item \texttt{BCH\_IOCTL\_DISK\_GET\_IDX} \\
	\item \texttt{BCH\_IOCTL\_DISK\_RESIZE} \\
	\item \texttt{BCH\_IOCTL\_DISK\_RESIZE\_JOURNAL} \\
	\item \texttt{BCH\_IOCTL\_DATA} \\
		Starts a data job, which walks all data and/or metadata in a
		filesystem, performing some operations on each btree
		node and extent. Returns a file descriptor which can be read
		from to get the current status of the job, and closing the file
		descriptor (i.e. on process exit stops the data job.

	\item \texttt{BCH\_IOCTL\_SUBVOLUME\_CREATE} \\
	\item \texttt{BCH\_IOCTL\_SUBVOLUME\_DESTROY} \\

	\item \texttt{BCH\_IOCTL\_FSCK\_OFFLINE} \\
		Runs offline fsck on an unmounted filesystem via ioctl,
		reporting progress and results through a file descriptor.

	\item \texttt{BCH\_IOCTL\_FSCK\_ONLINE} \\
		Runs online fsck on a mounted filesystem, executing the
		same recovery passes that \texttt{-o fsck} would run at
		mount time.

	\item \texttt{BCH\_IOCTL\_QUERY\_ACCOUNTING} \\
		Queries disk accounting data from the accounting btree:
		per-replica-set usage, per-device usage, compression
		ratios, and other counters. Replaces the obsolete
		\texttt{FS\_USAGE} and \texttt{DEV\_USAGE} ioctls.

	\item \texttt{BCH\_IOCTL\_SUBVOLUME\_LIST} \\
		Lists subvolumes in a filesystem.

	\item \texttt{BCH\_IOCTL\_SNAPSHOT\_TREE} \\
		Queries the full snapshot tree with per-node disk accounting.

	\item \texttt{BCHFS\_IOC\_REINHERIT\_ATTRS} \\
\end{description}

Most disk management ioctls (\texttt{DISK\_ADD}, \texttt{DISK\_REMOVE},
\texttt{DISK\_ONLINE}, \texttt{DISK\_OFFLINE}, \texttt{DISK\_SET\_STATE},
\texttt{DISK\_RESIZE}, \texttt{DISK\_RESIZE\_JOURNAL},
\texttt{SUBVOLUME\_CREATE}, \texttt{SUBVOLUME\_DESTROY}) have v2 variants that
add structured error reporting via an embedded error message buffer.

\section{On disk format}

\subsection{Superblock}

The superblock is the first thing to be read when accessing a bcachefs
filesystem. It is located 4kb from the start of the device, with redundant
copies elsewhere - typically one immediately after the first superblock, and one
at the end of the device.

The \texttt{bch\_sb\_layout} records the amount of space reserved for the
superblock as well as the locations of all the superblocks. It is included with
every superblock, and additionally written 3584 bytes from the start of the
device (512 bytes before the first superblock).

Most of the superblock is identical across each device. The exceptions are the
\texttt{dev\_idx} field, and the journal section which gives the location of the
journal.

The main section of the superblock contains UUIDs, version numbers, number of
devices within the filesystem and device index, block size, filesystem creation
time, and various options and settings. The superblock also has a number of
variable length sections:

\bchdoc{sb-fields}

Several field types have versioned variants (e.g. \texttt{members\_v1}/\texttt{v2},
\texttt{replicas\_v0}/\texttt{replicas}, \texttt{journal}/\texttt{journal\_v2})
for backward compatibility; the kernel reads both old and new formats but writes
only the current version.

\subsection{Journal}

Every journal write (\texttt{struct jset}) contains a list of entries:
\texttt{struct jset\_entry}. Below are listed the various journal entry types.

\bchdoc{jset-entry-types}

\subsection{Btrees}

bcachefs uses 28 separate btrees for different data types, each identified by a
numeric ID. Below is the complete list:

\bchdoc{btree-ids}

Btrees marked as ``snapshot-aware'' include a snapshot ID in every key position,
allowing different snapshot versions to coexist within the same btree. Some
btrees are ``write-buffered'': updates are batched in memory and flushed
periodically rather than applied directly to btree nodes, reducing write
amplification for high-frequency per-IO metadata (backpointers, LRU timestamps,
accounting counters). Write-buffered updates are unordered and
eventually-consistent: the btree does not reflect pending updates until the
next flush, and flushing sorts by key position, discarding temporal ordering.
Flushing deduplicates redundant updates, then walks btree nodes in order for
efficient bulk insertion; this sort-merge-sweep is inherently single-threaded,
making its efficiency critical for multithreaded workloads.

\subsection{Btree keys}

\bchdoc{bkey-types}

\subsection{Metadata versions}

Each on-disk format change is identified by a version number using
\texttt{BCH\_VERSION(major, minor)}. The version history records what changed
and when, serving as a changelog for the on-disk format. The filesystem's
superblock records both the current version and the minimum version of any data
still on disk, allowing compatibility code to be safely dropped.

\bchdoc{metadata-versions}

\end{document}
