mirror of https://github.com/Qortal/Brooklyn
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
333 lines
13 KiB
333 lines
13 KiB
.. SPDX-License-Identifier: GPL-2.0-only |
|
|
|
======== |
|
dm-clone |
|
======== |
|
|
|
Introduction |
|
============ |
|
|
|
dm-clone is a device mapper target which produces a one-to-one copy of an |
|
existing, read-only source device into a writable destination device: It |
|
presents a virtual block device which makes all data appear immediately, and |
|
redirects reads and writes accordingly. |
|
|
|
The main use case of dm-clone is to clone a potentially remote, high-latency, |
|
read-only, archival-type block device into a writable, fast, primary-type device |
|
for fast, low-latency I/O. The cloned device is visible/mountable immediately |
|
and the copy of the source device to the destination device happens in the |
|
background, in parallel with user I/O. |
|
|
|
For example, one could restore an application backup from a read-only copy, |
|
accessible through a network storage protocol (NBD, Fibre Channel, iSCSI, AoE, |
|
etc.), into a local SSD or NVMe device, and start using the device immediately, |
|
without waiting for the restore to complete. |
|
|
|
When the cloning completes, the dm-clone table can be removed altogether and be |
|
replaced, e.g., by a linear table, mapping directly to the destination device. |
|
|
|
The dm-clone target reuses the metadata library used by the thin-provisioning |
|
target. |
|
|
|
Glossary |
|
======== |
|
|
|
Hydration |
|
The process of filling a region of the destination device with data from |
|
the same region of the source device, i.e., copying the region from the |
|
source to the destination device. |
|
|
|
Once a region gets hydrated we redirect all I/O regarding it to the destination |
|
device. |
|
|
|
Design |
|
====== |
|
|
|
Sub-devices |
|
----------- |
|
|
|
The target is constructed by passing three devices to it (along with other |
|
parameters detailed later): |
|
|
|
1. A source device - the read-only device that gets cloned and source of the |
|
hydration. |
|
|
|
2. A destination device - the destination of the hydration, which will become a |
|
clone of the source device. |
|
|
|
3. A small metadata device - it records which regions are already valid in the |
|
destination device, i.e., which regions have already been hydrated, or have |
|
been written to directly, via user I/O. |
|
|
|
The size of the destination device must be at least equal to the size of the |
|
source device. |
|
|
|
Regions |
|
------- |
|
|
|
dm-clone divides the source and destination devices in fixed sized regions. |
|
Regions are the unit of hydration, i.e., the minimum amount of data copied from |
|
the source to the destination device. |
|
|
|
The region size is configurable when you first create the dm-clone device. The |
|
recommended region size is the same as the file system block size, which usually |
|
is 4KB. The region size must be between 8 sectors (4KB) and 2097152 sectors |
|
(1GB) and a power of two. |
|
|
|
Reads and writes from/to hydrated regions are serviced from the destination |
|
device. |
|
|
|
A read to a not yet hydrated region is serviced directly from the source device. |
|
|
|
A write to a not yet hydrated region will be delayed until the corresponding |
|
region has been hydrated and the hydration of the region starts immediately. |
|
|
|
Note that a write request with size equal to region size will skip copying of |
|
the corresponding region from the source device and overwrite the region of the |
|
destination device directly. |
|
|
|
Discards |
|
-------- |
|
|
|
dm-clone interprets a discard request to a range that hasn't been hydrated yet |
|
as a hint to skip hydration of the regions covered by the request, i.e., it |
|
skips copying the region's data from the source to the destination device, and |
|
only updates its metadata. |
|
|
|
If the destination device supports discards, then by default dm-clone will pass |
|
down discard requests to it. |
|
|
|
Background Hydration |
|
-------------------- |
|
|
|
dm-clone copies continuously from the source to the destination device, until |
|
all of the device has been copied. |
|
|
|
Copying data from the source to the destination device uses bandwidth. The user |
|
can set a throttle to prevent more than a certain amount of copying occurring at |
|
any one time. Moreover, dm-clone takes into account user I/O traffic going to |
|
the devices and pauses the background hydration when there is I/O in-flight. |
|
|
|
A message `hydration_threshold <#regions>` can be used to set the maximum number |
|
of regions being copied, the default being 1 region. |
|
|
|
dm-clone employs dm-kcopyd for copying portions of the source device to the |
|
destination device. By default, we issue copy requests of size equal to the |
|
region size. A message `hydration_batch_size <#regions>` can be used to tune the |
|
size of these copy requests. Increasing the hydration batch size results in |
|
dm-clone trying to batch together contiguous regions, so we copy the data in |
|
batches of this many regions. |
|
|
|
When the hydration of the destination device finishes, a dm event will be sent |
|
to user space. |
|
|
|
Updating on-disk metadata |
|
------------------------- |
|
|
|
On-disk metadata is committed every time a FLUSH or FUA bio is written. If no |
|
such requests are made then commits will occur every second. This means the |
|
dm-clone device behaves like a physical disk that has a volatile write cache. If |
|
power is lost you may lose some recent writes. The metadata should always be |
|
consistent in spite of any crash. |
|
|
|
Target Interface |
|
================ |
|
|
|
Constructor |
|
----------- |
|
|
|
:: |
|
|
|
clone <metadata dev> <destination dev> <source dev> <region size> |
|
[<#feature args> [<feature arg>]* [<#core args> [<core arg>]*]] |
|
|
|
================ ============================================================== |
|
metadata dev Fast device holding the persistent metadata |
|
destination dev The destination device, where the source will be cloned |
|
source dev Read only device containing the data that gets cloned |
|
region size The size of a region in sectors |
|
|
|
#feature args Number of feature arguments passed |
|
feature args no_hydration or no_discard_passdown |
|
|
|
#core args An even number of arguments corresponding to key/value pairs |
|
passed to dm-clone |
|
core args Key/value pairs passed to dm-clone, e.g. `hydration_threshold |
|
256` |
|
================ ============================================================== |
|
|
|
Optional feature arguments are: |
|
|
|
==================== ========================================================= |
|
no_hydration Create a dm-clone instance with background hydration |
|
disabled |
|
no_discard_passdown Disable passing down discards to the destination device |
|
==================== ========================================================= |
|
|
|
Optional core arguments are: |
|
|
|
================================ ============================================== |
|
hydration_threshold <#regions> Maximum number of regions being copied from |
|
the source to the destination device at any |
|
one time, during background hydration. |
|
hydration_batch_size <#regions> During background hydration, try to batch |
|
together contiguous regions, so we copy data |
|
from the source to the destination device in |
|
batches of this many regions. |
|
================================ ============================================== |
|
|
|
Status |
|
------ |
|
|
|
:: |
|
|
|
<metadata block size> <#used metadata blocks>/<#total metadata blocks> |
|
<region size> <#hydrated regions>/<#total regions> <#hydrating regions> |
|
<#feature args> <feature args>* <#core args> <core args>* |
|
<clone metadata mode> |
|
|
|
======================= ======================================================= |
|
metadata block size Fixed block size for each metadata block in sectors |
|
#used metadata blocks Number of metadata blocks used |
|
#total metadata blocks Total number of metadata blocks |
|
region size Configurable region size for the device in sectors |
|
#hydrated regions Number of regions that have finished hydrating |
|
#total regions Total number of regions to hydrate |
|
#hydrating regions Number of regions currently hydrating |
|
#feature args Number of feature arguments to follow |
|
feature args Feature arguments, e.g. `no_hydration` |
|
#core args Even number of core arguments to follow |
|
core args Key/value pairs for tuning the core, e.g. |
|
`hydration_threshold 256` |
|
clone metadata mode ro if read-only, rw if read-write |
|
|
|
In serious cases where even a read-only mode is deemed |
|
unsafe no further I/O will be permitted and the status |
|
will just contain the string 'Fail'. If the metadata |
|
mode changes, a dm event will be sent to user space. |
|
======================= ======================================================= |
|
|
|
Messages |
|
-------- |
|
|
|
`disable_hydration` |
|
Disable the background hydration of the destination device. |
|
|
|
`enable_hydration` |
|
Enable the background hydration of the destination device. |
|
|
|
`hydration_threshold <#regions>` |
|
Set background hydration threshold. |
|
|
|
`hydration_batch_size <#regions>` |
|
Set background hydration batch size. |
|
|
|
Examples |
|
======== |
|
|
|
Clone a device containing a file system |
|
--------------------------------------- |
|
|
|
1. Create the dm-clone device. |
|
|
|
:: |
|
|
|
dmsetup create clone --table "0 1048576000 clone $metadata_dev $dest_dev \ |
|
$source_dev 8 1 no_hydration" |
|
|
|
2. Mount the device and trim the file system. dm-clone interprets the discards |
|
sent by the file system and it will not hydrate the unused space. |
|
|
|
:: |
|
|
|
mount /dev/mapper/clone /mnt/cloned-fs |
|
fstrim /mnt/cloned-fs |
|
|
|
3. Enable background hydration of the destination device. |
|
|
|
:: |
|
|
|
dmsetup message clone 0 enable_hydration |
|
|
|
4. When the hydration finishes, we can replace the dm-clone table with a linear |
|
table. |
|
|
|
:: |
|
|
|
dmsetup suspend clone |
|
dmsetup load clone --table "0 1048576000 linear $dest_dev 0" |
|
dmsetup resume clone |
|
|
|
The metadata device is no longer needed and can be safely discarded or reused |
|
for other purposes. |
|
|
|
Known issues |
|
============ |
|
|
|
1. We redirect reads, to not-yet-hydrated regions, to the source device. If |
|
reading the source device has high latency and the user repeatedly reads from |
|
the same regions, this behaviour could degrade performance. We should use |
|
these reads as hints to hydrate the relevant regions sooner. Currently, we |
|
rely on the page cache to cache these regions, so we hopefully don't end up |
|
reading them multiple times from the source device. |
|
|
|
2. Release in-core resources, i.e., the bitmaps tracking which regions are |
|
hydrated, after the hydration has finished. |
|
|
|
3. During background hydration, if we fail to read the source or write to the |
|
destination device, we print an error message, but the hydration process |
|
continues indefinitely, until it succeeds. We should stop the background |
|
hydration after a number of failures and emit a dm event for user space to |
|
notice. |
|
|
|
Why not...? |
|
=========== |
|
|
|
We explored the following alternatives before implementing dm-clone: |
|
|
|
1. Use dm-cache with cache size equal to the source device and implement a new |
|
cloning policy: |
|
|
|
* The resulting cache device is not a one-to-one mirror of the source device |
|
and thus we cannot remove the cache device once cloning completes. |
|
|
|
* dm-cache writes to the source device, which violates our requirement that |
|
the source device must be treated as read-only. |
|
|
|
* Caching is semantically different from cloning. |
|
|
|
2. Use dm-snapshot with a COW device equal to the source device: |
|
|
|
* dm-snapshot stores its metadata in the COW device, so the resulting device |
|
is not a one-to-one mirror of the source device. |
|
|
|
* No background copying mechanism. |
|
|
|
* dm-snapshot needs to commit its metadata whenever a pending exception |
|
completes, to ensure snapshot consistency. In the case of cloning, we don't |
|
need to be so strict and can rely on committing metadata every time a FLUSH |
|
or FUA bio is written, or periodically, like dm-thin and dm-cache do. This |
|
improves the performance significantly. |
|
|
|
3. Use dm-mirror: The mirror target has a background copying/mirroring |
|
mechanism, but it writes to all mirrors, thus violating our requirement that |
|
the source device must be treated as read-only. |
|
|
|
4. Use dm-thin's external snapshot functionality. This approach is the most |
|
promising among all alternatives, as the thinly-provisioned volume is a |
|
one-to-one mirror of the source device and handles reads and writes to |
|
un-provisioned/not-yet-cloned areas the same way as dm-clone does. |
|
|
|
Still: |
|
|
|
* There is no background copying mechanism, though one could be implemented. |
|
|
|
* Most importantly, we want to support arbitrary block devices as the |
|
destination of the cloning process and not restrict ourselves to |
|
thinly-provisioned volumes. Thin-provisioning has an inherent metadata |
|
overhead, for maintaining the thin volume mappings, which significantly |
|
degrades performance. |
|
|
|
Moreover, cloning a device shouldn't force the use of thin-provisioning. On |
|
the other hand, if we wish to use thin provisioning, we can just use a thin |
|
LV as dm-clone's destination device.
|
|
|