GNU ddrescue

GNU ddrescue

This manual is for GNU ddrescue (version 1.11, 10 July 2009).

• Introduction: Purpose and features of GNU ddrescue
• Basic concepts: Blocks, clusters, devices, files, sectors, etc
• Algorithm: How ddrescue recovers the data
• Invoking ddrescue: Command line interface
• Logfile Structure: Detailed format of the logfile
• Examples: A small tutorial with examples
• Direct Disc Access: Bypassing the kernel cache
• Fill Mode: Selectively overwriting the output file
• Generate Mode: Generating an approximate logfile
• Problems: Reporting bugs
• Concept Index: Index of concepts

This manual is free documentation: you have unlimited permission to copy, distribute and modify it.

1 Introduction

GNU ddrescue is a data recovery tool. It copies data from one file or block device (hard disc, cdrom, etc) to another, trying hard to rescue data in case of read errors.

The basic operation of ddrescue is fully automatic. That is, you don't have to wait for an error, stop the program, read the log, run it in reverse mode, etc.

If you use the logfile feature of ddrescue, the data is rescued very efficiently, (only the needed blocks are read). Also you can interrupt the rescue at any time and resume it later at the same point.

Automatic merging of backups: If you have two or more damaged copies of a file, cdrom, etc, and run ddrescue on all of them, one at a time, with the same output file, you will probably obtain a complete and error-free file. This is so because the probability of having damaged areas at the same places on different input files is very low. Using the logfile, only the needed blocks are read from the second and successive copies.

Recordable CD and DVD media keep their data only for a finite time (typically for many years). After that time, data loss develops slowly with read errors growing from the outer media region towards the inside. Just make two (or more) copies of every important CD/DVD you burn so that you can later recover them with ddrescue.

Ddrescue does not truncate the output file if not asked to. So, every time you run it on the same output file, it tries to fill in the gaps.

Because ddrescue needs to read and write at random places, it only works on seekable (random access) input and output files.

If your system supports it, ddrescue can use direct disc access to read the input file, bypassing the kernel cache.

Ddrescue also features a "fill mode" able to selectively overwrite parts of the output file, which has a number of interesting uses like wiping data, marking bad areas or even, in some cases, "repair" damaged sectors.

2 Basic concepts

Block

Any amount of data. A block is described by its starting position and its size.

Cluster

Group of consecutive sectors read or written in one go.

Device

Piece of hardware containing data. Hard disc drives, cdrom drives, USB pendrives, are devices. /dev/hda, /dev/sdb, are device names.

File

Files are named units of data which are stored by the operating system for you to retrieve later by name. Devices are accessed by means of their associated file names.

Partition

Every part in which a device is divided. A partition normally contains a filesystem. /dev/hda1, /dev/sdb3, are partition names.

Sector

Hardware block. Smallest accessible amount of data on a device.

3 Algorithm

GNU ddrescue manages efficiently the status of the rescue in progress and tries to rescue the good parts first, maximizing the amount of data finally recovered from a failing drive.

The standard dd utility can be used to save data from a failing drive, but it reads the data secuentially, which may wear out the drive without rescuing anything if the errors are at the beginning of the drive.

Other programs switch to small size reads when they find errors, but this is a bad idea because it means spending more time at error areas, damaging the surface, the heads and the drive mechanics, instead of getting out of them as fast as possible. This behavior reduces the chances of rescuing the remaining good data.

The algorithm of ddrescue is as follows: (the user may interrupt the process at any point, but be aware that a bad drive can block ddrescue for a long time until the kernel gives up)

1) Optionally read a logfile describing the status of a multi-part or previously interrupted rescue. If no logfile is specified or is empty or does not exist, mark all the rescue domain as non-tried.

2) Read the non-tried parts of the input file, marking the failed blocks as non-trimmed and skipping beyond them, until all the rescue domain is tried. Only non-tried areas are read in large blocks. Trimming, splitting and retrying are done sector by sector. Each sector is tried at most two times; the first in this step as part of a large block read, the second in one of the steps below as a single sector read.

3) Read backwards one sector at a time the non-trimmed blocks, until a bad sector is found. For each non-trimmed block, mark the bad sector found as bad-sector and mark the rest of that block as non-split.

4) Read forwards one sector at a time the non-split blocks, marking the bad sectors found as bad-sector. After a number of consecutive bad sectors is found in a block larger enough, the block is split by half and the reading continues on the second half. This recursively splits the largest failed blocks without producing a logfile too large.

5) Optionally try to read again the bad sectors until the specified number of retries is reached.

6) Optionally write a logfile for later use.

Also, the same logfile can be used for multiple commands that copy different areas of the input file, and for multiple recovery attempts over different subsets. See this example:

Rescue the most important part of the disc first.

     ddrescue -i0 -s50M /dev/hdc hdimage logfile
     ddrescue -i0 -s1M -r3 -d /dev/hdc hdimage logfile

Then rescue some key disc areas.

     ddrescue -i30G -s10G /dev/hdc hdimage logfile
     ddrescue -i230G -s5G /dev/hdc hdimage logfile

Now rescue the rest (does not recopy what is already done).

     ddrescue /dev/hdc hdimage logfile
     ddrescue -r3 -d /dev/hdc hdimage logfile

4 Invoking ddrescue

The format for running ddrescue is:

     ddrescue [options] infile outfile [logfile]

ddrescue supports the following options:

`--help'

`-h'

Print an informative help message describing the options and exit.

`--version'

`-V'

Print the version number of ddrescue on the standard output and exit.

`--block-size=bytes'

`-b bytes'

Hardware block (sector) size of input device in bytes (usually 512 for hard discs and 3.5" floppies, 1024 for 5.25" floppies, and 2048 for cdroms). Defaults to 512.

`--binary-prefixes'

`-B'

Show units with binary prefixes (powers of 1024).
SI prefixes (powers of 1000) are used by default. (See table below).

`--cluster-size=sectors'

`-c sectors'

Number of sectors to copy at a time. Defaults to 64KiB / sector_size. Try smaller values for slow drives. The number of sectors per track (18 or 9) is a good value for floppies.

`--complete-only'

`-C'

Limit rescue domain to the blocks listed in the logfile. Do not read new data beyond logfile limits. This is useful when reading from devices of undefined size, like raw devices.

`--direct'

`-d'

Use direct disc access to read the input file, bypassing the kernel cache. (Open the file with the O_DIRECT flag). Use it only on devices or partitions, not on regular files. Sector size must be correctly set for this to work. Not all systems support this.

If your system does not support direct disc access, ddrescue will warn you. If the sector size is not correctly set, all reads will result in errors, and no data will be rescued.

`--synchronous'

`-D'

Use synchronous writes for output file. (Issue a fsync call after every write). May be useful when forcing the drive to remap its bad sectors.

`--max-errors=n'

`-e n'

Maximum number of error areas allowed before giving up. Defaults to infinity.

`--fill=types'

`-F types'

Fill the output file blocks specified as any of types in the logfile, with data read from the input file. types contains one or more of the status characters defined in the chapter Logfile Structure (see Logfile Structure). See the chapter Fill Mode (see Fill Mode) for a complete description of the fill mode.

`--generate-logfile'

`-g'

Generate an approximate logfile from the input and output files of the original rescue run. Note that you should keep the original offset between `--input-position' and `--output-position' of the original rescue run.

`--input-position=pos'

`-i pos'

Starting position in input file, in bytes. Defaults to 0. In fill mode it refers to the original input file. See the chapter Fill Mode (see Fill Mode) for details.

`--domain-logfile=file'

`-m file'

Restrict the rescue domain to the blocks marked as finished in the logfile file. This is useful if the destination drive fails during the rescue.

`--no-split'

`-n'

Stop after the trimming pass. Avoids spending a lot of time trying to rescue the most difficult parts of the file. This option overrides the `--max-retries' option.

`--output-position=pos'

`-o pos'

Starting position in output file, in bytes. Defaults to `--input-position'. The bytes below pos aren't touched if they exist and truncation is not requested. Else they are set to 0.

`--quiet'

`-q'

Quiet operation.

`--max-retries=n'

`-r n'

Exit after given number of retry passes. Defaults to 0. -1 means infinity. Every bad sector is tried only one time per pass. To retry bad sectors detected on a previous run, you must specify a non-zero number of retries.

`--retrim'

`-R'

Mark all failed blocks inside the rescue domain as non-trimmed before beginning the rescue. The effect is similar to `--max-retries=1', but the bad sectors are tried in a different order, making perhaps possible to rescue some of them.

`--max-size=bytes'

`-s bytes'

Maximum size of the input data to be copied, in bytes. If ddrescue can't determine the size of the input device, you may need to specify it with this option. Note that this option specifies the size of the input data to be copied, not the size of the output file. So, for example, the following command creates an output file 300 bytes long, but only writes data on the last 200 bytes:

          ddrescue -i 100 -s 200 infile outfile logfile
     

`--sparse'

`-S'

Use sparse writes for output file. (The blocks of zeros are not actually allocated on disc). May save a lot of disc space in some cases. Not all systems support this.

`--truncate'

`-t'

Truncate output file to zero size before writing to it. Only works for regular files, not for drives or partitions.

`--try-again'

`-T'

Mark all non-split and non-trimmed blocks inside the rescue domain as non-tried before beginning the rescue. Try this if the drive stops responding and ddrescue immediately starts splitting failed blocks when restarted. If `--retrim' is also specified, mark all failed blocks inside the rescue domain as non-tried.

`--verbose'

`-v'

Verbose mode.

Table of SI and binary prefixes (unit multipliers):

Prefix	Value	|	Prefix	Value
		|	b	hardware blocks
k	kilobyte (10^3 = 1000)	|	Ki	kibibyte (2^10 = 1024)
M	megabyte (10^6)	|	Mi	mebibyte (2^20)
G	gigabyte (10^9)	|	Gi	gibibyte (2^30)
T	terabyte (10^12)	|	Ti	tebibyte (2^40)
P	petabyte (10^15)	|	Pi	pebibyte (2^50)
E	exabyte (10^18)	|	Ei	exbibyte (2^60)
Z	zettabyte (10^21)	|	Zi	zebibyte (2^70)
Y	yottabyte (10^24)	|	Yi	yobibyte (2^80)

5 Logfile Structure

The logfile is a text file easy to read and edit. It is formed of two parts, the status line and the list of data blocks.

NOTE: Logfiles generated by a version of ddrescue prior to 1.6 lack the status line. If you want to use an old logfile with ddrescue 1.6 or later, you will have to insert a line like `0 +' at the beginning of the logfile.

The first non-comment line is the status line. It contains a non-negative integer and a status character. The integer is the position being tried in the input file. The status character is one of these:

Character	Meaning
'?'	copying non-tried blocks
'*'	trimming non-trimmed blocks
'/'	splitting non-split blocks
'-'	retrying bad sectors
'F'	filling specified blocks
'G'	generating approximate logfile
'+'	finished

Every line in the list of data blocks describes a block of data. It contains 2 non-negative integers and a status character. The first integer is the starting position of the block in the input file, the second integer is the size (in bytes) of the block. The status character is one of these:

Character	Meaning
'?'	non-tried block
'*'	failed block non-trimmed
'/'	failed block non-split
'-'	failed block bad-sector(s)
'+'	finished block

And here is an example logfile:

# Rescue Logfile. Created by GNU ddrescue version 1.11
# current_pos current_status
0x00120000 ?
# pos size status

0x00000000	0x00117000	+
0x00117000	0x00000200	-
0x00117200	0x00001000	/
0x00118200	0x00007E00	*
0x00120000	0x00048000	?

Any line beginning with `#' is a comment line. The blocks must be contiguous and non-overlapping.

If you edit the file, you may use decimal, hexadecimal or octal values, using the same syntax that integer constants in C++.

6 A small tutorial with examples

Ddrescue is like any other power tool. You need to understand what it does, and you need to understand some things about the machines it does those things to, in order to use it safely.

A failing drive tends to develop more and more errors as time passes. Because of this, you should rescue the data from a drive as soon as you notice the first error.

IMPORTANT! Always use a logfile unless you know you won't need it. Without a logfile, ddrescue can't resume a rescue, only reinitiate it.

IMPORTANT! Never try to rescue a r/w mounted partition. The resulting copy may be useless.

You should make a copy of the failing drive with ddrescue, and then try to repair the copy. If your data is really important, use the first copy as a master for a second copy, and try to repair the second copy. If something goes wrong, you have the master intact to try again.

IMPORTANT! Never try to repair a file system on a drive with I/O errors; you will probably lose even more data.

If you are trying to rescue a whole partition, first repair the copy with e2fsck or some other tool appropiate for the type of partition you are trying to rescue, then mount the repaired copy somewhere and try to recover the files in it.

If the drive is so damaged that the file system in the rescued partition can't be repaired or mounted, you will have to browse the rescued data with an hex editor and extract the desired parts by hand or use a file recovery tool like photorec.

If the partition table is damaged, you may try to rescue the whole disc, then try to repair the partition table and the partitions on the copy.

If the damaged drive is not listed in /dev, then you cannot rescue it. At least not with ddrescue.

     ddrescue -n /dev/hda /dev/hdb logfile
     ddrescue -dr3 /dev/hda /dev/hdb logfile
     fdisk /dev/hdb
     e2fsck -v -f /dev/hdb1
     e2fsck -v -f /dev/hdb2

     ddrescue -n /dev/hda2 /dev/hdb2 logfile
     ddrescue -d -r3 /dev/hda2 /dev/hdb2 logfile
     e2fsck -v -f /dev/hdb2
     mount -t ext2 -o ro /dev/hdb2 /mnt
     read files from /mnt

     ddrescue -n -b2048 /dev/cdrom cdimage logfile
     ddrescue -d -b2048 /dev/cdrom cdimage logfile
     write cdimage to a blank CD-ROM

     ddrescue -n /dev/hda /dev/hdb logfile1     <-- /dev/hdb fails here
     ddrescue -m logfile1 /dev/hdb /dev/hdc logfile2
     ddrescue -n /dev/hda /dev/hdc logfile2
     ddrescue -dr3 /dev/hda /dev/hdc logfile2

     ddrescue -n /dev/hda /dev/hdb logfile      <-- /dev/hda fails here
       (restart /dev/hda or reboot computer as many times as needed)
     ddrescue -nT /dev/hda /dev/hdb logfile
     ddrescue -dr3 /dev/hda /dev/hdb logfile

7 Direct Disc Access

If you notice that the sizes and offsets in the log file are ALWAYS multiples of the sector size, maybe your kernel is caching the disc accesses and grouping them. In this case you may want to use direct disc access or a raw device to bypass the kernel cache and rescue more of your data.

NOTE! Sector size must be correctly set with the `--block-size' option for this to work. Try the `--direct' option first. If direct disc access is not available in your system, try raw devices. Read your system documentation to find how to bind a raw device to a regular block device.

Ddrescue aligns its I/O buffer to the sector size so that it can be used for direct disc access or to read from raw devices. For efficiency reasons, also aligns it to the memory page size if page size is a multiple of sector size. Ddrescue can't determine the size of a raw device, so a explicit `--max-size' or `--complete-only' option is needed.

Using direct disc access, or reading from a raw device, may be slower than normal cached reading for hard discs (but faster for floppies). In this case you may want to make a first pass using normal cached reads, and use direct disc access, or a raw device, only to recover the good sectors inside the failed blocks.

     ddrescue -n /dev/hdb1 /dev/hdc1 logfile
     ddrescue -d -r3 /dev/hdb1 /dev/hdc1 logfile
     e2fsck -v -f /dev/hdc1
     mount -t ext2 -o ro /dev/hdc1 /mnt

     raw /dev/raw/raw1 /dev/hdb1
     ddrescue -n /dev/hdb1 /dev/hdc1 logfile
     ddrescue -r3 -C /dev/raw/raw1 /dev/hdc1 logfile
     raw /dev/raw/raw1 0 0
     e2fsck -v -f /dev/hdc1
     mount -t ext2 -o ro /dev/hdc1 /mnt

8 Fill Mode

When ddrescue is invoked with the `--fill' option it operates in "fill mode", which is different from the default "rescue mode". That is, if you use the `--fill' option, ddrescue does not rescue anything. It only fills with data read from the input file the blocks of the output file whose status character from the logfile coincides with one of the type characters specified as argument to the `--fill' option.

In fill mode the input file may have any size. If it is too small, the data will be duplicated as many times as necessary to fill the input buffer. If it is too big, only the needed data will be read.

Note that in fill mode the input file is always read from position 0. If you specify a `--input-position', it refers to the original input file from which the logfile was built, and is only used to calculate the offset between input and output positions.

Note also that when filling the input file of the original rescue run you should set `--input-position' and `--output-position' to identical values, whereas when filling the output file of the original rescue run you should keep the original offset between `--input-position' and `--output-position'.

The `--fill' option implies the `--complete-only' option.

In fill mode the logfile is updated to allow resumability when interrupted or in case of a crash, but as nothing is being rescued the logfile is not destroyed. The status line is the only part of the logfile that is modified.

Example 1: Mark parts of the rescued copy to allow finding them when examined in an hex editor. For example, the following command line fills all blocks marked as `-' (bad-sector) with copies of the string `BAD SECTOR ':

     echo -n "BAD SECTOR " > tmpfile
     ddrescue --fill=- tmpfile outfile logfile

Example 2: Wipe only the good sectors, leaving the bad sectors alone. This way, the drive will still test bad (i.e., with unreadable sectors). This is the fastest way of wiping a failing drive, and is specially useful when sending the drive back to the manufacturer for warranty replacement.

     ddrescue --fill=+ /dev/zero bad_drive logfile

Example 3: Force the drive to remap the bad sectors, making it usable again. If the drive has only a few bad sectors, and they are not caused by drive age, you can probably just rewrite those sectors, and the drive will reallocate them automatically to new "spare" sectors that it keeps for just this purpose. WARNING! This may not work on your drive.

     ddrescue --fill=- --synchronous /dev/zero bad_drive logfile

1) Copy the damaged drive with ddrescue until finished. Do not use sparse writes. This yields a logfile with only finished (`+') and bad-sector (`-') blocks.

2) Mount the copied drive (or the image file, via loopback device).

3) Compute a md5sum or other checksum for every file. Build a list of all the files and their checksums.

4) Fill the bad-sector blocks of the copied drive or image file with a byte value different from zero.

5) Verify the checksums. Those files which have different checksums this time reside (at least partially) in damaged disc areas.

6) Optionally fill the bad-sector blocks of the copied drive or image file with zeros to restore the disc image.

9 Generate-logfile Mode

So you didn't read the tutorial and started ddrescue without a logfile. Now, two days later, your computer crashed and you can't know how much data ddrescue managed to save. And even worse, you can't resume the rescue; you have to restart it from the very beginning.

Or maybe you started copying a drive with `dd conv=noerror,sync' and are now in the same situation described above. In this case, note that you can't use a copy made by dd unless it was invoked with the `sync' conversion argument.

Don't despair (yet). Ddrescue can in some cases generate an approximate logfile, from the input file and the (partial) copy, that is almost as good as an exact logfile. It makes this by simply assuming that sectors containing all zeros were not rescued.

However, if the destination of the copy was a drive or a partition, (or an existing regular file and truncation was not requested), most probably you will need to restart ddrescue from the very beginning. (This time with a logfile, of course). The reason is that old data may be present in the drive that have not been overwritten yet, and may be thus non-tried but non-zero.

For example, if you first tried one of these commands:

     ddrescue infile outfile
     or
     dd if=infile of=outfile conv=noerror,sync

you can generate an approximate logfile with this command:

     ddrescue --generate-logfile infile outfile logfile

10 Reporting Bugs

There are probably bugs in ddrescue. There are certainly errors and omissions in this manual. If you report them, they will get fixed. If you don't, no one will ever know about them and they will remain unfixed for all eternity, if not longer.

If you find a bug in GNU ddrescue, please send electronic mail to bug-ddrescue@gnu.org. Include the version number, which you can find by running `ddrescue --version'.

Concept Index

• algorithm: Algorithm
• basic concepts: Basic concepts
• bugs: Problems
• direct disc access: Direct Disc Access
• examples: Examples
• Fill Mode: Fill Mode
• Generate-logfile Mode: Generate Mode
• getting help: Problems
• introduction: Introduction
• invoking: Invoking ddrescue
• logfile structure: Logfile Structure
• options: Invoking ddrescue
• raw devices: Direct Disc Access
• usage: Invoking ddrescue
• version: Invoking ddrescue