GNU tar 1.35: Appendix E Tar Internals

While an archive may contain many files, the archive itself is a single ordinary file. Like any other file, an archive file can be written to a storage device such as a tape or disk, sent through a pipe or over a network, saved on the active file system, or even stored in another archive. An archive file is not easy to read or manipulate without using the tar utility or Tar mode in GNU Emacs.

Physically, an archive consists of a series of file entries terminated by an end-of-archive entry, which consists of two 512 blocks of zero bytes. A file entry usually describes one of the files in the archive (an archive member), and consists of a file header and the contents of the file. File headers contain file names and statistics, checksum information which tar uses to detect file corruption, and information about file types.

Archives are permitted to have more than one member with the same member name. One way this situation can occur is if more than one version of a file has been stored in the archive. For information about adding new versions of a file to an archive, see Updating an Archive.

In addition to entries describing archive members, an archive may contain entries which tar itself uses to store information. See section Including a Label in the Archive, for an example of such an archive entry.

A tar archive file contains a series of blocks. Each block contains BLOCKSIZE bytes. Although this format may be thought of as being on magnetic tape, other media are often used.

Each file archived is represented by a header block which describes the file, followed by zero or more blocks which give the contents of the file. At the end of the archive file there are two 512-byte blocks filled with binary zeros as an end-of-file marker. A reasonable system should write such end-of-file marker at the end of an archive, but must not assume that such a block exists when reading an archive. In particular, GNU tar does not treat missing end-of-file marker as an error and silently ignores the fact. You can instruct it to issue a warning, however, by using the ‘--warning=missing-zero-blocks’ option (see section missing-zero-blocks).

The blocks may be blocked for physical I/O operations. Each record of n blocks (where n is set by the ‘--blocking-factor=512-size’ (‘-b 512-size’) option to tar) is written with a single ‘write ()’ operation. On magnetic tapes, the result of such a write is a single record. When writing an archive, the last record of blocks should be written at the full size, with blocks after the zero block containing all zeros. When reading an archive, a reasonable system should properly handle an archive whose last record is shorter than the rest, or which contains garbage records after a zero block.

The header block is defined in C as follows. In the GNU tar distribution, this is part of file ‘src/tar.h’:

/* tar Header Block, from POSIX 1003.1-1990.  */

/* POSIX header.  */

struct posix_header
{                              /* byte offset */
  char name[100];               /*   0 */
  char mode[8];                 /* 100 */
  char uid[8];                  /* 108 */
  char gid[8];                  /* 116 */
  char size[12];                /* 124 */
  char mtime[12];               /* 136 */
  char chksum[8];               /* 148 */
  char typeflag;                /* 156 */
  char linkname[100];           /* 157 */
  char magic[6];                /* 257 */
  char version[2];              /* 263 */
  char uname[32];               /* 265 */
  char gname[32];               /* 297 */
  char devmajor[8];             /* 329 */
  char devminor[8];             /* 337 */
  char prefix[155];             /* 345 */
                                /* 500 */
};

#define TMAGIC   "ustar"        /* ustar and a null */
#define TMAGLEN  6
#define TVERSION "00"           /* 00 and no null */
#define TVERSLEN 2

/* Values used in typeflag field.  */
#define REGTYPE  '0'            /* regular file */
#define AREGTYPE '\0'           /* regular file */
#define LNKTYPE  '1'            /* link */
#define SYMTYPE  '2'            /* reserved */
#define CHRTYPE  '3'            /* character special */
#define BLKTYPE  '4'            /* block special */
#define DIRTYPE  '5'            /* directory */
#define FIFOTYPE '6'            /* FIFO special */
#define CONTTYPE '7'            /* reserved */

#define XHDTYPE  'x'            /* Extended header referring to the
                                   next file in the archive */
#define XGLTYPE  'g'            /* Global extended header */

/* Bits used in the mode field, values in octal.  */
#define TSUID    04000          /* set UID on execution */
#define TSGID    02000          /* set GID on execution */
#define TSVTX    01000          /* reserved */
                                /* file permissions */
#define TUREAD   00400          /* read by owner */
#define TUWRITE  00200          /* write by owner */
#define TUEXEC   00100          /* execute/search by owner */
#define TGREAD   00040          /* read by group */
#define TGWRITE  00020          /* write by group */
#define TGEXEC   00010          /* execute/search by group */
#define TOREAD   00004          /* read by other */
#define TOWRITE  00002          /* write by other */
#define TOEXEC   00001          /* execute/search by other */

/* tar Header Block, GNU extensions.  */

/* In GNU tar, SYMTYPE is for to symbolic links, and CONTTYPE is for
   contiguous files, so maybe disobeying the "reserved" comment in POSIX
   header description.  I suspect these were meant to be used this way, and
   should not have really been "reserved" in the published standards.  */

/* *BEWARE* *BEWARE* *BEWARE* that the following information is still
   boiling, and may change.  Even if the OLDGNU format description should be
   accurate, the so-called GNU format is not yet fully decided.  It is
   surely meant to use only extensions allowed by POSIX, but the sketch
   below repeats some ugliness from the OLDGNU format, which should rather
   go away.  Sparse files should be saved in such a way that they do *not*
   require two passes at archive creation time.  Huge files get some POSIX
   fields to overflow, alternate solutions have to be sought for this.  */

/* Descriptor for a single file hole.  */

struct sparse
{                              /* byte offset */
  char offset[12];              /*   0 */
  char numbytes[12];            /*  12 */
                                /*  24 */
};

/* Sparse files are not supported in POSIX ustar format.  For sparse files
   with a POSIX header, a GNU extra header is provided which holds overall
   sparse information and a few sparse descriptors.  When an old GNU header
   replaces both the POSIX header and the GNU extra header, it holds some
   sparse descriptors too.  Whether POSIX or not, if more sparse descriptors
   are still needed, they are put into as many successive sparse headers as
   necessary.  The following constants tell how many sparse descriptors fit
   in each kind of header able to hold them.  */

#define SPARSES_IN_EXTRA_HEADER  16
#define SPARSES_IN_OLDGNU_HEADER 4
#define SPARSES_IN_SPARSE_HEADER 21

/* Extension header for sparse files, used immediately after the GNU extra
   header, and used only if all sparse information cannot fit into that
   extra header.  There might even be many such extension headers, one after
   the other, until all sparse information has been recorded.  */

struct sparse_header
{                              /* byte offset */
  struct sparse sp[SPARSES_IN_SPARSE_HEADER];
                                /*   0 */
  char isextended;              /* 504 */
                                /* 505 */
};

/* The old GNU format header conflicts with POSIX format in such a way that
   POSIX archives may fool old GNU tar's, and POSIX tar's might well be
   fooled by old GNU tar archives.  An old GNU format header uses the space
   used by the prefix field in a POSIX header, and cumulates information
   normally found in a GNU extra header.  With an old GNU tar header, we
   never see any POSIX header nor GNU extra header.  Supplementary sparse
   headers are allowed, however.  */

struct oldgnu_header
{                              /* byte offset */
  char unused_pad1[345];        /*   0 */
  char atime[12];               /* 345 Incr. archive: atime of the file */
  char ctime[12];               /* 357 Incr. archive: ctime of the file */
  char offset[12];              /* 369 Multivolume archive: the offset of
                                   the start of this volume */
  char longnames[4];            /* 381 Not used */
  char unused_pad2;             /* 385 */
  struct sparse sp[SPARSES_IN_OLDGNU_HEADER];
                                /* 386 */
  char isextended;              /* 482 Sparse file: Extension sparse header
                                   follows */
  char realsize[12];            /* 483 Sparse file: Real size*/
                                /* 495 */
};

/* OLDGNU_MAGIC uses both magic and version fields, which are contiguous.
   Found in an archive, it indicates an old GNU header format, which will be
   hopefully become obsolescent.  With OLDGNU_MAGIC, uname and gname are
   valid, though the header is not truly POSIX conforming.  */
#define OLDGNU_MAGIC "ustar  "  /* 7 chars and a null */

/* The standards committee allows only capital A through capital Z for
   user-defined expansion.  Other letters in use include:

   'A' Solaris Access Control List
   'E' Solaris Extended Attribute File
   'I' Inode only, as in 'star'
   'N' Obsolete GNU tar, for file names that do not fit into the main header.
   'X' POSIX 1003.1-2001 eXtended (VU version)  */

/* This is a dir entry that contains the names of files that were in the
   dir at the time the dump was made.  */
#define GNUTYPE_DUMPDIR 'D'

/* Identifies the *next* file on the tape as having a long linkname.  */
#define GNUTYPE_LONGLINK 'K'

/* Identifies the *next* file on the tape as having a long name.  */
#define GNUTYPE_LONGNAME 'L'

/* This is the continuation of a file that began on another volume.  */
#define GNUTYPE_MULTIVOL 'M'

/* This is for sparse files.  */
#define GNUTYPE_SPARSE 'S'

/* This file is a tape/volume header.  Ignore it on extraction.  */
#define GNUTYPE_VOLHDR 'V'

/* Solaris extended header */
#define SOLARIS_XHDTYPE 'X'

/* Jörg Schilling star header */

struct star_header
{                              /* byte offset */
  char name[100];               /*   0 */
  char mode[8];                 /* 100 */
  char uid[8];                  /* 108 */
  char gid[8];                  /* 116 */
  char size[12];                /* 124 */
  char mtime[12];               /* 136 */
  char chksum[8];               /* 148 */
  char typeflag;                /* 156 */
  char linkname[100];           /* 157 */
  char magic[6];                /* 257 */
  char version[2];              /* 263 */
  char uname[32];               /* 265 */
  char gname[32];               /* 297 */
  char devmajor[8];             /* 329 */
  char devminor[8];             /* 337 */
  char prefix[131];             /* 345 */
  char atime[12];               /* 476 */
  char ctime[12];               /* 488 */
                                /* 500 */
};

#define SPARSES_IN_STAR_HEADER      4
#define SPARSES_IN_STAR_EXT_HEADER  21

struct star_in_header
{
  char fill[345];       /*   0  Everything that is before t_prefix */
  char prefix[1];       /* 345  t_name prefix */
  char fill2;           /* 346  */
  char fill3[8];        /* 347  */
  char isextended;      /* 355  */
  struct sparse sp[SPARSES_IN_STAR_HEADER]; /* 356  */
  char realsize[12];    /* 452  Actual size of the file */
  char offset[12];      /* 464  Offset of multivolume contents */
  char atime[12];       /* 476  */
  char ctime[12];       /* 488  */
  char mfill[8];        /* 500  */
  char xmagic[4];       /* 508  "tar" */
};

struct star_ext_header
{
  struct sparse sp[SPARSES_IN_STAR_EXT_HEADER];
  char isextended;
};

All characters in header blocks are represented by using 8-bit characters in the local variant of ASCII. Each field within the structure is contiguous; that is, there is no padding used within the structure. Each character on the archive medium is stored contiguously.

Bytes representing the contents of files (after the header block of each file) are not translated in any way and are not constrained to represent characters in any character set. The tar format does not distinguish text files from binary files, and no translation of file contents is performed.

The name, linkname, magic, uname, and gname are null-terminated character strings. All other fields are zero-filled octal numbers in ASCII. Each numeric field of width w contains w minus 1 digits, and a null. (In the extended GNU format, the numeric fields can take other forms.)

The name field is the file name of the file, with directory names (if any) preceding the file name, separated by slashes.

The mode field provides nine bits specifying file permissions and three bits to specify the Set UID, Set GID, and Save Text (sticky) modes. Values for these bits are defined above. When special permissions are required to create a file with a given mode, and the user restoring files from the archive does not hold such permissions, the mode bit(s) specifying those special permissions are ignored. Modes which are not supported by the operating system restoring files from the archive will be ignored. Unsupported modes should be faked up when creating or updating an archive; e.g., the group permission could be copied from the other permission.

The uid and gid fields are the numeric user and group ID of the file owners, respectively. If the operating system does not support numeric user or group IDs, these fields should be ignored.

The size field is the size of the file in bytes; for archive members that are symbolic or hard links to another file, this field is specified as zero.

The mtime field represents the data modification time of the file at the time it was archived. It represents the integer number of seconds since January 1, 1970, 00:00 Coordinated Universal Time.

The chksum field represents the simple sum of all bytes in the header block. Each 8-bit byte in the header is added to an unsigned integer, initialized to zero, the precision of which shall be no less than seventeen bits. When calculating the checksum, the chksum field is treated as if it were filled with spaces (ASCII 32).

The typeflag field specifies the type of file archived. If a particular implementation does not recognize or permit the specified type, the file will be extracted as if it were a regular file. As this action occurs, tar issues a warning to the standard error.

The atime and ctime fields are used in making incremental backups; they store, respectively, the particular file’s access and status change times.

The offset is used by the ‘--multi-volume’ (‘-M’) option, when making a multi-volume archive. The offset is number of bytes into the file that we need to restart at to continue the file on the next tape, i.e., where we store the location that a continued file is continued at.

The following fields were added to deal with sparse files. A file is sparse if it takes in unallocated blocks which end up being represented as zeros, i.e., no useful data. A test to see if a file is sparse is to look at the number blocks allocated for it versus the number of characters in the file; if there are fewer blocks allocated for the file than would normally be allocated for a file of that size, then the file is sparse. This is the method tar uses to detect a sparse file, and once such a file is detected, it is treated differently from non-sparse files.

Sparse files are often dbm files, or other database-type files which have data at some points and emptiness in the greater part of the file. Such files can appear to be very large when an ‘ls -l’ is done on them, when in truth, there may be a very small amount of important data contained in the file. It is thus undesirable to have tar think that it must back up this entire file, as great quantities of room are wasted on empty blocks, which can lead to running out of room on a tape far earlier than is necessary. Thus, sparse files are dealt with so that these empty blocks are not written to the tape. Instead, what is written to the tape is a description, of sorts, of the sparse file: where the holes are, how big the holes are, and how much data is found at the end of the hole. This way, the file takes up potentially far less room on the tape, and when the file is extracted later on, it will look exactly the way it looked beforehand. The following is a description of the fields used to handle a sparse file:

The sp is an array of struct sparse. Each

struct
sparse

contains two 12-character strings which represent an offset into the file and a number of bytes to be written at that offset. The offset is absolute, and not relative to the offset in preceding array element.

The header can hold four of these struct sparse at the moment; if more are needed, they are not stored in the header.

The isextended flag is set when an extended_header is needed to deal with a file. Note that this means that this flag can only be set when dealing with a sparse file, and it is only set in the event that the description of the file will not fit in the allotted room for sparse structures in the header. In other words, an extended_header is needed.

The extended_header structure is used for sparse files which need more sparse structures than can fit in the header. The header can fit 4 such structures; if more are needed, the flag isextended gets set and the next block is an extended_header.

Each extended_header structure contains an array of 21 sparse structures, along with a similar isextended flag that the header had. There can be an indeterminate number of such extended_headers to describe a sparse file.

Other values are reserved for specification in future revisions of the P1003 standard, and should not be used by any tar program.

The magic field indicates that this archive was output in the P1003 archive format. If this field contains TMAGIC, the uname and gname fields will contain the ASCII representation of the owner and group of the file respectively. If found, the user and group IDs are used rather than the values in the uid and gid fields.

For references, see ISO/IEC 9945-1:1990 or IEEE Std 1003.1-1990, pages 169-173 (section 10.1) for Archive/Interchange File Format; and IEEE Std 1003.2-1992, pages 380-388 (section 4.48) and pages 936-940 (section E.4.48) for pax - Portable archive interchange.

GNU Extensions to the Archive Format

The GNU format uses additional file types to describe new types of files in an archive. These are listed below.

For fields containing numbers or timestamps that are out of range for the basic format, the GNU format uses a base-256 representation instead of an ASCII octal number. If the leading byte is 0xff (255), all the bytes of the field (including the leading byte) are concatenated in big-endian order, with the result being a negative number expressed in two’s complement form. If the leading byte is 0x80 (128), the non-leading bytes of the field are concatenated in big-endian order, with the result being a positive number expressed in binary form. Leading bytes other than 0xff, 0x80 and ASCII octal digits are reserved for future use, as are base-256 representations of values that would be in range for the basic format.

You may have trouble reading a GNU format archive on a non-GNU system if the options ‘--incremental’ (‘-G’), ‘--multi-volume’ (‘-M’), ‘--sparse’ (‘-S’), or ‘--label=archive-label’ (‘-V archive-label’) were used when writing the archive. In general, if tar does not use the GNU-added fields of the header, other versions of tar should be able to read the archive. Otherwise, the tar program will give an error, the most likely one being a checksum error.

Storing Sparse Files

The notion of sparse file, and the ways of handling it from the point of view of GNU tar user have been described in detail in Archiving Sparse Files. This chapter describes the internal format GNU tar uses to store such files.

The support for sparse files in GNU tar has a long history. The earliest version featuring this support that I was able to find was 1.09, released in November, 1990. The format introduced back then is called old GNU sparse format and in spite of the fact that its design contained many flaws, it was the only format GNU tar supported until version 1.14 (May, 2004), which introduced initial support for sparse archives in PAX archives (see section GNU tar and POSIX tar). This format was not free from design flaws, either and it was subsequently improved in versions 1.15.2 (November, 2005) and 1.15.92 (June, 2006).

In addition to GNU sparse format, GNU tar is able to read and extract sparse files archived by star.

E.0.1 Old GNU Format

The format introduced in November 1990 (v. 1.09) was designed on top of standard ustar headers in such an unfortunate way that some of its fields overwrote fields required by POSIX.

An old GNU sparse header is designated by type ‘S’ (GNUTYPE_SPARSE) and has the following layout:

Offset	Size	Name	Data type	Contents
0	345		N/A	Not used.
345	12	atime	Number	`atime` of the file.
357	12	ctime	Number	`ctime` of the file .
369	12	offset	Number	For multivolume archives: the offset of the start of this volume.
381	4		N/A	Not used.
385	1		N/A	Not used.
386	96	sp	`sparse_header`	(4 entries) File map.
482	1	isextended	Bool	`1` if an extension sparse header follows, `0` otherwise.
483	12	realsize	Number	Real size of the file.

Each of sparse_header object at offset 386 describes a single data chunk. It has the following structure:

Offset	Size	Data type	Contents
0	12	Number	Offset of the beginning of the chunk.
12	12	Number	Size of the chunk.

If the member contains more than four chunks, the isextended field of the header has the value 1 and the main header is followed by one or more extension headers. Each such header has the following structure:

E.0.2 PAX Format, Versions 0.0 and 0.1

Offset	Size	Name	Data type	Contents
0	21	sp	`sparse_header`	(21 entries) File map.
504	1	isextended	Bool	`1` if an extension sparse header follows, or `0` otherwise.

There are two formats available in this branch. The version 0.0 is the initial version of sparse format used by tar versions 1.14–1.15.1. The sparse file map is kept in extended (x) PAX header variables:

The latter two variables repeat for each data block, so the overall structure is like this:

GNU.sparse.size=size
GNU.sparse.numblocks=numblocks
repeat numblocks times
  GNU.sparse.offset=offset
  GNU.sparse.numbytes=numbytes
end repeat

GNU tar 1.15.2 introduced sparse format version 0.1, which attempted to solve these problems. As its predecessor, this format stores sparse map in the extended POSIX header. It retains GNU.sparse.size and GNU.sparse.numblocks variables, but instead of GNU.sparse.offset/GNU.sparse.numbytes pairs it uses a single variable:

To address the 2nd problem, the name field in ustar is replaced with a special name, constructed using the following pattern:

%d/GNUSparseFile.%p/%f

The real name of the sparse file is stored in the variable GNU.sparse.name. Thus, those tar implementations that are not aware of GNU extensions will at least extract the files into separate directories, giving the user a possibility to expand it afterwards. See Extraction of sparse members in v.0.1 format, for the detailed description of how to restore such members using non-GNU tars.

The resulting GNU.sparse.map string can be very long. Although POSIX does not impose any limit on the length of a x header variable, this possibly can confuse some tars.

E.0.3 PAX Format, Version 1.0

The version 1.0 of sparse format was introduced with GNU tar 1.15.92. Its main objective was to make the resulting file extractable with little effort even by non-posix aware tar implementations. Starting from this version, the extended header preceding a sparse member always contains the following variables that identify the format being used:

The name field in ustar header contains a special name, constructed using the following pattern:

%d/GNUSparseFile.%p/%f

The real name of the sparse file is stored in the variable GNU.sparse.name. The real size of the file is stored in the variable GNU.sparse.realsize.

The sparse map itself is stored in the file data block, preceding the actual file data. It consists of a series of decimal numbers delimited by newlines. The map is padded with nulls to the nearest block boundary.

The first number gives the number of entries in the map. Following are map entries, each one consisting of two numbers giving the offset and size of the data block it describes.

The format is designed in such a way that non-posix aware tars and tars not supporting GNU.sparse.* keywords will extract each sparse file in its condensed form with the file map prepended and will place it into a separate directory. Then, using a simple program it would be possible to expand the file to its original form even without GNU tar. See section Extracting Sparse Members, for the detailed information on how to extract sparse members without GNU tar.

Format of the Incremental Snapshot Files

A snapshot file (or directory file) is created during incremental backups (see section Using tar to Perform Incremental Dumps). It contains the status of the file system at the time of the dump and is used to determine which files were modified since the last backup.

GNU tar version 1.35 supports three snapshot file formats. The first format, called format 0, is the one used by GNU tar versions up to and including 1.15.1. The second format, called format 1 is an extended version of this format, that contains more metadata and allows for further extensions. It was used by alpha release version 1.15.90. For alpha version 1.15.91 and stable releases version 1.16 up through 1.35, the format 2 is used.

GNU tar is able to read all three formats, but will create snapshots only in format 2.

Dumpdir

Field	Type	Description
nfs	Character	‘`1`’ if the directory is located on an NFS-mounted partition, or ‘`0`’ otherwise;
timestamp_sec	Number	Modification time, seconds;
timestamp_nsec	Number	Modification time, nanoseconds;
dev	Number	Device number;
ino	Number	I-node number;
name	String	Directory name; in contrast to the previous versions it is not quoted;
contents	Dumpdir	Contents of the directory; See section Dumpdir, for a description of its format.

Incremental archives keep information about contents of each dumped directory in special data blocks called dumpdirs.

where C is one of the control codes described below, filename is the name of the file C operates upon, and ‘\0’ represents a nul character (ASCII 0). The white space characters were added for readability, real dumpdirs do not contain them.

Codes ‘Y’, ‘N’ and ‘D’ require filename argument to be a relative file name to the directory this dumpdir describes, whereas codes ‘R’, ‘T’ and ‘X’ require their argument to be an absolute file name.

The three codes ‘R’, ‘T’ and ‘X’ specify a renaming operation. In the simplest case it is:

R‘source’\0T‘dest’\0

However, there are cases that require using a temporary directory. For example, consider the following scenario:

This case cannot be handled by three successive renames, since renaming ‘a’ to ‘b’ will destroy the existing directory. To correctly process it, GNU tar needs a temporary directory, so it creates the following dumpdir (newlines have been added for readability):

Xfoo\0
Rfoo/a\0T\0
Rfoo/b\0Tfoo/c\0
Rfoo/c\0Tfoo/a\0
R\0Tfoo/a\0

The first command, ‘Xfoo\0’, instructs the extractor to create a temporary directory in the directory ‘foo’. Second command, ‘Rfoo/aT\0’, says “rename file ‘foo/a’ to the temporary directory that has just been created” (empty file name after a command means use temporary directory). Third and fourth commands work as usual, and, finally, the last command, ‘R\0Tfoo/a\0’ tells tar to rename the temporary directory to ‘foo/a’.

The exact placement of a dumpdir in the archive depends on the archive format (see section Controlling the Archive Format):

Basic Tar Format
GNU Extensions to the Archive Format
Storing Sparse Files
Format of the Incremental Snapshot Files
Dumpdir

Appendix E Tar Internals