This manual documents version 7.2 of the gnu core utilities, including the standard programs for text and file manipulation.
Copyright © 1994-1996, 2000-2009 Free Software Foundation, Inc.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the section entitled “GNU Free Documentation License”.
--- The Detailed Node Listing ---
Common Options
Output of entire files
Formatting file contents
Output of parts of files
Summarizing files
Operating on sorted files
ptx: Produce permuted indexes
Operating on fields within a line
Operating on characters
tr: Translate, squeeze, and/or delete characters
Directory listing
ls: List directory contents
Basic operations
Special file types
Changing file attributes
Disk usage
Printing text
Conditions
test: Check file types and compare values
expr: Evaluate expression
Redirection
File name manipulation
Working context
stty: Print or change terminal characteristics
User information
System context
date: Print or set system date and time
SELinux context
Modified command invocation
Process control
Delaying
Numeric operations
File permissions
Date input formats
Opening the software toolbox
Copying This Manual
This manual is a work in progress: many sections make no attempt to explain basic concepts in a way suitable for novices. Thus, if you are interested, please get involved in improving this manual. The entire gnu community will benefit.
The gnu utilities documented here are mostly compatible with the POSIX standard. Please report bugs to bug-coreutils@gnu.org. Remember to include the version number, machine architecture, input files, and any other information needed to reproduce the bug: your input, what you expected, what you got, and why it is wrong. Diffs are welcome, but please include a description of the problem as well, since this is sometimes difficult to infer. See Bugs.
This manual was originally derived from the Unix man pages in the distributions, which were written by David MacKenzie and updated by Jim Meyering. What you are reading now is the authoritative documentation for these utilities; the man pages are no longer being maintained. The original fmt man page was written by Ross Paterson. François Pinard did the initial conversion to Texinfo format. Karl Berry did the indexing, some reorganization, and editing of the results. Brian Youmans of the Free Software Foundation office staff combined the manuals for textutils, fileutils, and sh-utils to produce the present omnibus manual. Richard Stallman contributed his usual invaluable insights to the overall process.
Certain options are available in all of these programs. Rather than writing identical descriptions for each of the programs, they are described here. (In fact, every gnu program accepts (or should accept) these options.)
Normally options and operands can appear in any order, and programs act as if all the options appear before any operands. For example, ‘sort -r passwd -t :’ acts like ‘sort -r -t : passwd’, since ‘:’ is an option-argument of -t. However, if the POSIXLY_CORRECT environment variable is set, options must appear before operands, unless otherwise specified for a particular command.
A few programs can usefully have trailing operands with leading ‘-’. With such a program, options must precede operands even if POSIXLY_CORRECT is not set, and this fact is noted in the program description. For example, the env command's options must appear before its operands, since in some cases the operands specify a command that itself contains options.
Most programs that accept long options recognize unambiguous abbreviations of those options. For example, ‘rmdir --ignore-fail-on-non-empty’ can be invoked as ‘rmdir --ignore-fail’ or even ‘rmdir --i’. Ambiguous options, such as ‘ls --h’, are identified as such.
Some of these programs recognize the --help and --version options only when one of them is the sole command line argument. For these programs, abbreviations of the long options are not always recognized.
A single ‘-’ operand is not really an option, though it looks like one. It stands for standard input, or for standard output if that is clear from the context. For example, ‘sort -’ reads from standard input, and is equivalent to plain ‘sort’, and ‘tee -’ writes an extra copy of its input to standard output. Unless otherwise specified, ‘-’ can appear as any operand that requires a file name.
Nearly every command invocation yields an integral exit status that can be used to change how other commands work. For the vast majority of commands, an exit status of zero indicates success. Failure is indicated by a nonzero value—typically ‘1’, though it may differ on unusual platforms as POSIX requires only that it be nonzero.
However, some of the programs documented here do produce other exit status values and a few associate different meanings with the values ‘0’ and ‘1’. Here are some of the exceptions: chroot, env, expr, nice, nohup, printenv, sort, su, test, timeout, tty.
Some gnu programs (at least cp, install, ln, and mv) optionally make backups of files before writing new versions. These options control the details of these backups. The options are also briefly mentioned in the descriptions of the particular programs.
Note that the short form of this option, -b does not accept any argument. Using -b is equivalent to using --backup=existing.
This option corresponds to the Emacs variable ‘version-control’; the values for method are the same as those used in Emacs. This option also accepts more descriptive names. The valid methods are (unique abbreviations are accepted):
Some gnu programs (at least df, du, and ls) display sizes in “blocks”. You can adjust the block size and method of display to make sizes easier to read. The block size used for display is independent of any file system block size. Fractional block counts are rounded up to the nearest integer.
The default block size is chosen by examining the following environment variables in turn; the first one that is set determines the block size.
DF_BLOCK_SIZEBLOCK_SIZEBLOCKSIZEls -l output.
POSIXLY_CORRECTIf none of the above environment variables are set, the block size currently defaults to 1024 bytes in most contexts, but this number may change in the future. For ls file sizes, the block size defaults to 1 byte.
A block size specification can be a positive integer specifying the number
of bytes per block, or it can be human-readable or si to
select a human-readable format. Integers may be followed by suffixes
that are upward compatible with the
SI prefixes
for decimal multiples and with the
IEC 60027-2 prefixes for binary multiples.
With human-readable formats, output sizes are followed by a size letter
such as ‘M’ for megabytes. BLOCK_SIZE=human-readable uses
powers of 1024; ‘M’ stands for 1,048,576 bytes.
BLOCK_SIZE=si is similar, but uses powers of 1000 and appends
‘B’; ‘MB’ stands for 1,000,000 bytes.
A block size specification preceded by ‘'’ causes output sizes to be displayed with thousands separators. The LC_NUMERIC locale specifies the thousands separator and grouping. For example, in an American English locale, ‘--block-size="'1kB"’ would cause a size of 1234000 bytes to be displayed as ‘1,234’. In the default C locale, there is no thousands separator so a leading ‘'’ has no effect.
An integer block size can be followed by a suffix to specify a multiple of that size. A bare size letter, or one followed by ‘iB’, specifies a multiple using powers of 1024. A size letter followed by ‘B’ specifies powers of 1000 instead. For example, ‘1M’ and ‘1MiB’ are equivalent to ‘1048576’, whereas ‘1MB’ is equivalent to ‘1000000’.
A plain suffix without a preceding integer acts as if ‘1’ were prepended, except that it causes a size indication to be appended to the output. For example, ‘--block-size="kB"’ displays 3000 as ‘3kB’.
The following suffixes are defined. Large sizes like 1Y
may be rejected by your computer due to limitations of its arithmetic.
Block size defaults can be overridden by an explicit --block-size=size option. The -k option is equivalent to --block-size=1K, which is the default unless the POSIXLY_CORRECT environment variable is set. The -h or --human-readable option is equivalent to --block-size=human-readable. The --si option is equivalent to --block-size=si.
A signal may be a signal name like ‘HUP’, or a signal number like ‘1’, or an exit status of a process terminated by the signal. A signal name can be given in canonical form or prefixed by ‘SIG’. The case of the letters is ignored. The following signal names and numbers are supported on all POSIX compliant systems:
Other supported signal names have system-dependent corresponding numbers. All systems conforming to POSIX 1003.1-2001 also support the following signals:
POSIX 1003.1-2001 systems that support the XSI extension also support the following signals:
POSIX 1003.1-2001 systems that support the XRT extension also support at least eight real-time signals called ‘RTMIN’, ‘RTMIN+1’, ..., ‘RTMAX-1’, ‘RTMAX’.
Since the owner and group arguments to chown and
chgrp may be specified as names or numeric IDs, there is an
apparent ambiguity.
What if a user or group name is a string of digits?
1
Should the command interpret it as a user name or as an ID?
POSIX requires that chown and chgrp
first attempt to resolve the specified string as a name, and
only once that fails, then try to interpret it as an ID.
This is troublesome when you want to specify a numeric ID, say 42,
and it must work even in a pathological situation where
‘42’ is a user name that maps to some other user ID, say 1000.
Simply invoking chown 42 F, will set Fs owner ID to
1000—not what you intended.
GNU chown and chgrp provide a way to work around this, that at the same time may result in a significant performance improvement by eliminating a database look-up. Simply precede each numeric user ID and/or group ID with a ‘+’, in order to force its interpretation as an integer:
chown +42 F
chgrp +$numeric_group_id another-file
chown +0:+0 /
GNU chown and chgrp skip the name look-up process for each ‘+’-prefixed string, because a string containing ‘+’ is never a valid user or group name. This syntax is accepted on most common Unix systems, but not on Solaris 10.
The shuf, shred, and sort commands sometimes need random data to do their work. For example, ‘sort -R’ must choose a hash function at random, and it needs random data to make this selection.
Normally these commands use the device file /dev/urandom as the source of random data. Typically, this device gathers environmental noise from device drivers and other sources into an entropy pool, and uses the pool to generate random bits. If the pool is short of data, the device reuses the internal pool to produce more bits, using a cryptographically secure pseudorandom number generator.
/dev/urandom suffices for most practical uses, but applications requiring high-value or long-term protection of private data may require an alternate data source like /dev/random or /dev/arandom. The set of available sources depends on your operating system.
To use such a source, specify the --random-source=file option, e.g., ‘shuf --random-source=/dev/random’. The contents of file should be as random as possible. An error is reported if file does not contain enough bytes to randomize the input adequately.
To reproduce the results of an earlier invocation of a command, you can save some random data into a file and then use that file as the random source in earlier and later invocations of the command.
Some old-fashioned or stripped-down operating systems lack support for /dev/urandom. On these systems commands like shuf by default fall back on an internal pseudorandom generator initialized by a small amount of entropy.
The cp, install, ln, and mv commands normally treat the last operand specially when it is a directory or a symbolic link to a directory. For example, ‘cp source dest’ is equivalent to ‘cp source dest/source’ if dest is a directory. Sometimes this behavior is not exactly what is wanted, so these commands support the following options to allow more fine-grained control:
In the opposite situation, where you want the last operand to be
treated as a directory and want a diagnostic otherwise, you can use
the --target-directory (-t) option.
The interface for most programs is that after processing options and a finite (possibly zero) number of fixed-position arguments, the remaining argument list is either expected to be empty, or is a list of items (usually files) that will all be handled identically. The xargs program is designed to work well with this convention.
The commands in the mv-family are unusual in that they take
a variable number of arguments with a special case at the end
(namely, the target directory). This makes it nontrivial to perform some
operations, e.g., “move all files from here to ../d/”, because
mv * ../d/ might exhaust the argument space, and ls | xargs ...
doesn't have a clean way to specify an extra final argument for each
invocation of the subject command. (It can be done by going through a
shell command, but that requires more human labor and brain power than
it should.)
The --target-directory (-t) option allows the cp,
install, ln, and mv programs to be used
conveniently with xargs. For example, you can move the files
from the current directory to a sibling directory, d like this:
ls | xargs mv -t ../d --
However, this doesn't move files whose names begin with ‘.’. If you use the gnu find program, you can move those files too, with this command:
find . -mindepth 1 -maxdepth 1 \
| xargs mv -t ../d
But both of the above approaches fail if there are no files in the current directory, or if any file has a name containing a blank or some other special characters. The following example removes those limitations and requires both gnu find and gnu xargs:
find . -mindepth 1 -maxdepth 1 -print0 \
| xargs --null --no-run-if-empty \
mv -t ../d
The --target-directory (-t) and --no-target-directory (-T) options cannot be combined.
Some gnu programs (at least cp and mv) allow you to remove any trailing slashes from each source argument before operating on it. The --strip-trailing-slashes option enables this behavior.
This is useful when a source argument may have a trailing slash and specify a symbolic link to a directory. This scenario is in fact rather common because some shells can automatically append a trailing slash when performing file name completion on such symbolic links. Without this option, mv, for example, (via the system's rename function) must interpret a trailing slash as a request to dereference the symbolic link and so must rename the indirectly referenced directory and not the symbolic link. Although it may seem surprising that such behavior be the default, it is required by POSIX and is consistent with other parts of that standard.
The following options modify how chown and chgrp traverse a hierarchy when the --recursive (-R) option is also specified. If more than one of the following options is specified, only the final one takes effect. These options specify whether processing a symbolic link to a directory entails operating on just the symbolic link or on all files in the hierarchy rooted at that directory.
These options are independent of --dereference and --no-dereference (-h), which control whether to modify a symlink or its referent.
Certain commands can operate destructively on entire hierarchies. For example, if a user with appropriate privileges mistakenly runs ‘rm -rf / tmp/junk’, that may remove all files on the entire system. Since there are so few legitimate uses for such a command, gnu rm normally declines to operate on any directory that resolves to /. If you really want to try to remove all the files on your system, you can use the --no-preserve-root option, but the default behavior, specified by the --preserve-option, is safer for most purposes.
The commands chgrp, chmod and chown can also operate destructively on entire hierarchies, so they too support these options. Although, unlike rm, they don't actually unlink files, these commands are arguably more dangerous when operating recursively on /, since they often work much more quickly, and hence damage more files before an alert user can interrupt them. Tradition and POSIX require these commands to operate recursively on /, so they default to --no-preserve-root, but using the --preserve-root option makes them safer for most purposes. For convenience you can specify --preserve-root in an alias or in a shell function.
Note that the --preserve-root option also ensures that chgrp and chown do not modify / even when dereferencing a symlink pointing to /.
Some programs like nice can invoke other programs; for example, the command ‘nice cat file’ invokes the program cat by executing the command ‘cat file’. However, special built-in utilities like exit cannot be invoked this way. For example, the command ‘nice exit’ does not have a well-defined behavior: it may generate an error message instead of exiting.
Here is a list of the special built-in utilities that are standardized by POSIX 1003.1-2004.
. : break continue eval exec exit export readonly return set shift times trap unset
For example, because ‘.’, ‘:’, and ‘exec’ are special, the commands ‘nice . foo.sh’, ‘nice :’, and ‘nice exec pwd’ do not work as you might expect.
Many shells extend this list. For example, Bash has several extra special built-in utilities like history, and suspend, and with Bash the command ‘nice suspend’ generates an error message instead of suspending.
In a few cases, the gnu utilities' default behavior is incompatible with the POSIX standard. To suppress these incompatibilities, define the POSIXLY_CORRECT environment variable. Unless you are checking for POSIX conformance, you probably do not need to define POSIXLY_CORRECT.
Newer versions of POSIX are occasionally incompatible with older versions. For example, older versions of POSIX required the command ‘sort +1’ to sort based on the second and succeeding fields in each input line, but starting with POSIX 1003.1-2001 the same command is required to sort the file named +1, and you must instead use the command ‘sort -k 2’ to get the field-based sort.
The gnu utilities normally conform to the version of POSIX that is standard for your system. To cause them to conform to a different version of POSIX, define the _POSIX2_VERSION environment variable to a value of the form yyyymm specifying the year and month the standard was adopted. Two values are currently supported for _POSIX2_VERSION: ‘199209’ stands for POSIX 1003.2-1992, and ‘200112’ stands for POSIX 1003.1-2001. For example, if you have a newer system but are running software that assumes an older version of POSIX and uses ‘sort +1’ or ‘tail +10’, you can work around any compatibility problems by setting ‘_POSIX2_VERSION=199209’ in your environment.
These commands read and write entire files, possibly transforming them in some way.
cat copies each file (‘-’ means standard input), or standard input if none are given, to standard output. Synopsis:
cat [option] [file]...
The program accepts the following options. Also see Common options.
On systems like MS-DOS that distinguish between text and binary files, cat normally reads and writes in binary mode. However, cat reads in text mode if one of the options -bensAE is used or if cat is reading from standard input and standard input is a terminal. Similarly, cat writes in text mode if one of the options -bensAE is used or if standard output is a terminal.
An exit status of zero indicates success, and a nonzero value indicates failure.
Examples:
# Output f's contents, then standard input, then g's contents.
cat f - g
# Copy standard input to standard output.
cat
tac copies each file (‘-’ means standard input), or standard input if none are given, to standard output, reversing the records (lines by default) in each separately. Synopsis:
tac [option]... [file]...
Records are separated by instances of a string (newline by default). By default, this separator string is attached to the end of the record that it follows in the file.
The program accepts the following options. Also see Common options.
An exit status of zero indicates success, and a nonzero value indicates failure.
nl writes each file (‘-’ means standard input), or standard input if none are given, to standard output, with line numbers added to some or all of the lines. Synopsis:
nl [option]... [file]...
nl decomposes its input into (logical) pages; by default, the line number is reset to 1 at the top of each logical page. nl treats all of the input files as a single document; it does not reset line numbers or logical pages between files.
A logical page consists of three sections: header, body, and footer. Any of the sections can be empty. Each can be numbered in a different style from the others.
The beginnings of the sections of logical pages are indicated in the input file by a line containing exactly one of these delimiter strings:
The two characters from which these strings are made can be changed from ‘\’ and ‘:’ via options (see below), but the pattern and length of each string cannot be changed.
A section delimiter is replaced by an empty line on output. Any text that comes before the first section delimiter string in the input file is considered to be part of a body section, so nl treats a file that contains no section delimiters as a single body section.
The program accepts the following options. Also see Common options.
rn):
An exit status of zero indicates success, and a nonzero value indicates failure.
od writes an unambiguous representation of each file (‘-’ means standard input), or standard input if none are given. Synopses:
od [option]... [file]...
od [-abcdfilosx]... [file] [[+]offset[.][b]]
od [option]... --traditional [file] [[+]offset[.][b] [[+]label[.][b]]]
Each line of output consists of the offset in the input, followed by
groups of data from the file. By default, od prints the offset in
octal, and each group of file data is a C short int's worth of input
printed as a single octal number.
If offset is given, it specifies how many input bytes to skip before formatting and writing. By default, it is interpreted as an octal number, but the optional trailing decimal point causes it to be interpreted as decimal. If no decimal is specified and the offset begins with ‘0x’ or ‘0X’ it is interpreted as a hexadecimal number. If there is a trailing ‘b’, the number of bytes skipped will be offset multiplied by 512.
If a command is of both the first and second forms, the second form is assumed if the last operand begins with ‘+’ or (if there are two operands) a digit. For example, in ‘od foo 10’ and ‘od +10’ the ‘10’ is an offset, whereas in ‘od 10’ the ‘10’ is a file name.
The program accepts the following options. Also see Common options.
The default is octal.
‘b’ => 512 ("blocks") ‘KB’ => 1000 (KiloBytes) ‘K’ => 1024 (KibiBytes) ‘MB’ => 1000*1000 (MegaBytes) ‘M’ => 1024*1024 (MebiBytes) ‘GB’ => 1000*1000*1000 (GigaBytes) ‘G’ => 1024*1024*1024 (GibiBytes)
and so on for ‘T’, ‘P’, ‘E’, ‘Z’, and ‘Y’.
bytes are interpreted as for the -j option.
bytes are interpreted as for the
-j option.
If n is omitted with --strings, the default is 3.
Adding a trailing “z” to any type specification appends a display of the ASCII character representation of the printable characters to the output line generated by the type specification.
The type a outputs things like ‘sp’ for space, ‘nl’ for
newline, and ‘nul’ for a zero byte. Only the least significant
seven bits of each byte is used; the high-order bit is ignored.
Type c outputs
‘ ’, ‘\n’, and \0, respectively.
Except for types ‘a’ and ‘c’, you can specify the number of bytes to use in interpreting each number in the given data type by following the type indicator character with a decimal integer. Alternately, you can specify the size of one of the C compiler's built-in data types by following the type indicator character with one of the following characters. For integers (‘d’, ‘o’, ‘u’, ‘x’):
For floating point (f):
n input bytes per output line. This must be a multiple of
the least common multiple of the sizes associated with the specified
output types.
If this option is not given at all, the default is 16. If n is omitted, the default is 32.
The next several options are shorthands for format specifications. gnu od accepts any combination of shorthands and format specification options. These options accumulate.
od --traditional [file] [[+]offset[.][b] [[+]label[.][b]]]
can be used to specify at most one file and optional arguments specifying an offset and a pseudo-start address, label. The label argument is interpreted just like offset, but it specifies an initial pseudo-address. The pseudo-addresses are displayed in parentheses following any normal address.
An exit status of zero indicates success, and a nonzero value indicates failure.
base64 transforms data read from a file, or standard input, into (or from) base64 encoded form. The base64 encoded form uses printable ASCII characters to represent binary data. Synopses:
base64 [option]... [file]
base64 --decode [option]... [file]
The base64 encoding expands data to roughly 133% of the original. The format conforms to RFC 4648.
The program accepts the following options. Also see Common options.
The default is to wrap after 76 characters. Use the value 0 to
disable line wrapping altogether.
An exit status of zero indicates success, and a nonzero value indicates failure.
These commands reformat the contents of files.
fmt fills and joins lines to produce output lines of (at most) a given number of characters (75 by default). Synopsis:
fmt [option]... [file]...
fmt reads from the specified file arguments (or standard input if none are given), and writes to standard output.
By default, blank lines, spaces between words, and indentation are preserved in the output; successive input lines with different indentation are not joined; tabs are expanded on input and introduced on output.
fmt prefers breaking lines at the end of a sentence, and tries to avoid line breaks after the first word of a sentence or before the last word of a sentence. A sentence break is defined as either the end of a paragraph or a word ending in any of ‘.?!’, followed by two spaces or end of line, ignoring any intervening parentheses or quotes. Like TeX, fmt reads entire “paragraphs” before choosing line breaks; the algorithm is a variant of that given by Donald E. Knuth and Michael F. Plass in “Breaking Paragraphs Into Lines”, Software—Practice & Experience 11, 11 (November 1981), 1119–1184.
The program accepts the following options. Also see Common options.
An exit status of zero indicates success, and a nonzero value indicates failure.
pr writes each file (‘-’ means standard input), or standard input if none are given, to standard output, paginating and optionally outputting in multicolumn format; optionally merges all files, printing all in parallel, one per column. Synopsis:
pr [option]... [file]...
By default, a 5-line header is printed at each page: two blank lines; a line with the date, the file name, and the page count; and two more blank lines. A footer of five blank lines is also printed. The default page_length is 66 lines. The default number of text lines is therefore 56. The text line of the header takes the form ‘date string page’, with spaces inserted around string so that the line takes up the full page_width. Here, date is the date (see the -D or --date-format option for details), string is the centered header string, and page identifies the page number. The LC_MESSAGES locale category affects the spelling of page; in the default C locale, it is ‘Page number’ where number is the decimal page number.
Form feeds in the input cause page breaks in the output. Multiple form feeds produce empty pages.
Columns are of equal width, separated by an optional string (default is ‘space’). For multicolumn output, lines will always be truncated to page_width (default 72), unless you use the -J option. For single column output no line truncation occurs by default. Use -W option to truncate lines in that case.
The following changes were made in version 1.22i and apply to later versions of pr: - Brian
The program accepts the following options. Also see Common options.
The default date format is ‘%Y-%m-%d %H:%M’ (for example, ‘2001-12-04 23:59’); but if the POSIXLY_CORRECT environment variable is set and the LC_TIME locale category specifies the POSIX locale, the default is ‘%b %e %H:%M %Y’ (for example, ‘Dec 4 23:59 2001’.
Time stamps are listed according to the time zone rules specified by
the TZ environment variable, or by the system default rules if
TZ is not set. See Specifying the Time Zone with TZ.
An exit status of zero indicates success, and a nonzero value indicates failure.
fold writes each file (- means standard input), or standard input if none are given, to standard output, breaking long lines. Synopsis:
fold [option]... [file]...
By default, fold breaks lines wider than 80 columns. The output is split into as many lines as necessary.
fold counts screen columns by default; thus, a tab may count more than one column, backspace decreases the column count, and carriage return sets the column to zero.
The program accepts the following options. Also see Common options.
For compatibility fold supports an obsolete option syntax -width. New scripts should use -w width instead.
An exit status of zero indicates success, and a nonzero value indicates failure.
These commands output pieces of the input.
head prints the first part (10 lines by default) of each file; it reads from standard input if no files are given or when given a file of -. Synopsis:
head [option]... [file]...
If more than one file is specified, head prints a one-line header consisting of:
==> file name <==
before the output for each file.
The program accepts the following options. Also see Common options.
‘b’ => 512 ("blocks") ‘KB’ => 1000 (KiloBytes) ‘K’ => 1024 (KibiBytes) ‘MB’ => 1000*1000 (MegaBytes) ‘M’ => 1024*1024 (MebiBytes) ‘GB’ => 1000*1000*1000 (GigaBytes) ‘G’ => 1024*1024*1024 (GibiBytes)
and so on for ‘T’, ‘P’, ‘E’, ‘Z’, and ‘Y’.
For compatibility head also supports an obsolete option syntax -countoptions, which is recognized only if it is specified first. count is a decimal number optionally followed by a size letter (‘b’, ‘k’, ‘m’) as in -c, or ‘l’ to mean count by lines, or other option letters (‘cqv’). Scripts intended for standard hosts should use -c count or -n count instead. If your script must also run on hosts that support only the obsolete syntax, it is usually simpler to avoid head, e.g., by using ‘sed 5q’ instead of ‘head -5’.
An exit status of zero indicates success, and a nonzero value indicates failure.
tail prints the last part (10 lines by default) of each file; it reads from standard input if no files are given or when given a file of ‘-’. Synopsis:
tail [option]... [file]...
If more than one file is specified, tail prints a one-line header consisting of:
==> file name <==
before the output for each file.
gnu tail can output any amount of data (some other versions of tail cannot). It also has no -r option (print in reverse), since reversing a file is really a different job from printing the end of a file; BSD tail (which is the one with -r) can only reverse files that are at most as large as its buffer, which is typically 32 KiB. A more reliable and versatile way to reverse files is the gnu tac command.
The program accepts the following options. Also see Common options.
‘b’ => 512 ("blocks") ‘KB’ => 1000 (KiloBytes) ‘K’ => 1024 (KibiBytes) ‘MB’ => 1000*1000 (MegaBytes) ‘M’ => 1024*1024 (MebiBytes) ‘GB’ => 1000*1000*1000 (GigaBytes) ‘G’ => 1024*1024*1024 (GibiBytes)
and so on for ‘T’, ‘P’, ‘E’, ‘Z’, and ‘Y’.
There are two ways to specify how you'd like to track files with this option, but that difference is noticeable only when a followed file is removed or renamed. If you'd like to continue to track the end of a growing file even after it has been unlinked, use --follow=descriptor. This is the default behavior, but it is not useful if you're tracking a log file that may be rotated (removed or renamed, then reopened). In that case, use --follow=name to track the named file by reopening it periodically to see if it has been removed and recreated by some other program.
No matter which method you use, if the tracked file is determined to have shrunk, tail prints a message saying the file has been truncated and resumes tracking the end of the file from the newly-determined endpoint.
When a file is removed, tail's behavior depends on whether it is following the name or the descriptor. When following by name, tail can detect that a file has been removed and gives a message to that effect, and if --retry has been specified it will continue checking periodically to see if the file reappears. When following a descriptor, tail does not detect that the file has been unlinked or renamed and issues no message; even though the file may no longer be accessible via its original name, it may still be growing.
The option values ‘descriptor’ and ‘name’ may be specified only with the long form of the option, not with -f.
If POSIXLY_CORRECT is set, the -f option is ignored if
no file operand is specified and standard input is a FIFO or a pipe.
tail -f
process yourself.
$ make >& makerr & tail --pid=$! -f makerr
If you specify a pid that is not in use or that does not correspond to the process that is writing to the tailed files, then tail may terminate long before any files stop growing or it may not terminate until long after the real writer has terminated. Note that --pid cannot be supported on some systems; tail will print a warning if this is the case.
open/fstat the file to determine if that file name is
still associated with the same device/inode-number pair as before.
When following a log file that is rotated, this is approximately the
number of seconds between when tail prints the last pre-rotation lines
and when it prints the lines that have accumulated in the new log file.
This option is meaningful only when following by name.
For compatibility tail also supports an obsolete usage ‘tail -[count][bcl][f] [file]’, which is recognized only if it does not conflict with the usage described above. This obsolete form uses exactly one option and at most one file. In the option, count is an optional decimal number optionally followed by a size letter (‘b’, ‘c’, ‘l’) to mean count by 512-byte blocks, bytes, or lines, optionally followed by ‘f’ which has the same meaning as -f.
On older systems, the leading ‘-’ can be replaced by ‘+’ in the obsolete option syntax with the same meaning as in counts, and obsolete usage overrides normal usage when the two conflict. This obsolete behavior can be enabled or disabled with the _POSIX2_VERSION environment variable (see Standards conformance).
Scripts intended for use on standard hosts should avoid obsolete syntax and should use -c count[b], -n count, and/or -f instead. If your script must also run on hosts that support only the obsolete syntax, you can often rewrite it to avoid problematic usages, e.g., by using ‘sed -n '$p'’ rather than ‘tail -1’. If that's not possible, the script can use a test like ‘if tail -c +1 </dev/null >/dev/null 2>&1; then ...’ to decide which syntax to use.
Even if your script assumes the standard behavior, you should still beware usages whose behaviors differ depending on the POSIX version. For example, avoid ‘tail - main.c’, since it might be interpreted as either ‘tail main.c’ or as ‘tail -- - main.c’; avoid ‘tail -c 4’, since it might mean either ‘tail -c4’ or ‘tail -c 10 4’; and avoid ‘tail +4’, since it might mean either ‘tail ./+4’ or ‘tail -n +4’.
An exit status of zero indicates success, and a nonzero value indicates failure.
split creates output files containing consecutive sections of input (standard input if none is given or input is ‘-’). Synopsis:
split [option] [input [prefix]]
By default, split puts 1000 lines of input (or whatever is left over for the last section), into each output file.
The output files' names consist of prefix (‘x’ by default) followed by a group of characters (‘aa’, ‘ab’, ... by default), such that concatenating the output files in traditional sorted order by file name produces the original input file. If the output file names are exhausted, split reports an error without deleting the output files that it did create.
The program accepts the following options. Also see Common options.
For compatibility split also supports an obsolete
option syntax -lines. New scripts should use -l
lines instead.
‘b’ => 512 ("blocks") ‘KB’ => 1000 (KiloBytes) ‘K’ => 1024 (KibiBytes) ‘MB’ => 1000*1000 (MegaBytes) ‘M’ => 1024*1024 (MebiBytes) ‘GB’ => 1000*1000*1000 (GigaBytes) ‘G’ => 1024*1024*1024 (GibiBytes)
and so on for ‘T’, ‘P’, ‘E’, ‘Z’, and ‘Y’.
An exit status of zero indicates success, and a nonzero value indicates failure.
csplit creates zero or more output files containing sections of input (standard input if input is ‘-’). Synopsis:
csplit [option]... input pattern...
The contents of the output files are determined by the pattern arguments, as detailed below. An error occurs if a pattern argument refers to a nonexistent line of the input file (e.g., if no remaining line matches a given regular expression). After every pattern has been matched, any remaining input is copied into one last output file.
By default, csplit prints the number of bytes written to each output file after it has been created.
The types of pattern arguments are:
The output files' names consist of a prefix (‘xx’ by default) followed by a suffix. By default, the suffix is an ascending sequence of two-digit decimal numbers from ‘00’ to ‘99’. In any case, concatenating the output files in sorted order by file name produces the original input file.
By default, if csplit encounters an error or receives a hangup, interrupt, quit, or terminate signal, it removes any output files that it has created so far before it exits.
The program accepts the following options. Also see Common options.
printf(3)-style conversion specification, possibly including
format specification flags, a field width, a precision specifications,
or all of these kinds of modifiers. The format letter must convert a
binary integer argument to readable form; thus, only ‘d’, ‘i’,
‘u’, ‘o’, ‘x’, and ‘X’ conversions are allowed. The
entire suffix is given (with the current output file number) to
sprintf(3) to form the file name suffixes for each of the
individual output files in turn. If this option is used, the
--digits option is ignored.
An exit status of zero indicates success, and a nonzero value indicates failure.
Here is an example of its usage. First, create an empty directory for the exercise, and cd into it:
$ mkdir d && cd d
Now, split the sequence of 1..14 on lines that end with 0 or 5:
$ seq 14 | csplit - '/[05]$/' '{*}'
8
10
15
Each number printed above is the size of an output file that csplit has just created. List the names of those output files:
$ ls
xx00 xx01 xx02
Use head to show their contents:
$ head xx*
==> xx00 <==
1
2
3
4
==> xx01 <==
5
6
7
8
9
==> xx02 <==
10
11
12
13
14
These commands generate just a few numbers representing entire contents of files.
wc counts the number of bytes, characters, whitespace-separated words, and newlines in each given file, or standard input if none are given or for a file of ‘-’. Synopsis:
wc [option]... [file]...
wc prints one line of counts for each file, and if the file was given as an argument, it prints the file name following the counts. If more than one file is given, wc prints a final line containing the cumulative counts, with the file name total. The counts are printed in this order: newlines, words, characters, bytes, maximum line length. Each count is printed right-justified in a field with at least one space between fields so that the numbers and file names normally line up nicely in columns. The width of the count fields varies depending on the inputs, so you should not depend on a particular field width. However, as a GNU extension, if only one count is printed, it is guaranteed to be printed without leading spaces.
By default, wc prints three counts: the newline, words, and byte counts. Options can specify that only certain counts be printed. Options do not undo others previously given, so
wc --bytes --words
prints both the byte counts and the word counts.
With the --max-line-length option, wc prints the length of the longest line per file, and if there is more than one file it prints the maximum (not the sum) of those lengths. The line lengths here are measured in screen columns, according to the current locale and assuming tab positions in every 8th column.
The program accepts the following options. Also see Common options.
For example, to find the length of the longest line in any .c or .h file in the current hierarchy, do this:
find . -name '*.[ch]' -print0 |
wc -L --files0-from=- | tail -n1
An exit status of zero indicates success, and a nonzero value indicates failure.
sum computes a 16-bit checksum for each given file, or standard input if none are given or for a file of ‘-’. Synopsis:
sum [option]... [file]...
sum prints the checksum for each file followed by the number of blocks in the file (rounded up). If more than one file is given, file names are also printed (by default). (With the --sysv option, corresponding file names are printed when there is at least one file argument.)
By default, gnu sum computes checksums using an algorithm compatible with BSD sum and prints file sizes in units of 1024-byte blocks.
The program accepts the following options. Also see Common options.
sum is provided for compatibility; the cksum program (see next section) is preferable in new applications.
An exit status of zero indicates success, and a nonzero value indicates failure.
cksum computes a cyclic redundancy check (CRC) checksum for each given file, or standard input if none are given or for a file of ‘-’. Synopsis:
cksum [option]... [file]...
cksum prints the CRC checksum for each file along with the number of bytes in the file, and the file name unless no arguments were given.
cksum is typically used to ensure that files transferred by unreliable means (e.g., netnews) have not been corrupted, by comparing the cksum output for the received files with the cksum output for the original files (typically given in the distribution).
The CRC algorithm is specified by the POSIX standard. It is not compatible with the BSD or System V sum algorithms (see the previous section); it is more robust.
The only options are --help and --version. See Common options.
An exit status of zero indicates success, and a nonzero value indicates failure.
md5sum computes a 128-bit checksum (or fingerprint or message-digest) for each specified file.
Note: The MD5 digest is more reliable than a simple CRC (provided by the cksum command) for detecting accidental file corruption, as the chances of accidentally having two files with identical MD5 are vanishingly small. However, it should not be considered truly secure against malicious tampering: although finding a file with a given MD5 fingerprint, or modifying a file so as to retain its MD5 are considered infeasible at the moment, it is known how to produce different files with identical MD5 (a “collision”), something which can be a security issue in certain contexts. For more secure hashes, consider using SHA-1 or SHA-2. See sha1sum invocation, and sha2 utilities.
If a file is specified as ‘-’ or if no files are given md5sum computes the checksum for the standard input. md5sum can also determine whether a file and checksum are consistent. Synopsis:
md5sum [option]... [file]...
For each file, ‘md5sum’ outputs the MD5 checksum, a flag indicating a binary or text input file, and the file name. If file contains a backslash or newline, the line is started with a backslash, and each problematic character in the file name is escaped with a backslash, making the output unambiguous even in the presence of arbitrary file names. If file is omitted or specified as ‘-’, standard input is read.
The program accepts the following options. Also see Common options.
An exit status of zero indicates success, and a nonzero value indicates failure.
sha1sum computes a 160-bit checksum for each specified file. The usage and options of this command are precisely the same as for md5sum. See md5sum invocation.
Note: The SHA-1 digest is more secure than MD5, and no collisions of it are known (different files having the same fingerprint). However, it is known that they can be produced with considerable, but not unreasonable, resources. For this reason, it is generally considered that SHA-1 should be gradually phased out in favor of the more secure SHA-2 hash algorithms. See sha2 utilities.
The commands sha224sum, sha256sum, sha384sum and sha512sum compute checksums of various lengths (respectively 224, 256, 384 and 512 bits), collectively known as the SHA-2 hashes. The usage and options of these commands are precisely the same as for md5sum. See md5sum invocation.
Note: The SHA384 and SHA512 digests are considerably slower to compute, especially on 32-bit computers, than SHA224 or SHA256.
These commands work with (or produce) sorted files.
sort sorts, merges, or compares all the lines from the given files, or standard input if none are given or for a file of ‘-’. By default, sort writes the results to standard output. Synopsis:
sort [option]... [file]...
sort has three modes of operation: sort (the default), merge, and check for sortedness. The following options change the operation mode:
A pair of lines is compared as follows: sort compares each pair of fields, in the order specified on the command line, according to the associated ordering options, until a difference is found or no fields are left. If no key fields are specified, sort uses a default key of the entire line. Finally, as a last resort when all keys compare equal, sort compares entire lines as if no ordering options other than --reverse (-r) were specified. The --stable (-s) option disables this last-resort comparison so that lines in which all fields compare equal are left in their original relative order. The --unique (-u) option also disables the last-resort comparison.
Unless otherwise specified, all comparisons use the character collating sequence specified by the LC_COLLATE locale.2
gnu sort (as specified for all gnu utilities) has no limit on input line length or restrictions on bytes allowed within lines. In addition, if the final byte of an input file is not a newline, gnu sort silently supplies one. A line's trailing newline is not part of the line for comparison purposes.
0 if no error occurred
1 if invoked with -c or -C and the input is not sorted
2 if an error occurred
If the environment variable TMPDIR is set, sort uses its value as the directory for temporary files instead of /tmp. The --temporary-directory (-T) option in turn overrides the environment variable.
The following options affect the ordering of output lines. They may be specified globally or as part of a specific key field. If no key fields are specified, global options apply to comparison of entire lines; otherwise the global options are inherited by key fields that do not specify any special options of their own. In pre-POSIX versions of sort, global options affect only later key fields, so portable shell scripts should specify global options first.
strtod to convert
a prefix of each line to a double-precision floating point number.
This allows floating point numbers to be specified in scientific notation,
like 1.0e-34 and 10e100.
The LC_NUMERIC locale determines the decimal-point character.
Do not report overflow, underflow, or conversion errors.
Use the following collating sequence:
Use this option only if there is no alternative; it is much slower than
--numeric-sort (-n) and it can lose information when
converting to floating point.
Comparison is exact; there is no rounding error.
Neither a leading ‘+’ nor exponential notation is recognized.
To compare such strings numerically, use the
--general-numeric-sort (-g) option.
strverscmp(3). This is a normal string comparison, except
that embedded decimal numbers are sorted by numeric value
(see --numeric-sort above).
If multiple random sort fields are specified, the same random hash function is used for all fields. To use different random hash functions for different fields, you can invoke sort more than once.
The choice of hash function is affected by the --random-source option.
Other options are:
With no arguments, prog must compress standard input to standard output, and when given the -d option it must decompress standard input to standard output.
Terminate with an error if prog exits with nonzero status.
White space and the backslash character should not appear in prog; they are reserved for future use.
Each pos has the form ‘f[.c][opts]’, where f is the number of the field to use, and c is the number of the first character from the beginning of the field. Fields and character positions are numbered starting with 1; a character position of zero in pos2 indicates the field's last character. If ‘.c’ is omitted from pos1, it defaults to 1 (the beginning of the field); if omitted from pos2, it defaults to 0 (the end of the field). opts are ordering options, allowing individual keys to be sorted according to different rules; see below for details. Keys can span multiple fields.
Example: To sort on the second field, use --key=2,2
(-k 2,2). See below for more examples.
When sort has to merge more than nmerge inputs, it merges them in groups of nmerge, saving the result in a temporary file, which is then used as an input in a subsequent merge.
A large value of nmerge may improve merge performance and decrease temporary storage utilization at the expense of increased memory usage and I/0. Conversely a small value of nmerge may reduce memory requirements and I/0 at the expense of temporary storage consumption and merge performance.
The value of nmerge must be at least 2. The default value is currently 16, but this is implementation-dependent and may change in the future.
The value of nmerge may be bounded by a resource limit for open
file descriptors. The commands ‘ulimit -n’ or ‘getconf
OPEN_MAX’ may display limits for your systems; these limits may be
modified further if your program already has some files open, or if
the operating system has other limits on the number of open files. If
the value of nmerge exceeds the resource limit, sort
silently uses a smaller value.
sort -o F F and cat F | sort -o F.
However, sort with --merge (-m) can open
the output file before reading all input, so a command like cat
F | sort -m -o F - G is not safe as sort might start
writing F before cat is done reading it.
On newer systems, -o cannot appear after an input file if
POSIXLY_CORRECT is set, e.g., ‘sort F -o F’. Portable
scripts should specify -o output-file before any input
files.
This option can improve the performance of sort by causing it
to start with a larger or smaller sort buffer than the default.
However, this option affects only the initial buffer size. The buffer
grows beyond size if sort encounters input lines larger
than size.
That is, given the input line ‘ foo bar’, sort breaks it into fields ‘ foo’ and ‘ bar’. The field separator is not considered to be part of either the field preceding or the field following, so with ‘sort -t " "’ the same input line has three fields: an empty field, ‘foo’, and ‘bar’. However, fields that extend to the end of the line, as -k 2, or fields consisting of a range, as -k 2,3, retain the field separators present between the endpoints of the range.
To specify ASCII nul as the field separator,
use the two-character string ‘\0’, e.g., ‘sort -t '\0'’.
This option also disables the default last-resort comparison.
The commands sort -u and sort | uniq are equivalent, but
this equivalence does not extend to arbitrary sort options.
For example, sort -n -u inspects only the value of the initial
numeric string when checking for uniqueness, whereas sort -n |
uniq inspects the entire line. See uniq invocation.
Historical (BSD and System V) implementations of sort have differed in their interpretation of some options, particularly -b, -f, and -n. gnu sort follows the POSIX behavior, which is usually (but not always!) like the System V behavior. According to POSIX, -n no longer implies -b. For consistency, -M has been changed in the same way. This may affect the meaning of character positions in field specifications in obscure cases. The only fix is to add an explicit -b.
A position in a sort field specified with -k may have any of the option letters ‘Mbdfinr’ appended to it, in which case the global ordering options are not used for that particular field. The -b option may be independently attached to either or both of the start and end positions of a field specification, and if it is inherited from the global options it will be attached to both. If input lines can contain leading or adjacent blanks and -t is not used, then -k is typically combined with -b, -g, -M, or -n; otherwise the varying numbers of leading blanks in fields can cause confusing results.
If the start position in a sort field specifier falls after the end of the line or after the end field, the field is empty. If the -b option was specified, the ‘.c’ part of a field specification is counted from the first nonblank character of the field.
On older systems, sort supports an obsolete origin-zero syntax ‘+pos1 [-pos2]’ for specifying sort keys. This obsolete behavior can be enabled or disabled with the _POSIX2_VERSION environment variable (see Standards conformance); it can also be enabled when POSIXLY_CORRECT is not set by using the obsolete syntax with ‘-pos2’ present.
Scripts intended for use on standard hosts should avoid obsolete syntax and should use -k instead. For example, avoid ‘sort +2’, since it might be interpreted as either ‘sort ./+2’ or ‘sort -k 3’. If your script must also run on hosts that support only the obsolete syntax, it can use a test like ‘if sort -k 1 </dev/null >/dev/null 2>&1; then ...’ to decide which syntax to use.
Here are some examples to illustrate various combinations of options.
sort -n -r
sort -k 3b
sort -t : -k 2,2n -k 5.3,5.4
Note that if you had written -k 2n instead of -k 2,2n sort would have used all characters beginning in the second field and extending to the end of the line as the primary numeric key. For the large majority of applications, treating keys spanning more than one field as numeric will not do what you expect.
Also note that the ‘n’ modifier was applied to the field-end specifier for the first key. It would have been equivalent to specify -k 2n,2 or -k 2n,2n. All modifiers except ‘b’ apply to the associated field, regardless of whether the modifier character is attached to the field-start and/or the field-end part of the key specifier.
sort -t : -k 5b,5 -k 3,3n /etc/passwd
sort -t : -n -k 5b,5 -k 3,3 /etc/passwd
sort -t : -b -k 5,5 -k 3,3n /etc/passwd
These three commands have equivalent effect. The first specifies that the first key's start position ignores leading blanks and the second key is sorted numerically. The other two commands rely on global options being inherited by sort keys that lack modifiers. The inheritance works in this case because -k 5b,5b and -k 5b,5 are equivalent, as the location of a field-end lacking a ‘.c’ character position is not affected by whether initial blanks are skipped.
4.150.156.3 - - [01/Apr/2004:06:31:51 +0000] message 1
211.24.3.231 - - [24/Apr/2004:20:17:39 +0000] message 2
Fields are separated by exactly one space. Sort IPv4 addresses lexicographically, e.g., 212.61.52.2 sorts before 212.129.233.201 because 61 is less than 129.
sort -s -t ' ' -k 4.9n -k 4.5M -k 4.2n -k 4.14,4.21 file*.log |
sort -s -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n
This example cannot be done with a single sort invocation, since IPv4 address components are separated by ‘.’ while dates come just after a space. So it is broken down into two invocations of sort: the first sorts by time stamp and the second by IPv4 address. The time stamp is sorted by year, then month, then day, and finally by hour-minute-second field, using -k to isolate each field. Except for hour-minute-second there's no need to specify the end of each key field, since the ‘n’ and ‘M’ modifiers sort based on leading prefixes that cannot cross field boundaries. The IPv4 addresses are sorted lexicographically. The second sort uses ‘-s’ so that ties in the primary key are broken by the secondary key; the first sort uses ‘-s’ so that the combination of the two sorts is stable.
find src -type f -print0 | sort -z -f | xargs -0 etags --append
The use of -print0, -z, and -0 in this case means that file names that contain blanks or other special characters are not broken up by the sort operation.
ls */* | sort -t / -k 1,1R -k 2,2
shuf shuffles its input by outputting a random permutation of its input lines. Each output permutation is equally likely. Synopses:
shuf [option]... [file]
shuf -e [option]... [arg]...
shuf -i lo-hi [option]...
shuf has three modes of operation that affect where it obtains its input lines. By default, it reads lines from standard input. The following options change the operation mode:
shuf's other options can affect its behavior in all operation modes:
shuf -o F <F and cat F | shuf -o F.
For example:
shuf <<EOF
A man,
a plan,
a canal:
Panama!
EOF
might produce the output
Panama!
A man,
a canal:
a plan,
Similarly, the command:
shuf -e clubs hearts diamonds spades
might output:
clubs
diamonds
spades
hearts
and the command ‘shuf -i 1-4’ might output:
4
2
1
3
These examples all have four input lines, so shuf might produce any of the twenty-four possible permutations of the input. In general, if there are n input lines, there are n! (i.e., n factorial, or n * (n - 1) * ... * 1) possible output permutations.
An exit status of zero indicates success, and a nonzero value indicates failure.
uniq writes the unique lines in the given input, or standard input if nothing is given or for an input name of ‘-’. Synopsis:
uniq [option]... [input [output]]
By default, uniq prints its input lines, except that it discards all but the first of adjacent repeated lines, so that no output lines are repeated. Optionally, it can instead discard lines that are not repeated, or all repeated lines.
The input need not be sorted, but repeated input lines are detected
only if they are adjacent. If you want to discard non-adjacent
duplicate lines, perhaps you want to use sort -u.
See sort invocation.
Comparisons use the character collating sequence specified by the LC_COLLATE locale category.
If no output file is specified, uniq writes to standard output.
The program accepts the following options. Also see Common options.
For compatibility uniq supports an obsolete option syntax
-n. New scripts should use -f n instead.
On older systems, uniq supports an obsolete option syntax
+n.
This obsolete behavior can be enabled or disabled with the
_POSIX2_VERSION environment variable (see Standards conformance), but portable scripts should avoid commands whose
behavior depends on this variable.
For example, use ‘uniq ./+10’ or ‘uniq -s 10’ rather than
the ambiguous ‘uniq +10’.
Note that when groups are delimited and the input stream contains two or more consecutive blank lines, then the output is ambiguous. To avoid that, filter the input through ‘tr -s '\n'’ to replace each sequence of consecutive newlines with a single newline.
This is a gnu extension.
An exit status of zero indicates success, and a nonzero value indicates failure.
comm writes to standard output lines that are common, and lines that are unique, to two input files; a file name of ‘-’ means standard input. Synopsis:
comm [option]... file1 file2
Before comm can be used, the input files must be sorted using the collating sequence specified by the LC_COLLATE locale. If an input file ends in a non-newline character, a newline is silently appended. The sort command with no options always outputs a file that is suitable input to comm.
With no options, comm produces three-column output. Column one contains lines unique to file1, column two contains lines unique to file2, and column three contains lines common to both files. Columns are separated by a single TAB character.
The options -1, -2, and -3 suppress printing of the corresponding columns. Also see Common options.
Unlike some other comparison utilities, comm has an exit status that does not depend on the result of the comparison. Upon normal completion comm produces an exit code of zero. If there is an error it exits with nonzero status.
If the --check-order option is given, unsorted inputs will cause a fatal error message. If the option --nocheck-order is given, unsorted inputs will never cause an error message. If neither of these options is given, wrongly sorted inputs are diagnosed only if an input file is found to contain unpairable lines. If an input file is diagnosed as being unsorted, the comm command will exit with a nonzero status (and the output should not be used).
Forcing comm to process wrongly sorted input files containing unpairable lines by specifying --nocheck-order is not guaranteed to produce any particular output. The output will probably not correspond with whatever you hoped it would be.
Other options are:
The delimiter str may not be empty.
ptx reads a text file and essentially produces a permuted index, with each keyword in its context. The calling sketch is either one of:
ptx [option ...] [file ...]
ptx -G [option ...] [input [output]]
The -G (or its equivalent: --traditional) option disables all gnu extensions and reverts to traditional mode, thus introducing some limitations and changing several of the program's default option values. When -G is not specified, gnu extensions are always enabled. gnu extensions to ptx are documented wherever appropriate in this document. For the full list, see See Compatibility in ptx.
Individual options are explained in the following sections.
When gnu extensions are enabled, there may be zero, one or several files after the options. If there is no file, the program reads the standard input. If there is one or several files, they give the name of input files which are all read in turn, as if all the input files were concatenated. However, there is a full contextual break between each file and, when automatic referencing is requested, file names and line numbers refer to individual text input files. In all cases, the program outputs the permuted index to the standard output.
When gnu extensions are not enabled, that is, when the program operates in traditional mode, there may be zero, one or two parameters besides the options. If there are no parameters, the program reads the standard input and outputs the permuted index to the standard output. If there is only one parameter, it names the text input to be read instead of the standard input. If two parameters are given, they give respectively the name of the input file to read and the name of the output file to produce. Be very careful to note that, in this case, the contents of file given by the second parameter is destroyed. This behavior is dictated by System V ptx compatibility; gnu Standards normally discourage output parameters not introduced by an option.
Note that for any file named as the value of an option or as an input text file, a single dash - may be used, in which case standard input is assumed. However, it would not make sense to use this convention more than once per program invocation.
An exit status of zero indicates success, and a nonzero value indicates failure.
As it is set up now, the program assumes that the input file is coded using 8-bit ISO 8859-1 code, also known as Latin-1 character set, unless it is compiled for MS-DOS, in which case it uses the character set of the IBM-PC. (gnu ptx is not known to work on smaller MS-DOS machines anymore.) Compared to 7-bit ASCII, the set of characters which are letters is different; this alters the behavior of regular expression matching. Thus, the default regular expression for a keyword allows foreign or diacriticized letters. Keyword sorting, however, is still crude; it obeys the underlying character set ordering quite blindly.
When gnu extensions are enabled, the only way to avoid newline as a
break character is to write all the break characters in the file with no
newline at all, not even at the end of the file. When gnu extensions
are disabled, spaces, tabs and newlines are always considered as break
characters even if not included in the Break file.
There is no default for the Only file. When both an Only file and an
Ignore file are specified, a word is considered a keyword only
if it is listed in the Only file and not in the Ignore file.
Using this option, the program does not try very hard to remove
references from contexts in output, but it succeeds in doing so
when the context ends exactly at the newline. If option
-r is used with -S default value, or when gnu extensions
are disabled, this condition is always met and references are completely
excluded from the output contexts.
[.?!][]\"')}]*\\($\\|\t\\| \\)[ \t\n]*
Whenever gnu extensions are disabled or if -r option is used, end of lines are used; in this case, the default regexp is just:
\n
Using an empty regexp is equivalent to completely disabling end of line or end of sentence recognition. In this case, the whole file is considered to be a single big line or sentence. The user might want to disallow all truncation flag generation as well, through option -F "". See Syntax of Regular Expressions.
When the keywords happen to be near the beginning of the input line or sentence, this often creates an unused area at the beginning of the output context line; when the keywords happen to be near the end of the input line or sentence, this often creates an unused area at the end of the output context line. The program tries to fill those unused areas by wrapping around context in them; the tail of the input line or sentence is used to fill the unused area on the left of the output line; the head of the input line or sentence is used to fill the unused area on the right of the output line.
As a matter of convenience to the user, many usual backslashed escape
sequences from the C language are recognized and converted to the
corresponding characters by ptx itself.
An empty regexp is equivalent to not using this option. See Syntax of Regular Expressions.
As a matter of convenience to the user, many usual backslashed escape sequences, as found in the C language, are recognized and converted to the corresponding characters by ptx itself.
Output format is mainly controlled by the -O and -T options
described in the table below. When neither -O nor -T are
selected, and if gnu extensions are enabled, the program chooses an
output format suitable for a dumb terminal. Each keyword occurrence is
output to the center of one line, surrounded by its left and right
contexts. Each field is properly justified, so the concordance output
can be readily observed. As a special feature, if automatic
references are selected by option -A and are output before the
left context, that is, if option -R is not selected, then
a colon is added after the reference; this nicely interfaces with gnu
Emacs next-error processing. In this default output format, each
white space character, like newline and tab, is merely changed to
exactly one space, with no special attempt to compress consecutive
spaces. This might change in the future. Except for those white space
characters, every other character of the underlying set of 256
characters is transmitted verbatim.
Output format is further controlled by the following options.
This option is automatically selected whenever gnu extensions are
disabled.
string may have more than one character, as in -F .... Also, in the particular case when string is empty (-F ""), truncation flagging is disabled, and no truncation marks are appended in this case.
As a matter of convenience to the user, many usual backslashed escape
sequences, as found in the C language, are recognized and converted to
the corresponding characters by ptx itself.
.xx "tail" "before" "keyword_and_after" "head" "ref"
so it will be possible to write a ‘.xx’ roff macro to take care of the output typesetting. This is the default output format when gnu extensions are disabled. Option -M can be used to change ‘xx’ to another macro name.
In this output format, each non-graphical character, like newline and
tab, is merely changed to exactly one space, with no special attempt to
compress consecutive spaces. Each quote character: " is doubled
so it will be correctly processed by nroff or troff.
\xx {tail}{before}{keyword}{after}{head}{ref}
so it will be possible to write a \xx definition to take care of
the output typesetting. Note that when references are not being
produced, that is, neither option -A nor option -r is
selected, the last parameter of each \xx call is inhibited.
Option -M can be used to change ‘xx’ to another macro
name.
In this output format, some special characters, like $, %,
&, # and _ are automatically protected with a
backslash. Curly brackets {, } are protected with a
backslash and a pair of dollar signs (to force mathematical mode). The
backslash itself produces the sequence \backslash{}.
Circumflex and tilde diacritical marks produce the sequence ^\{ } and
~\{ } respectively. Other diacriticized characters of the
underlying character set produce an appropriate TeX sequence as far
as possible. The other non-graphical characters, like newline and tab,
and all other characters which are not part of ASCII, are merely
changed to exactly one space, with no special attempt to compress
consecutive spaces. Let me know how to improve this special character
processing for TeX.
This version of ptx contains a few features which do not exist in System V ptx. These extra features are suppressed by using the -G command line option, unless overridden by other command line options. Some gnu extensions cannot be recovered by overriding, so the simple rule is to avoid -G if you care about gnu extensions. Here are the differences between this program and System V ptx.
Having output parameters not introduced by options is a dangerous practice which gnu avoids as far as possible. So, for using ptx portably between gnu and System V, you should always use it with a single input file, and always expect the result on standard output. You might also want to automatically configure in a -G option to ptx calls in products using ptx, if the configurator finds that the installed ptx accepts -G.
tsort performs a topological sort on the given file, or standard input if no input file is given or for a file of ‘-’. For more details and some history, see tsort background. Synopsis:
tsort [option] [file]
tsort reads its input as pairs of strings, separated by blanks, indicating a partial ordering. The output is a total ordering that corresponds to the given partial ordering.
For example
tsort <<EOF
a b c
d
e f
b c d e
EOF
will produce the output
a
b
c
d
e
f
Consider a more realistic example.
You have a large set of functions all in one file, and they may all be
declared static except one. Currently that one (say main) is the
first function defined in the file, and the ones it calls directly follow
it, followed by those they call, etc. Let's say that you are determined
to take advantage of prototypes, so you have to choose between declaring
all of those functions (which means duplicating a lot of information from
the definitions) and rearranging the functions so that as many as possible
are defined before they are used. One way to automate the latter process
is to get a list for each function of the functions it calls directly.
Many programs can generate such lists. They describe a call graph.
Consider the following list, in which a given line indicates that the
function on the left calls the one on the right directly.
main parse_options
main tail_file
main tail_forever
tail_file pretty_name
tail_file write_header
tail_file tail
tail_forever recheck
tail_forever pretty_name
tail_forever write_header
tail_forever dump_remainder
tail tail_lines
tail tail_bytes
tail_lines start_lines
tail_lines dump_remainder
tail_lines file_lines
tail_lines pipe_lines
tail_bytes xlseek
tail_bytes start_bytes
tail_bytes dump_remainder
tail_bytes pipe_bytes
file_lines dump_remainder
recheck pretty_name
then you can use tsort to produce an ordering of those functions that satisfies your requirement.
example$ tsort call-graph | tac
dump_remainder
start_lines
file_lines
pipe_lines
xlseek
start_bytes
pipe_bytes
tail_lines
tail_bytes
pretty_name
write_header
tail
recheck
parse_options
tail_file
tail_forever
main
tsort detects any cycles in the input and writes the first cycle encountered to standard error.
Note that for a given partial ordering, generally there is no unique
total ordering. In the context of the call graph above, the function
parse_options may be placed anywhere in the list as long as it
precedes main.
The only options are --help and --version. See Common options.
An exit status of zero indicates success, and a nonzero value indicates failure.
tsort exists because very early versions of the Unix linker processed an archive file exactly once, and in order. As ld read each object in the archive, it decided whether it was needed in the program based on whether it defined any symbols which were undefined at that point in the link.
This meant that dependencies within the archive had to be handled
specially. For example, scanf probably calls read. That means
that in a single pass through an archive, it was important for scanf.o
to appear before read.o, because otherwise a program which calls
scanf but not read might end up with an unexpected unresolved
reference to read.
The way to address this problem was to first generate a set of dependencies of one object file on another. This was done by a shell script called lorder. The GNU tools don't provide a version of lorder, as far as I know, but you can still find it in BSD distributions.
Then you ran tsort over the lorder output, and you used the resulting sort to define the order in which you added objects to the archive.
This whole procedure has been obsolete since about 1980, because Unix archives now contain a symbol table (traditionally built by ranlib, now generally built by ar itself), and the Unix linker uses the symbol table to effectively make multiple passes over an archive file.
Anyhow, that's where tsort came from. To solve an old problem with the way the linker handled archive files, which has since been solved in different ways.
cut writes to standard output selected parts of each line of each input file, or standard input if no files are given or for a file name of ‘-’. Synopsis:
cut option... [file]...
In the table which follows, the byte-list, character-list, and field-list are one or more numbers or ranges (two numbers separated by a dash) separated by commas. Bytes, characters, and fields are numbered starting at 1. Incomplete ranges may be given: -m means ‘1-m’; ‘n-’ means ‘n’ through end of line or last field. The list elements can be repeated, can overlap, and can be specified in any order; but the selected input is written in the same order that it is read, and is written exactly once.
The program accepts the following options. Also see Common options.
An exit status of zero indicates success, and a nonzero value indicates failure.
paste writes to standard output lines consisting of sequentially corresponding lines of each given file, separated by a TAB character. Standard input is used for a file name of ‘-’ or if no input files are given.
For example:
$ cat num2
1
2
$ cat let3
a
b
c
$ paste num2 let3
1 a
2 b
c
Synopsis:
paste [option]... [file]...
The program accepts the following options. Also see Common options.
$ paste -s num2 let3
1 2
a b c
$ paste -d '%_' num2 let3 num2
1%a_1
2%b_2
%c_
An exit status of zero indicates success, and a nonzero value indicates failure.
join writes to standard output a line for each pair of input lines that have identical join fields. Synopsis:
join [option]... file1 file2
Either file1 or file2 (but not both) can be ‘-’, meaning standard input. file1 and file2 should be sorted on the join fields.
Normally, the sort order is that of the
collating sequence specified by the LC_COLLATE locale. Unless
the -t option is given, the sort comparison ignores blanks at
the start of the join field, as in sort -b. If the
--ignore-case option is given, the sort comparison ignores
the case of characters in the join field, as in sort -f.
The sort and join commands should use consistent locales and options if the output of sort is fed to join. You can use a command like ‘sort -k 1b,1’ to sort a file on its default join field, but if you select a non-default locale, join field, separator, or comparison options, then you should do so consistently between join and sort.
If the input has no unpairable lines, a GNU extension is available; the sort order can be any order that considers two fields to be equal if and only if the sort comparison described above considers them to be equal. For example:
$ cat file1
a a1
c c1
b b1
$ cat file2
a a2
c c2
b b2
$ join file1 file2
a a1 a2
c c1 c2
b b1 b2
If the --check-order option is given, unsorted inputs will cause a fatal error message. If the option --nocheck-order is given, unsorted inputs will never cause an error message. If neither of these options is given, wrongly sorted inputs are diagnosed only if an input file is found to contain unpairable lines. If an input file is diagnosed as being unsorted, the join command will exit with a nonzero status (and the output should not be used).
Forcing join to process wrongly sorted input files containing unpairable lines by specifying --nocheck-order is not guaranteed to produce any particular output. The output will probably not correspond with whatever you hoped it would be.
The defaults are:
The program accepts the following options. Also see Common options.
A field specification of ‘0’ denotes the join field. In most cases, the functionality of the ‘0’ field spec may be reproduced using the explicit m.n that corresponds to the join field. However, when printing unpairable lines (using either of the -a or -v options), there is no way to specify the join field using m.n in field-list if there are unpairable lines in both files. To give join that functionality, POSIX invented the ‘0’ field specification notation.
The elements in field-list are separated by commas or blanks. Blank separators typically need to be quoted for the shell. For example, the commands ‘join -o 1.2,2.2’ and ‘join -o '1.2 2.2'’ are equivalent.
All output lines—including those printed because of any -a or -v
option—are subject to the specified field-list.
An exit status of zero indicates success, and a nonzero value indicates failure.
This commands operate on individual characters.
tr [option]... set1 [set2]
tr copies standard input to standard output, performing one of the following operations:
The set1 and (if given) set2 arguments define ordered sets of characters, referred to below as set1 and set2. These sets are the characters of the input that tr operates on. The --complement (-c, -C) option replaces set1 with its complement (all of the characters that are not in set1).
Currently tr fully supports only single-byte characters. Eventually it will support multibyte characters; when it does, the -C option will cause it to complement the set of characters, whereas -c will cause it to complement the set of values. This distinction will matter only when some values are not characters, and this is possible only in locales using multibyte encodings when the input contains encoding errors.
The program accepts the --help and --version options. See Common options. Options must precede operands.
An exit status of zero indicates success, and a nonzero value indicates failure.
The format of the set1 and set2 arguments resembles the format of regular expressions; however, they are not regular expressions, only lists of characters. Most characters simply represent themselves in these strings, but the strings can contain the shorthands listed below, for convenience. Some of them can be used only in set1 or set2, as noted below.
While a backslash followed by a character not listed above is
interpreted as that character, the backslash also effectively
removes any special significance, so it is useful to escape
‘[’, ‘]’, ‘*’, and ‘-’.
gnu tr does not support the System V syntax that uses square brackets to enclose ranges. Translations specified in that format sometimes work as expected, since the brackets are often transliterated to themselves. However, they should be avoided because they sometimes behave unexpectedly. For example, ‘tr -d '[0-9]'’ deletes brackets as well as digits.
Many historically common and even accepted uses of ranges are not
portable. For example, on EBCDIC hosts using the ‘A-Z’
range will not do what most would expect because ‘A’ through ‘Z’
are not contiguous as they are in ASCII.
If you can rely on a POSIX compliant version of tr, then
the best way to work around this is to use character classes (see below).
Otherwise, it is most portable (and most ugly) to enumerate the members
of the ranges.
upper and lower classes,
which expand in ascending order. When the --delete (-d)
and --squeeze-repeats (-s) options are both given, any
character class can be used in set2. Otherwise, only the
character classes lower and upper are accepted in
set2, and then only if the corresponding character class
(upper and lower, respectively) is specified in the same
relative position in set1. Doing this specifies case conversion.
The class names are given below; an error results when an invalid class
name is given.
alnumalphablankcntrldigitgraphlowerprintpunctspaceupperxdigittr performs translation when set1 and set2 are both given and the --delete (-d) option is not given. tr translates each character of its input that is in set1 to the corresponding character in set2. Characters not in set1 are passed through unchanged. When a character appears more than once in set1 and the corresponding characters in set2 are not all the same, only the final one is used. For example, these two commands are equivalent:
tr aaa xyz
tr a z
A common use of tr is to convert lowercase characters to uppercase. This can be done in many ways. Here are three of them:
tr abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ
tr a-z A-Z
tr '[:lower:]' '[:upper:]'
But note that using ranges like a-z above is not portable.
When tr is performing translation, set1 and set2 typically have the same length. If set1 is shorter than set2, the extra characters at the end of set2 are ignored.
On the other hand, making set1 longer than set2 is not portable; POSIX says that the result is undefined. In this situation, BSD tr pads set2 to the length of set1 by repeating the last character of set2 as many times as necessary. System V tr truncates set1 to the length of set2.
By default, gnu tr handles this case like BSD tr. When the --truncate-set1 (-t) option is given, gnu tr handles this case like the System V tr instead. This option is ignored for operations other than translation.
Acting like System V tr in this case breaks the relatively common BSD idiom:
tr -cs A-Za-z0-9 '\012'
because it converts only zero bytes (the first element in the complement of set1), rather than all non-alphanumerics, to newlines.
By the way, the above idiom is not portable because it uses ranges, and it assumes that the octal code for newline is 012. Assuming a POSIX compliant tr, here is a better way to write it:
tr -cs '[:alnum:]' '[\n*]'
When given just the --delete (-d) option, tr removes any input characters that are in set1.
When given just the --squeeze-repeats (-s) option, tr replaces each input sequence of a repeated character that is in set1 with a single occurrence of that character.
When given both --delete and --squeeze-repeats, tr first performs any deletions using set1, then squeezes repeats from any remaining characters using set2.
The --squeeze-repeats option may also be used when translating, in which case tr first performs translation, then squeezes repeats from any remaining characters using set2.
Here are some examples to illustrate various combinations of options:
tr -d '\0'
tr -cs '[:alnum:]' '[\n*]'
tr -s '\n'
#!/bin/sh
cat -- "$@" \
| tr -s '[:punct:][:blank:]' '[\n*]' \
| tr '[:upper:]' '[:lower:]' \
| uniq -d
tr -d axM
However, when ‘-’ is one of those characters, it can be tricky because
‘-’ has special meanings. Performing the same task as above but also
removing all ‘-’ characters, we might try tr -d -axM, but
that would fail because tr would try to interpret -a as
a command-line option. Alternatively, we could try putting the hyphen
inside the string, tr -d a-xM, but that wouldn't work either because
it would make tr interpret a-x as the range of characters
‘a’...‘x’ rather than the three.
One way to solve the problem is to put the hyphen at the end of the list
of characters:
tr -d axM-
Or you can use ‘--’ to terminate option processing:
tr -d -- -axM
More generally, use the character class notation [=c=]
with ‘-’ (or any other character) in place of the ‘c’:
tr -d '[=-=]axM'
Note how single quotes are used in the above example to protect the square brackets from interpretation by a shell.
expand writes the contents of each given file, or standard input if none are given or for a file of ‘-’, to standard output, with tab characters converted to the appropriate number of spaces. Synopsis:
expand [option]... [file]...
By default, expand converts all tabs to spaces. It preserves backspace characters in the output; they decrement the column count for tab calculations. The default action is equivalent to -t 8 (set tabs every 8 columns).
The program accepts the following options. Also see Common options.
For compatibility, GNU expand also accepts the obsolete
option syntax, -t1[,t2].... New scripts
should use -t t1[,t2]... instead.
An exit status of zero indicates success, and a nonzero value indicates failure.
unexpand writes the contents of each given file, or standard input if none are given or for a file of ‘-’, to standard output, converting blanks at the beginning of each line into as many tab characters as needed. In the default POSIX locale, a blank is a space or a tab; other locales may specify additional blank characters. Synopsis:
unexpand [option]... [file]...
By default, unexpand converts only initial blanks (those that precede all non-blank characters) on each line. It preserves backspace characters in the output; they decrement the column count for tab calculations. By default, tabs are set at every 8th column.
The program accepts the following options. Also see Common options.
For compatibility, GNU unexpand supports the obsolete option syntax,
-tab1[,tab2]..., where tab stops must be
separated by commas. (Unlike -t, this obsolete option does
not imply -a.) New scripts should use --first-only -t
tab1[,tab2]... instead.
An exit status of zero indicates success, and a nonzero value indicates failure.
This chapter describes the ls command and its variants dir and vdir, which list information about files.
The ls program lists information about files (of any type, including directories). Options and file arguments can be intermixed arbitrarily, as usual.
For non-option command-line arguments that are directories, by default ls lists the contents of directories, not recursively, and omitting files with names beginning with ‘.’. For other non-option arguments, by default ls lists just the file name. If no non-option argument is specified, ls operates on the current directory, acting as if it had been invoked with a single argument of ‘.’.
By default, the output is sorted alphabetically, according to the locale settings in effect.3 If standard output is a terminal, the output is in columns (sorted vertically) and control characters are output as question marks; otherwise, the output is listed one per line and control characters are output as-is.
Because ls is such a fundamental program, it has accumulated many options over the years. They are described in the subsections below; within each section, options are listed alphabetically (ignoring case). The division of options into the subsections is not absolute, since some options affect more than one aspect of ls's operation.
0 success
1 minor problems (e.g., failure to access a file or directory not
specified as a command line argument. This happens when listing a
directory in which entries are actively being removed or renamed.)
2 serious trouble (e.g., memory exhausted, invalid option or failure
to access file or directory specified as a command line argument)
Also see Common options.
These options determine which files ls lists information for. By default, ls lists files and the contents of any directories on the command line, except that in directories it ignores files whose names start with ‘.’.
This option can be useful in shell aliases. For example, if
lx is an alias for ‘ls --hide='*~'’ and ly is
an alias for ‘ls --ignore='*~'’, then the command ‘lx -A’
lists the file README~ even though ‘ly -A’ would not.
$ ls --ignore='.??*' --ignore='.[^.]' --ignore='#*'
The first option ignores names of length 3 or more that start with ‘.’,
the second ignores all two-character names that start with ‘.’
except ‘..’, and the third ignores names that start with ‘#’.
These options affect the information that ls displays. By default, only file names are shown.
//DIRED// beg1 end1 beg2 end2 ...
The begn and endn are unsigned integers that record the byte position of the beginning and end of each file name in the output. This makes it easy for Emacs to find the names, even when they contain unusual characters such as space or newline, without fancy searching.
If directories are being listed recursively (-R), output a similar line with offsets for each subdirectory name:
//SUBDIRED// beg1 end1 ...
Finally, output a line of the form:
//DIRED-OPTIONS// --quoting-style=word
where word is the quoting style (see Formatting the file names).
Here is an actual example:
$ mkdir -p a/sub/deeper a/sub2
$ touch a/f1 a/f2
$ touch a/sub/deeper/file
$ ls -gloRF --dired a
a:
total 8
-rw-r--r-- 1 0 Jun 10 12:27 f1
-rw-r--r-- 1 0 Jun 10 12:27 f2
drwxr-xr-x 3 4096 Jun 10 12:27 sub/
drwxr-xr-x 2 4096 Jun 10 12:27 sub2/
a/sub:
total 4
drwxr-xr-x 2 4096 Jun 10 12:27 deeper/
a/sub/deeper:
total 0
-rw-r--r-- 1 0 Jun 10 12:27 file
a/sub2:
total 0
//DIRED// 48 50 84 86 120 123 158 162 217 223 282 286
//SUBDIRED// 2 3 167 172 228 240 290 296
//DIRED-OPTIONS// --quoting-style=literal
Note that the pairs of offsets on the ‘//DIRED//’ line above delimit these names: f1, f2, sub, sub2, deeper, file. The offsets on the ‘//SUBDIRED//’ line delimit the following directory names: a, a/sub, a/sub/deeper, a/sub2.
Here is an example of how to extract the fifth entry name, ‘deeper’, corresponding to the pair of offsets, 222 and 228:
$ ls -gloRF --dired a > out
$ dd bs=1 skip=222 count=6 < out 2>/dev/null; echo
deeper
Note that although the listing above includes a trailing slash for the ‘deeper’ entry, the offsets select the name without the trailing slash. However, if you invoke ls with --dired along with an option like --escape (aka -b) and operate on a file whose name contains special characters, notice that the backslash is included:
$ touch 'a b'
$ ls -blog --dired 'a b'
-rw-r--r-- 1 0 Jun 10 12:28 a\ b
//DIRED// 30 34
//DIRED-OPTIONS// --quoting-style=escape
If you use a quoting style that adds quote marks
(e.g., --quoting-style=c), then the offsets include the quote marks.
So beware that the user may select the quoting style via the environment
variable QUOTING_STYLE. Hence, applications using --dired
should either specify an explicit --quoting-style=literal option
(aka -N or --literal) on the command line, or else be
prepared to parse the escaped names.
Normally the size is printed as a byte count without punctuation, but this can be overridden (see Block size). For example, -h prints an abbreviated, human-readable count, and ‘--block-size="'1"’ prints a byte count with the thousands separator of the current locale.
For each directory that is listed, preface the files with a line ‘total blocks’, where blocks is the total disk allocation for all files in that directory. The block size currently defaults to 1024 bytes, but this can be overridden (see Block size). The blocks computed counts each hard link separately; this is arguably a deficiency.
The file type is one of the following characters:
The file mode bits listed are similar to symbolic mode specifications (see Symbolic Modes). But ls combines multiple bits into the third character of each set of permissions as follows:
Following the file mode bits is a single character that specifies whether an alternate access method such as an access control list applies to the file. When the character following the file mode bits is a space, there is no alternate access method. When it is a printing character, then there is such a method.
GNU ls uses a ‘.’ character to indicate a file with an SELinux security context, but no other alternate access method.
A file with any other combination of alternate access methods
is marked with a ‘+’ character.
Normally the disk allocation is printed in units of 1024 bytes, but this can be overridden (see Block size).
For files that are NFS-mounted from an HP-UX system to a BSD system, this option reports sizes that are half the correct values. On HP-UX systems, it reports sizes that are twice the correct values for files that are NFS-mounted from BSD systems. This is due to a flaw in HP-UX; it also affects the HP-UX ls program.
These options change the order in which ls sorts the information it outputs. By default, sorting is done by character code (e.g., ASCII order).
The version sort takes into account the fact that file names frequently include indices or version numbers. Standard sorting functions usually do not produce the ordering that people expect because comparisons are made on a character-by-character basis. The version sort addresses this problem, and is especially useful when browsing directories that contain many files with indices/version numbers in their names:
$ ls -1 $ ls -1v
foo.zml-1.gz foo.zml-1.gz
foo.zml-100.gz foo.zml-2.gz
foo.zml-12.gz foo.zml-6.gz
foo.zml-13.gz foo.zml-12.gz
foo.zml-2.gz foo.zml-13.gz
foo.zml-25.gz foo.zml-25.gz
foo.zml-6.gz foo.zml-100.gz
Version-sorted strings are compared such that if ver1 and ver2 are version numbers and prefix and suffix (suffix matching the regular expression ‘(\.[A-Za-z~][A-Za-z0-9~]*)*’) are strings then ver1 < ver2 implies that the name composed of “prefix ver1 suffix” sorts before “prefix ver2 suffix”.
Note also that leading zeros of numeric parts are ignored:
$ ls -1 $ ls -1v
abc-1.007.tgz abc-1.01a.tgz
abc-1.012b.tgz abc-1.007.tgz
abc-1.01a.tgz abc-1.012b.tgz
This functionality is implemented using gnulib's filevercmp function.
One result of that implementation decision is that ‘ls -v’
and ‘sort -V’ do not use the locale category, LC_COLLATE,
which means non-numeric prefixes are sorted as if LC_COLLATE were set
to ‘C’.
These options affect the appearance of the overall output.
more -f does seem to work.
Some terminal emulators (at least Apple Terminal 1.5 (133) from Mac OS X 10.4.8)
do not properly align columns to the right of a TAB following a
non-ASCII byte. If you use such a terminal emulator, use the
-T0 option or put TABSIZE=0 in your environment to tell
ls to align using spaces, not tabs.
By default, file timestamps are listed in abbreviated form. Most locales use a timestamp like ‘2002-03-30 23:45’. However, the default POSIX locale uses a date like ‘Mar 30 2002’ for non-recent timestamps, and a date-without-year and time like ‘Mar 30 23:45’ for recent timestamps.
A timestamp is considered to be recent if it is less than six months old, and is not dated in the future. If a timestamp dated today is not listed in recent form, the timestamp is in the future, which means you probably have clock skew problems which may break programs like make that rely on file timestamps.
Time stamps are listed according to the time zone rules specified by the TZ environment variable, or by the system default rules if TZ is not set. See Specifying the Time Zone with TZ.
The following option changes how file timestamps are printed.
If format contains two format strings separated by a newline,
the former is used for non-recent files and the latter for recent
files; if you want output columns to line up, you may need to insert
spaces in one of the two formats.
This is useful because the time output includes all the information that
is available from the operating system. For example, this can help
explain make's behavior, since GNU make
uses the full timestamp to determine whether a file is out of date.
newline='
'
ls -l --time-style="+%Y-%m-%d $newline%m-%d %H:%M"
ls -l --time-style="iso"
The LC_TIME locale category specifies the timestamp format. The default POSIX locale uses timestamps like ‘Mar 30 2002’ and ‘Mar 30 23:45’; in this locale, the following two ls invocations are equivalent:
newline='
'
ls -l --time-style="+%b %e %Y$newline%b %e %H:%M"
ls -l --time-style="locale"
Other locales behave differently. For example, in a German locale,
--time-style="locale" might be equivalent to
--time-style="+%e. %b %Y $newline%e. %b %H:%M"
and might generate timestamps like ‘30. Mär 2002 ’ and
‘30. Mär 23:45’.
You can specify the default value of the --time-style option with the environment variable TIME_STYLE; if TIME_STYLE is not set the default style is ‘locale’. GNU Emacs 21.3 and later use the --dired option and therefore can parse any date format, but if you are using Emacs 21.1 or 21.2 and specify a non-POSIX locale you may need to set ‘TIME_STYLE="posix-long-iso"’.
To avoid certain denial-of-service attacks, timestamps that would be longer than 1000 bytes may be treated as errors.
These options change how file names themselves are printed.
You can specify the default value of the --quoting-style option
with the environment variable QUOTING_STYLE. If that environment
variable is not set, the default value is ‘literal’, but this
default may change to ‘shell’ in a future version of this package.
dir is equivalent to ls -C
-b; that is, by default files are listed in columns, sorted vertically,
and special characters are represented by backslash escape sequences.
See ls.
vdir is equivalent to ls -l
-b; that is, by default files are listed in long format and special
characters are represented by backslash escape sequences.
dircolors outputs a sequence of shell commands to set up the terminal for color output from ls (and dir, etc.). Typical usage:
eval "`dircolors [option]... [file]`"
If file is specified, dircolors reads it to determine which colors to use for which file types and extensions. Otherwise, a precompiled database is used. For details on the format of these files, run ‘dircolors --print-database’.
To make dircolors read a ~/.dircolors file if it exists, you can put the following lines in your ~/.bashrc (or adapt them to your favorite shell):
d=.dircolors
test -r $d && eval "$(dircolors $d)"
The output is a shell command to set the LS_COLORS environment variable. You can specify the shell syntax to use on the command line, or dircolors will guess it from the value of the SHELL environment variable.
The program accepts the following options. Also see Common options.
SHELL ends with
csh or tcsh.
An exit status of zero indicates success, and a nonzero value indicates failure.
This chapter describes the commands for basic file manipulation: copying, moving (renaming), and deleting (removing).
cp copies files (or, optionally, directories). The copy is completely independent of the original. You can either copy one file to another, or copy arbitrarily many files to a destination directory. Synopses:
cp [option]... [-T] source dest
cp [option]... source... directory
cp [option]... -t directory source...
Generally, files are written just as they are read. For exceptions, see the --sparse option below.
By default, cp does not copy directories. However, the -R, -a, and -r options cause cp to copy recursively by descending into source directories and copying files to corresponding destination directories.
When copying from a symbolic link, cp normally follows the link only when not copying recursively. This default can be overridden with the --archive (-a), -d, --dereference (-L), --no-dereference (-P), and -H options. If more than one of these options is specified, the last one silently overrides the others.
When copying to a symbolic link, cp follows the link only when it refers to an existing regular file. However, when copying to a dangling symbolic link, cp refuses by default, and fails with a diagnostic, since the operation is inherently dangerous. This behavior is contrary to historical practice and to POSIX. Set POSIXLY_CORRECT to make cp attempt to create the target of a dangling destination symlink, in spite of the possible risk. Also, when an option like --backup or --link acts to rename or remove the destination before copying, cp renames or removes the symbolic link rather than the file it points to.
By default, cp copies the contents of special files only when not copying recursively. This default can be overridden with the --copy-contents option.
cp generally refuses to copy a file onto itself, with the following exception: if --force --backup is specified with source and dest identical, and referring to a regular file, cp will make a backup file, either regular or numbered, as specified in the usual ways (see Backup options). This is useful when you simply want to make a backup of an existing file before changing it.
The program accepts the following options. Also see Common options.
#!/bin/sh
# Usage: backup FILE...
# Create a gnu-style backup of each listed FILE.
for i; do
cp --backup --force -- "$i" "$i"
done
cp -R --copy-contents will hang indefinitely trying to read
from FIFOs and special files like /dev/console, and it will
fill up your destination disk if you use it to copy /dev/zero.
This option has no effect unless copying recursively, and it does not
affect the copying of symbolic links.
This option is independent of the --interactive or -i option: neither cancels the effect of the other.
This option is redundant if the --no-clobber or -n option is
used.
lutimes function, which makes
it possible even for symbolic links. However, this implementation does
not yet take advantage of that.
Using --preserve with no attribute_list is equivalent to --preserve=mode,ownership,timestamps.
In the absence of this option, each destination file is created with the mode bits of the corresponding source file, minus the bits set in the umask and minus the set-user-ID and set-group-ID bits. See File permissions.
cp --parents a/b/c existing_dir
copies the file a/b/c to existing_dir/a/b/c, creating
any missing intermediate directories.
The when value can be one of the following:
An exit status of zero indicates success, and a nonzero value indicates failure.
dd copies a file (from standard input to standard output, by default) with a changeable I/O block size, while optionally performing conversions on it. Synopses:
dd [operand]...
dd option
The only options are --help and --version. See Common options. dd accepts the following operands.
Conversions:
The ‘ascii’, ‘ebcdic’, and ‘ibm’ conversions are
mutually exclusive.
The ‘block’ and ‘unblock’ conversions are mutually exclusive.
The ‘lcase’ and ‘ucase’ conversions are mutually exclusive.
The ‘excl’ and ‘nocreat’ conversions are mutually exclusive.
Here are the flags. Not every flag is supported on every operating system.
read system call
may return early if a full block is not available.
When that happens, continue calling read to fill the remainder
of the block.
This flag can be used only with iflag.
These flags are not supported on all systems, and ‘dd’ rejects attempts to use them when they are not supported. When reading from standard input or writing to standard output, the ‘nofollow’ and ‘noctty’ flags should not be specified, and the other flags (e.g., ‘nonblock’) can affect how other processes behave with the affected file descriptors, even after dd exits.
The numeric-valued strings above (bytes and blocks) can be followed by a multiplier: ‘b’=512, ‘c’=1, ‘w’=2, ‘xm’=m, or any of the standard block size suffixes like ‘k’=1024 (see Block size).
Use different dd invocations to use different block sizes for skipping and I/O. For example, the following shell commands copy data in 512 KiB blocks between a disk and a tape, but do not save or restore a 4 KiB label at the start of the disk:
disk=/dev/rdsk/c0t1d0s2
tape=/dev/rmt/0
# Copy all but the label from disk to tape.
(dd bs=4k skip=1 count=0 && dd bs=512k) <$disk >$tape
# Copy from tape back to disk, but leave the disk label alone.
(dd bs=4k seek=1 count=0 && dd bs=512k) <$tape >$disk
Sending an ‘INFO’ signal to a running dd
process makes it print I/O statistics to standard error
and then resume copying. In the example below,
dd is run in the background to copy 10 million blocks.
The kill command makes it output intermediate I/O statistics,
and when dd completes normally or is killed by the
SIGINT signal, it outputs the final statistics.
$ dd if=/dev/zero of=/dev/null count=10MB & pid=$!
$ kill -s INFO $pid; wait $pid
3385223+0 records in
3385223+0 records out
1733234176 bytes (1.7 GB) copied, 6.42173 seconds, 270 MB/s
10000000+0 records in
10000000+0 records out
5120000000 bytes (5.1 GB) copied, 18.913 seconds, 271 MB/s
On systems lacking the ‘INFO’ signal dd responds to the ‘USR1’ signal instead, unless the POSIXLY_CORRECT environment variable is set.
An exit status of zero indicates success, and a nonzero value indicates failure.
install copies files while setting their file mode bits and, if possible, their owner and group. Synopses:
install [option]... [-T] source dest
install [option]... source... directory
install [option]... -t directory source...
install [option]... -d directory...
install is similar to cp, but allows you to control the attributes of destination files. It is typically used in Makefiles to copy programs into their destination directories. It refuses to copy files onto themselves.
install never preserves extended attributes (xattr).
The program accepts the following options. Also see Common options.
root. owner may be either a user name or a numeric user
ID.
An exit status of zero indicates success, and a nonzero value indicates failure.
mv moves or renames files (or directories). Synopses:
mv [option]... [-T] source dest
mv [option]... source... directory
mv [option]... -t directory source...
mv can move any type of file from one file system to another.
Prior to version 4.0 of the fileutils,
mv could move only regular files between file systems.
For example, now mv can move an entire directory hierarchy
including special device files from one partition to another. It first
uses some of the same code that's used by cp -a to copy the
requested directories and files, then (assuming the copy succeeded)
it removes the originals. If the copy fails, then the part that was
copied to the destination partition is removed. If you were to copy
three directories from one partition to another and the copy of the first
directory succeeded, but the second didn't, the first would be left on
the destination partition and the second and third would be left on the
original partition.
mv always tries to copy extended attributes (xattr).
If a destination file exists but is normally unwritable, standard input is a terminal, and the -f or --force option is not given, mv prompts the user for whether to replace the file. (You might own the file, or have write permission on its directory.) If the response is not affirmative, the file is skipped.
Warning: Avoid specifying a source name with a trailing slash,
when it might be a symlink to a directory.
Otherwise, mv may do something very surprising, since
its behavior depends on the underlying rename system call.
On a system with a modern Linux-based kernel, it fails with errno=ENOTDIR.
However, on other systems (at least FreeBSD 6.1 and Solaris 10) it silently
renames not the symlink but rather the directory referenced by the symlink.
See Trailing slashes.
The program accepts the following options. Also see Common options.
An exit status of zero indicates success, and a nonzero value indicates failure.
rm removes each given file. By default, it does not remove directories. Synopsis:
rm [option]... [file]...
If the -I or --interactive=once option is given, and there are more than three files or the -r, -R, or --recursive are given, then rm prompts the user for whether to proceed with the entire operation. If the response is not affirmative, the entire command is aborted.
Otherwise, if a file is unwritable, standard input is a terminal, and the -f or --force option is not given, or the -i or --interactive=always option is given, rm prompts the user for whether to remove the file. If the response is not affirmative, the file is skipped.
Any attempt to remove a file whose last file name component is . or .. is rejected without any prompting.
Warning: If you use rm to remove a file, it is usually possible to recover the contents of that file. If you want more assurance that the contents are truly unrecoverable, consider using shred.
The program accepts the following options. Also see Common options.
This option is useful when removing a build “chroot” hierarchy, which normally contains no valuable data. However, it is not uncommon to bind-mount /home into such a hierarchy, to make it easier to use one's start-up file. The catch is that it's easy to forget to unmount /home. Then, when you use rm -rf to remove your normally throw-away chroot, that command will remove everything under /home, too. Use the --one-file-system option, and it will warn about and skip directories on other file systems. Of course, this will not save your /home if it and your chroot happen to be on the same file system.
One common question is how to remove files whose names begin with a
‘-’. gnu rm, like every program that uses the getopt
function to parse its arguments, lets you use the ‘--’ option to
indicate that all following arguments are non-options. To remove a file
called -f in the current directory, you could type either:
rm -- -f
or:
rm ./-f
The Unix rm program's use of a single ‘-’ for this purpose predates the development of the getopt standard syntax.
An exit status of zero indicates success, and a nonzero value indicates failure.
shred overwrites devices or files, to help prevent even very expensive hardware from recovering the data.
Ordinarily when you remove a file (see rm invocation), the data is not actually destroyed. Only the index listing where the file is stored is destroyed, and the storage is made available for reuse. There are undelete utilities that will attempt to reconstruct the index and can bring the file back if the parts were not reused.
On a busy system with a nearly-full drive, space can get reused in a few seconds. But there is no way to know for sure. If you have sensitive data, you may want to be sure that recovery is not possible by actually overwriting the file with non-sensitive data.
However, even after doing that, it is possible to take the disk back to a laboratory and use a lot of sensitive (and expensive) equipment to look for the faint “echoes” of the original data underneath the overwritten data. If the data has only been overwritten once, it's not even that hard.
The best way to remove something irretrievably is to destroy the media it's on with acid, melt it down, or the like. For cheap removable media like floppy disks, this is the preferred method. However, hard drives are expensive and hard to melt, so the shred utility tries to achieve a similar effect non-destructively.
This uses many overwrite passes, with the data patterns chosen to maximize the damage they do to the old data. While this will work on floppies, the patterns are designed for best effect on hard drives. For more details, see the source code and Peter Gutmann's paper Secure Deletion of Data from Magnetic and Solid-State Memory, from the proceedings of the Sixth USENIX Security Symposium (San Jose, California, July 22–25, 1996).
Please note that shred relies on a very important assumption: that the file system overwrites data in place. This is the traditional way to do things, but many modern file system designs do not satisfy this assumption. Exceptions include:
data=journal mode),
BFS, NTFS, etc. when they are configured to journal data.
In the particular case of ext3 file systems, the above disclaimer applies (and
shred is thus of limited effectiveness) only in data=journal
mode, which journals file data in addition to just metadata. In both
the data=ordered (default) and data=writeback modes,
shred works as usual. Ext3 journaling modes can be changed
by adding the data=something option to the mount options for a
particular file system in the /etc/fstab file, as documented in
the mount man page (man mount).
If you are not sure how your file system operates, then you should assume that it does not overwrite data in place, which means that shred cannot reliably operate on regular files in your file system.
Generally speaking, it is more reliable to shred a device than a file, since this bypasses the problem of file system design mentioned above. However, even shredding devices is not always completely reliable. For example, most disks map out bad sectors invisibly to the application; if the bad sectors contain sensitive data, shred won't be able to destroy it.
shred makes no attempt to detect or report this problem, just as it makes no attempt to do anything about backups. However, since it is more reliable to shred devices than files, shred by default does not truncate or remove the output file. This default is more suitable for devices, which typically cannot be truncated and should not be removed.
Finally, consider the risk of backups and mirrors. File system backups and remote mirrors may contain copies of the file that cannot be removed, and that will allow a shredded file to be recovered later. So if you keep any data you may later want to destroy using shred, be sure that it is not backed up or mirrored.
shred [option]... file[...]
The program accepts the following options. Also see Common options.
You might use the following command to erase all trace of the file system you'd created on the floppy disk in your first drive. That command takes about 20 minutes to erase a “1.44MB” (actually 1440 KiB) floppy.
shred --verbose /dev/fd0
Similarly, to erase all data on a selected partition of your hard disk, you could give a command like this:
shred --verbose /dev/sda5
A file of ‘-’ denotes standard output. The intended use of this is to shred a removed temporary file. For example:
i=`tempfile -m 0600`
exec 3<>"$i"
rm -- "$i"
echo "Hello, world" >&3
shred - >&3
exec 3>-
However, the command ‘shred - >file’ does not shred the contents of file, since the shell truncates file before invoking shred. Use the command ‘shred file’ or (if using a Bourne-compatible shell) the command ‘shred - 1<>file’ instead.
An exit status of zero indicates success, and a nonzero value indicates failure.
This chapter describes commands which create special types of files (and rmdir, which removes directories, one special file type).
Although Unix-like operating systems have markedly fewer special file types than others, not everything can be treated only as the undifferentiated byte stream of normal files. For example, when a file is created or removed, the system must record this information, which it does in a directory—a special type of file. Although you can read directories as normal files, if you're curious, in order for the system to do its job it must impose a structure, a certain order, on the bytes of the file. Thus it is a “special” type of file.
Besides directories, other special file types include named pipes (FIFOs), symbolic links, sockets, and so-called special files.
link creates a single hard link at a time.
It is a minimalist interface to the system-provided
link function. See Hard Links.
It avoids the bells and whistles of the more commonly-used
ln command (see ln invocation).
Synopsis:
link filename linkname
filename must specify an existing file, and linkname
must specify a nonexistent entry in an existing directory.
link simply calls link (filename, linkname)
to create the link.
On a GNU system, this command acts like ‘ln --directory --no-target-directory filename linkname’. However, the --directory and --no-target-directory options are not specified by POSIX, and the link command is more portable in practice.
An exit status of zero indicates success, and a nonzero value indicates failure.
ln makes links between files. By default, it makes hard links; with the -s option, it makes symbolic (or soft) links. Synopses:
ln [option]... [-T] target linkname
ln [option]... target
ln [option]... target... directory
ln [option]... -t directory target...
Normally ln does not remove existing files. Use the --force (-f) option to remove them unconditionally, the --interactive (-i) option to remove them conditionally, and the --backup (-b) option to rename them.
A hard link is another name for an existing file; the link and the original are indistinguishable. Technically speaking, they share the same inode, and the inode contains all the information about a file—indeed, it is not incorrect to say that the inode is the file. On all existing implementations, you cannot make a hard link to a directory, and hard links cannot cross file system boundaries. (These restrictions are not mandated by POSIX, however.)
Symbolic links (symlinks for short), on the other hand, are a special file type (which not all kernels support: System V release 3 (and older) systems lack symlinks) in which the link file actually refers to a different file, by name. When most operations (opening, reading, writing, and so on) are passed the symbolic link file, the kernel automatically dereferences the link and operates on the target of the link. But some operations (e.g., removing) work on the link file itself, rather than on its target. The owner, group, and mode of a symlink are not significant to file access performed through the link. See Symbolic Links.
Symbolic links can contain arbitrary strings; a dangling symlink occurs when the string in the symlink does not resolve to a file. There are no restrictions against creating dangling symbolic links. There are trade-offs to using absolute or relative symlinks. An absolute symlink always points to the same file, even if the directory containing the link is moved. However, if the symlink is visible from more than one machine (such as on a networked file system), the file pointed to might not always be the same. A relative symbolic link is resolved in relation to the directory that contains the link, and is often useful in referring to files on the same device without regards to what name that device is mounted on when accessed via networked machines.
When creating a relative symlink in a different location than the current directory, the resolution of the symlink will be different than the resolution of the same string from the current directory. Therefore, many users prefer to first change directories to the location where the relative symlink will be created, so that tab-completion or other file resolution will find the same target as what will be placed in the symlink.
The program accepts the following options. Also see Common options.
When the destination is an actual directory (not a symlink to one), there is no ambiguity. The link is created in that directory. But when the specified destination is a symlink to a directory, there are two ways to treat the user's request. ln can treat the destination just as it would a normal directory and create the link in it. On the other hand, the destination can be viewed as a non-directory—as the symlink itself. In that case, ln must delete or backup that symlink before creating the new link. The default is to treat a destination that is a symlink to a directory just like a directory.
This option is weaker than the --no-target-directory
(-T) option, so it has no effect if both options are given.