GNU Core Utilities Frequently Asked Questions

Next:  

1 GNU Core Utilities Frequently Asked Questions

Welcome to the GNU Core Utilities FAQ. This document answers the most frequently asked questions about the core utilities of the GNU Operating System.

The master location of this document is available online at http://www.gnu.org/software/coreutils/faq/.

If you have a question that is not answered in this FAQ then please check the mailing list archives. If you find a useful question and answer please send a message to the bug list and I will add it to the FAQ so that this document can be improved. If you still don’t find a suitable answer, consider posting the question to the bug list.

An excellent collection of FAQs is available by anonymous FTP at rtfm.mit.edu and in particular the Unix FAQ is pertinent here. ftp://rtfm.mit.edu/pub/usenet/news.answers/unix-faq/faq/contents

This FAQ was written by Bob Proulx <bob@proulx.com> as an amalgamation of the many questions asked and answered on the bug lists.

Copyright © 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Free Software Foundation

This document is free documentation; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This document and any included programs in the document are distributed in the hope that they will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

If you have not received a copy of the GNU General Public License see http://www.gnu.org/licenses/.


Next: , Previous: , Up: Top  

2 Fileutils, shellutils and textutils

Previously a set of three packages combined implemented the core set of GNU utilities. These were the GNU fileutils, shellutils and textutils. (Additionally shellutils with the version number attached as a suffix was one letter too long for 14 character limited filesystems and was also known as sh-utils.) Each had its own web page. Each had their own mailing list. But the three were generally considered a set.

In 2003 these three packages of fileutils, shellutils, and textutils were combined into the current coreutils package. This greatly simplified the maintenance and management of this project. The older packages are deprecated. All users are requested to update to the latest stable release of coreutils.


Next: , Previous: , Up: Top  

3 Where can I get more information about GNU coreutils?

The online info documentation is always the most up to date source of information. It should always be consulted first for the latest information on your particular installation. Here are example commands to invoke info to browse the documentation.

info coreutils
info coreutils ls
info coreutils sort
info coreutils head

Additionally the home page contains the canonical top level information and pointers to all things GNU coreutils.

http://www.gnu.org/software/coreutils/

The online web home page for the GNU coreutils.

Please browse the mailing list archives. It is possible, even likely, that your question has been asked before. You might find just the information you were looking for. Many questions are asked here and at least a few are answered.

http://lists.gnu.org/archive/html/bug-coreutils/

The GNU Core Utilities mailing list archives of the bug list.

http://lists.gnu.org/mailman/listinfo/bug-coreutils

The GNU Core Utilities bug mailing list subscription interface.

http://lists.gnu.org/archive/html/coreutils/

The GNU Core Utilities mailing list archives of the discussion list.

http://lists.gnu.org/mailman/listinfo/coreutils

The GNU Core Utilities discussion mailing list subscription interface.


Next: , Previous: , Up: Top  

4 Where can I get the latest version of GNU coreutils?

http://ftp.gnu.org/pub/gnu/coreutils

The GNU coreutils main stable release distribution site.

ftp://alpha.gnu.org/gnu/coreutils

The GNU coreutils alpha and beta test release distribution site.

https://savannah.gnu.org/git/?group=coreutils

The GNU coreutils developer’s site on Savannah documenting source code access.

http://git.savannah.gnu.org/gitweb/?p=coreutils.git

Browse the source code using the online web interface to the git version control system source code repository.

Development releases of GNU coreutils source code can be downloaded from ftp://alpha.gnu.org/gnu/coreutils. These are test releases and although they are generally of good quality they have not been tested well enough nor matured enough to be considered stable releases. Also by definition stable means stable and the test releases change frequently. Use with care.

The source code is maintained in the git version control system. The home page for git is http://git.or.cz/. A wiki page on Savannah documenting git use is at http://savannah.gnu.org/maintenance/UsingGit. A local working copy of the latest source code may be obtained using this command.

git clone git://git.savannah.gnu.org/coreutils

When possible it helps to report any bugs as seen against the latest development version. However the latest development version has dependencies upon the latest version of several other projects. See the README-hacking file available in the project source and here on the web at http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=README-hacking for details of how to build the latest development version from checkout sources.

To avoid needing the latest development tools the maintainers periodically make available snapshot releases. These may be built with only normal system dependencies. These are frequently short lived and available directly from the developer’s site. To find out information about these it is best to read the mailing list and look for the latest snapshot announcement. Here are search links that may be useful.

http://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=%2Bsubject%3Asnapshot&submit=Search%21&idxname=bug-coreutils
http://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=%2Bsubject%3Asnapshot&submit=Search%21&idxname=coreutils

Great effort is spent to ensure that the software builds easily on a large number of different systems. If you have not done it before then compiling the coreutils from a release or snapshot is probably easier than you think. If possible please build and test the latest test release to see if your problem is resolved there.

A note for Cygwin users. The MS-Windows platform requires special handling. For Cygwin I recommend using the latest precompiled binaries from the Cygwin site. They do an excellent job of handling the peculiarities and deficiencies of that platform.

Patches for bugs and enhancements are always appreciated. Please submit patches as unified diffs against the latest version control sources if possible. If unable to access Savannah then please use the latest test release that is available.

Be aware that submissions that are more than a few lines are legally significant and will require a copyright assignment to the FSF. In order to keep GNU free it is important for the FSF to be able to defend the GPL status in court if necessary. If you are unable to make a copyright assignment, such as for reasons of being unable to obtain a disclaimer from your employer, then your code cannot be incorporated, in which case it would be better to simply discuss your idea on the mailing list. Ideas cannot be copyrighted nor patented and someone else may be able to contribute the code to the project. Almost a worst case is when a person contributes a perfect fix or feature enhancement but it cannot be used because of the inability to obtain legal copyright assignment. In which case the contribution of a patch can actually hurt more than it helps and it would have been better simply to discuss it. The project ChangeLog is filled with the names of people making good contributions who have suggested ideas and reported bugs.


Next: , Previous: , Up: Top  

5 How do I add a question to the FAQ?

Send a plain text e-mail to the <coreutils@gnu.org> mailing list. Also including the Frequently Given Answer if you have one. Please make sure the message is formatted as a plain text message. Do not send HTML encoded messages.

Please allow some time for processing. Many people read the mail so please try to be concise. Even if you do not receive a personal response your message will have been read by the maintainers. But so many messages are posted that it can be overwhelming for us at times.


Next: , Previous: , Up: Top  

6 How do I report a bug?

Please be sure that the bug you are reporting is actually related to GNU coreutils. Many times people report problems in other unrelated programs. Those may be legitimate bugs but the GNU coreutils maintainers are in no position to do anything about other people’s software. People have reported bugs in their sound cards and crashes of their disk drives and panics of their operating system kernel. We can’t help you with any of those things.

If possible please test for the presence of your problem using the latest version of the software. The GNU utilities are widely used and many times the bug will already have been found and fixed in a later version.

Check the mailing list archives to see if someone else has already reported your problem. Keep in mind that many people will be reading the list and it can be difficult to keep up with the volume repeats. It can also be useful to search the gnu.org site using your favorite Internet search engines by specifying a site:lists.gnu.org search restriction.

http://lists.gnu.org/archive/html/bug-coreutils/

Searchable mailing list archives.

http://lists.gnu.org/archive/html/coreutils/

Searchable mailing list archives of the discussion list.

If you think you have a real bug then send mail to the bug discussion list. Use a subject that is descriptive of the issue. Think of how difficult it is to follow a thread of discussion when every subject line is simply bug.

Examples of good subjects:

mv && hardlinks problem
dd and skip on Linux tape devices and pipes
assertion failure in mv command
fails to compile on platform xyz

Examples of bad subjects:

I have a question.
help!
bug?
bug report
URGENT

In your description indicate the version of the program that you are seeing with the problem. Also note the operating system type and version. The GNU utilities are part of the GNU Operating System but have been ported to run standalone on many different types of systems and some problems will be unique to them. We are good at guessing your environment but it is much simpler if we have the information without guessing.

Please make sure the message is formatted as a plain text message. Do not send HTML encoded messages. Do not send overly large messages without first establishing contact. Once someone found a way to trigger a problem with sort by giving it a 200MB sized file. They sent that datafile to the mailing list. Needless to say that was not appreciated. People on limited bandwidth connections reading the list were severely affected.

Include as small of a test case as you can manage to create that will allow others to recreate the problem. If the problem cannot be recreated then it is very difficult to diagnose or fix.

Confused by all of this? Don’t worry. We all start somewhere. Here is a pointer to an article written by Eric S. Raymond on how to ask good questions.

http://catb.org/~esr/faqs/smart-questions.html

Patches to the source code are always appreciated. But reports of bugs are always welcome even if you do not feel comfortable working in the source code.


Next: , Previous: , Up: Top  

7 I reported a bug but did not get a response.

It may just be that the volunteers are busy. Please be patient. Every message to the bug lists is read by many people. Sometimes there is just nothing to say at the time. If sufficient time has elapsed, say one or two weeks, and you think your message may have been forgotten it is perfectly acceptable to send a second message asking about your first message.

But if you think your message might not have made it to the list or that you may have missed the response then check the mail archives as described above. If you do not see your message there then we did not see it either.

Or perhaps your formatting was such that it was unreadable. A number of posts are unreadable because the text was encoded and it could not be deciphered. If it does not show up as plain text in the archive then it did not get sent to the list in plain text format.


Next: , Previous: , Up: Top  

8 I use the Cygwin toolset and I have a problem.

The hard work the Cygwin team has done to port GNU Project software to MS-Windows is greatly admired. However the GNU team generally uses GNU Operating Systems and do not have access to Cygwin systems or MS-Windows systems which means that most of us can’t help you. It would be most appreciated if you would make your bug report directly to the Cygwin folks. They are the experts and best suited to handle your problem.

http://cygwin.com/

Cygwin Home Page

http://cygwin.com/faq/

Cygwin FAQ. Most excellent!

http://cygwin.com/bugs.html

Cygwin Bug Report Guidelines. This is a good reference for general bug reporting.


Next: , Previous: , Up: Top  

9 Why can only root chown files?

The GNU chown program will change the ownership if the operating system it is running upon allows it. If you can’t change file ownership then it is the operating system which is restricting you and not the chown program.

Actually, the GNU chown command does not know if this is the policy of the system or not. It calls the kernel system call chown() just like any other program (e.g. perl, ruby, etc.) If the OS allows it then it will change the ownership of the file. Different systems handle this differently. Traditional System V Unix systems allow anyone to give a file away to other owners. On those systems GNU chown does change the ownership of files.

But on most modern systems BSD semantics are followed and only the superuser can change the ownership of the file. The problem for documenting this is that GNU chown does not know which it will be running on. It could be one or it could be the other. Or it might even be running on a system without the concept of file ownership at all! This is really an OS policy decision and it is hard to track documentation to be different on different systems. But the documentation must be independent of operating system.

The reason an operating system needs to restrict changing ownership is mostly threefold.

  1. A user can create files that they cannot remove. With the old semantics it was possible for normal users to create situations that only the superuser could fix, such as creating a non-writable directory of files and then giving the ownership away. Since they no longer own the files they can’t change them, nor can they remove them. This is commonly a problem when untaring files that other people created with restrictive file permissions. The new semantics avoid this problem entirely. Therefore most systems today have been changed to disallow giving file ownership away. But as noted it has not always been that way.
  2. Additionally systems need to restrict who can remove files from system temporary directories such as /tmp. Otherwise anyone could remove any file there which might disrupt system processes. Some programs might even have security issues associated with this. Therefore modern systems set the ’sticky’ bit on /tmp. This is the ’t’ bit. On directories this prevents users from removing or renaming a file unless they own the file or directory. The reason this is important is that if chown is allowed then /tmp is another place that a file could be created by a user, chowned, and then could not be removed by the user. A sticky /tmp and a restricted chown policy go together. If you have one then you should have the other as well.
  3. People have used this to avoid disk space quota restrictions. Give the file to someone either with disk quota to spare or without any quota restrictions, like root. You can always copy the files back and then you own them again. This effectively defeats any quota system if allowed. Also, you can deny someone else service with a denial of service attack by using all of their quota until they cannot create any files. You use up their quota by chowning files to them someplace where they cannot find them.

Next: , Previous: , Up: Top  

10 chown fails when the username contains a ’.’ in it.

That was a problem with now quite old versions of fileutils that are unlikely to be in use anymore. Newer versions fix the problem. This entry is retained only to prevent ’makeinfo’ from renumbering the following entries.


Next: , Previous: , Up: Top  

11 How do I remove files that start with a dash?

Since the file name begins with a ’-’ it looks like an option to the command. You need to force it to not look like an option. Put a ./ in front of it. Or give it the full file name path. Or tell the command you are through with options by using the double dash to end all option processing. This is common to most traditional Unix commands.

rm ./-stuff
rm /full/path/-stuff
rm -- -stuff

And the same for other utilities too.

mv ./-stuff differentstuff
mv -- -stuff differentstuff

Next: , Previous: , Up: Top  

12 I have a file called --help. How do I remove it?

This is just a variant of the previous question. It also applies to files named ’-i’ and the like. But this question is asked often enough that it deserved an entry specifically for it.

rm ./--help
rm -- --help
rm ./-i
rm -- -i

In fact touching a file called ‘-i’ in a directory is an old trick to avoid accidentally saying ‘rm *’ and having it remove all of the files. Since the ‘*’ expands to match all file names then the first such name will be the ‘-i’. That will make the command become ‘rm -i file1 file2’. As you can see that will cause rm to prompt you and if that is not what you wanted then you can interrupt the command. I don’t personally like this and don’t recommend it.


Next: , Previous: , Up: Top  

13 I have a file ’-f’ and it affects rm.

I have a directory containing some files and also ’-f’ as a filename and rm thinks I gave it the -f option. Why?

$ ls
-f bar foo
$ rm -i *
$ ls
-f

This is not a bug. This is normal behavior. The shell expands globbing characters such as ’*’ before executing the command. The command that rm is seeing is the following.

rm -i -f bar baz foo fum

You can test the result by using the echo command to echo-print the command line without actually invoking it: echo rm -i *.

The -f option to rm overrides the -i option. Therefore the files are removed without asking. Since the shell expands globs like ’*’ before the program sees the command line it cannot distinguish between something the user typed and something that was expanded by the shell. The shell filters all command lines. This is a good thing and adds a lot of power to the system but it means you have to know that the shell expansion filter is there to write robust scripts and command lines.

To robustly write a command that does what you are wanting you need to do one of the following:

rm -i ./*
rm -i -- *

See the previous question with regards to a filename -i.


Next: , Previous: , Up: Top  

14 Why doesn’t rm -r *.pattern recurse like it should?

This question is asked a number of ways:

rm -R *.html
chmod -R 744 *.pl
chown -R user:user *.html

This is the same behavior as other typical programs such as ls -R, chmod -R, chown -R, etc. that also include recursive directory options. None of those commands search a directory hierarchy looking for matching files. Instead the shell expands the wildcards in the current directory only.

Here are the pieces of information you need to understand what is happening. The -R (and -r) option says that if any of the files listed on the command line are a directory then recurse down through those directories. Only arguments to the program which are directories are recursively acted upon. So any program argument which is a directory will be removed completely which would mean recursing down that directory and removing anything below it. But if the command line argument (after shell expansion) is not a directory then it won’t go searching trying to find a match. It is very unlikely that “*.html” will match a directory since that is typically a file suffix. It is very unlikely that -R with *.html will do anything useful. More often the recursive options are used with literal directory names or with the “.” current working directory.

The shell interpreter is expanding the command line file name glob characters prior to passing them to your command. This is a simple form of regular expression matching designed to make file name matching easier. This provides a consistent interface to all programs since the expansion code is common to all programs by being in the interpreting shell instead of in the program itself. Commands in Unix do not see the “*.html” or any of the shell meta-characters. Commands see the expanded names which the shell found matched file names in the current directory.

The “*” is the “glob” character because it matches a glob of characters. But it only matches files in the current directory. It does not list files in other directories. (Some modern shells have expanded this capability as a non-standard extension.) The shell matches and expands glob characters and passes the result to the command.

You can confirm this by using the echo command. This is built into most command shells, for example into bash. Try echo *.html. Try echo */*.html. In your example the first would print out “*.html” if nothing matched but would print out all file names that did match. The command would see the result and has no idea that you provided a wildcard to match against file names.

If you want to match files in subdirectories as well then you would need to say so explicitly with “*/*.html”. The first star would match all file names in the current directory. Then the second “*.html” would be matching files in the subdirectories under names already matched by the first “*” glob.

All of that was to explain why things are working as they should even if you are here reading this entry because you think they are not.

Given the question then here is what you really wanted to do. If you want to search all directories below a location, let’s say your present working directory or any arbitrary path on the file system, then you can use other Unix commands such as find to do so.

Here is an example using two commands, find and chmod, in combination in a way that keeps with the Unix philosophy.

chmod a+r $(find . -name '*.html' -print)

The $(command) executes the find command, takes the output of the find command and replaces it on the command line in place of the “$(...)” in a process known as command substitution and then passes the result to the rm command. Note that the ’*.html’ is quoted to keep the shell from expanding it. The find command will do the expansion itself in this case and so the ’*’ needs to be hidden in a string to keep the shell from expanding it first. The chmod command will receive a list of the files produced by the find as command line arguments.

Unfortunately this transferal of functionality from the command to the shell comes at a cost. In most operating system kernels there is a limited amount of argument space that is available for this argument expansion of file names. Combining commands using shell command substitution may fail if there are too many arguments. If you have a HUGE subdirectory with thousands of files the above command will fail to execute. (See the FAQ entry on “Argument list too long”.) The limit is different on different systems and getting larger as RAM gets cheaper but almost always there is still a limit. 20KB was typical for a time and now 2MB is common. But any limit is still a limit regardless. Additionally it is usually shared with environment variable space which also consumes it.

The xargs command was designed specifically to work around this argument space limit. Therefore in traditional systems a better method is to use find coupled with xargs.

find . -name '*.html' -print | xargs chmod a+r

You could substitute a full path in place of the ’find .’ such as something like ’find /class/home’.

By piping the data from one command to another the argument size limitation was avoided. This technique is very often seen in shell scripts. However this technique does not work as intended when filenames contain whitespace. A robust and safer method was implemented as an extension by GNU find using the “-print0” option and zero terminated strings instead of newline terminated strings:

find . -name '*.html' -print0 | xargs -0 chmod a+r

However that was never universally implemented. This made writing portable scripts difficult. The good news is that recent versions of find implement a new form of the -exec option using “{} +” which combine the xargs functionality into the find itself. This has been adopted as a POSIX standard and should now be available on all modern POSIX compliant systems.

find . -name '*.html' -exec chmod a+r {} +

Note the “+”. Traditionally a “;” was used with different behavior.


Next: , Previous: , Up: Top  

15 I used rm to remove a file. How can I get it back now?

It is extremely unlikely that you will be able to recover a file from the filesystem after you have removed it. It depends upon your operating system and the filesystem used with it. The disk blocks storing the file would have been freed and very likely reused by other files. This is especially true on an active file system. However, having said that disclaimer, it is frequently possible to recover deleted files.

Matt Schalit <mschalit at pacbell dot net> suggests in a message to the list that the user may want to look into The Coroner’s Toolkit, unrm, and lazarus.

http://www.porcupine.org/forensics/tct.html

Ciaran O’Riordan <coriordan at compsoc dot com> wrote suggesting that users should look at the programs ’recover’ and ’gtkrecover’. The latter being a gui interface for the former. The tool has been used with great success when O’Riordan’s 700 MB music collection was deleted. It is GPL and is a Debian package. It is ext2 specific but is available for both GNU/Linux and GNU/Hurd. Also, the ’recover’ homepage has a link to a manual way to retrieve deleted files.

http://recover.sourceforge.net/linux/recover/

On Debian systems:

apt-get install recover

See also “Linux Ext2fs Undeletion mini-HOWTO” by Aaron Crane at http://www.tldp.org/HOWTO/mini/Ext2fs-Undeletion.html. This contains very useful information about this topic. This may already be present on your system at /usr/share/doc/HOWTO/en-txt/mini/Ext2fs-Undeletion.gz or might be found other places using the locate Ext2fs-Undeletion command.

less /usr/share/doc/HOWTO/en-txt/mini/Ext2fs-Undeletion.gz

See also “Linux ext3 FAQ” by Juri Haberland at http://batleth.sapienti-sat.org/projects/FAQs/ext3-faq.html. Thanks to Paul Dorweiler for submitting that reference for this FAQ.

Q: How can I recover (undelete) deleted files from my ext3 partition?

Actually, you can’t! This is what one of the developers, Andreas Dilger, said about it:

In order to ensure that ext3 can safely resume an unlink after a crash, it actually zeros out the block pointers in the inode, whereas ext2 just marks these blocks as unused in the block bitmaps and marks the inode as "deleted" and leaves the block pointers alone.

Your only hope is to "grep" for parts of your files that have been deleted and hope for the best.

User green noted the ext4magic tool which is specifically written to recover files on ext3 and ext4 filesystems using the file system journal. See the homepage at http://developer.berlios.de/projects/ext4magic. This is also packaged for some software distributions such as Debian probably others too.

Apparently there is a process to undelete files that exist on ReiserFS file systems. See the “ReiserFS File Undelete HOWTO” page at http://www.cgsecurity.org/wiki/ReiserFS_File_Undelete_HOWTO for a description.

Sometimes people want to alias rm such that files are not actually removed but are instead saved off to a trashcan. When this is done files can be recovered from the trash if later this is found to be a mistake. Steve Ward suggested the following alias.

  alias del='mv --verbose --backup=simple --suffix=$(date +".(%F_%T)") --target-directory=$HOME/.Trash/'

Dennis German also suggested referencing this HOWTO at http://gentoo-wiki.com/HOWTO_Move_Files_to_a_Trash on how to configure your shell environment to use a trashcan. This is certainly not a good default for everyone but for those who desire it this may be a useful configuration.

You don’t want your files to be recovered and you want them to stay deleted? The GNU coreutils package includes a utility called shred which may be of some use to prevent it. Read the documentation on that program for more details especially the exceptions section where shred does not work. This operation does not work on journaled filesystems.

info shred

Next: , Previous: , Up: Top  

16 Why don’t the utilities have built in directory recursion?

People frequently wonder why cat, chgrp, chmod, chown, ls, mv, rmdir, rm, touch, do not support find type of operations. This is part of the subtle but beautiful design of the Unix system. Programs should be simple and modular. Common behavior should be modularized into a common location where it can be used by other programs. More complicated programs are created by chaining together simpler programs.

Here is an example of the Unix philosophy in action. Let’s say you want to restrict directory listings to only show directories. The normal thing would be to use a combination of commands. This following example will do what you want.

ls -al | grep ^d

This command assumes that you are using bash. Some other shells such as the old sh have ‘^’ as being a special character, a synonym for ‘|’, and you would need to quote it to avoid the special meaning. This command will generate the full listing and the grep will show only the lines starting with ’d’. Those are the directories. Doing this type of command chaining is a very typical way of doing things under Unix.

When people find that particular combinations of commands are ones that they use a lot then they will typically create a shell script, a shell function or a shell alias which does this with a shorter name. I like shell scripts because they are easier to pass around. If you found the above command one that you always used then perhaps you would create the following shell script.

File lsdir:

#!/bin/sh
ls -al "$@" | grep ^d

You could call it whatever you desired. Then it would work just like ls but with your special behavior. For all intents and purposes it would be just like a normal Unix command and its output could be piped into other commands.

This is embodied in what is known as The Unix Philosophy. There are many good discussions and without favoring one or another too much here are a few of them.

http://en.wikipedia.org/wiki/Unix_philosophy

Wikipedia entry.

http://www.faqs.org/docs/artu/ch01s06.html

Excerpts from Eric S. Raymond’s The Art of Unix Programming

http://www.google.com/search?q=the+unix+philosophy

Search for references on the web using Google’s search engine.

The best solution to use when needing file recursion is often to use the find program. It is a standard program that includes many powerful file finding features. It will work in combination with other commands to form a powerful combination for directory recursion.

Here is a simple example that changes the file permissions on a directory tree. The “{} +” is replaced by the largest set of filenames allowed by the operating system. This is a very efficient way to invoke commands on a large set of files.

find . -exec chmod a+r {} +

Next: , Previous: , Up: Top  

17 ls -d does not list directories!

I typed in ls -dl but it only showed me the current directory.

Well, yes, that is what it is supposed to do. The default directory to list if none is specified is the current directory. The -d option prevents recursively listing the directory. Therefore ls -ld lists only attributes of the current directory.

But I expected to see all of the directories.

That would be the output of ’ls -l’ without the -d option.

The -d option is meant to prevent ls from listing the contents of directories when you only want it to list the names of the directories.

To understand the usefulness of the -d option try this example. Compare the differences in the results of ls when you use the -d option versus when you do not.

ls -ld /etc/*.d
ls -l /etc/*.d

If you are trying to find files in the directory hierarchy then you should look into using the find command. It is very powerful and contains an interface which can be used with many other programs.

info find

Next: , Previous: , Up: Top  

18 ls -a * does not list dot files

When I list directories using ls *, or even ls -a * I never see hidden files, for example files with names that start with a dot as in ‘.profile’.

The command shell expands the ’*’ before ever handing it to a command. This is regardless of it being ls as it could be any command on the command line. The ’*’ is called the glob character because it matches a glob of characters. This process is called file name globbing. This is documented in your shell manual. Bash documents this well. In the bash info pages look for the section titled “Filename Expansion”.

A ’*’ is a shell pattern and is replaced by a list of files that match that pattern. Filenames that start with a ’.’ do not match that pattern. Neither does it match a ’?’. The dot character must be explicitly matched when it occurs at the start of the filename.

You should test what input is being given to commands by the shell with the echo command. Try these patterns as starting examples. Try this in your home directory where there are usually rich examples of dot files.

echo *
echo .*
echo .* *
echo .?*
echo .??*

As you will see from those examples the ls command is only listing out the files that were presented to it by the shell. Which answers why the dot files were not listed. In fact, what is ls doing that you can’t do yourself? Very little in this particular case. You might as well use echo for listing and then you can use the fmt command to word wrap to your screen. You can also use tr and grep to finish the job. Suddenly, using ls for this seems more convenient. Especially when coupled with the -l option!

echo * | fmt
echo .* | fmt
echo .* * | fmt
echo .* * | tr " " "\012" | grep profile

Okay, now I understand why ls * did not list out hidden dot files. But then how does one list out dot files?

There are several ways that are typically used. Here are a variety of examples that should spark ideas for you. Try them out and compare and contrast their differences.

ls -a | grep profile
ls -d .*
ls -d .??*
ls -d .[^.]*
ls -d .[!.]*

Some are more convenient than others. But dot files are meant to be hidden files. Therefore it is reasonable that you will need to do a little more work to unhide them.

You should also read over the answers to other questions in this FAQ as this is a very similar theme. Read the documentation on your shell. For GNU systems info bash will launch the info system on the bash documentation. I also recommend reading one of the many fine shell programming books available from the bookstore.


Next: , Previous: , Up: Top  

19 Argument list too long

I tried to move about 5000 files with mv, but it said:

$ mv * /some/other/directory/
bash: /bin/mv: Argument list too long

A scalable equivalent to the above is:

find . -maxdepth 1 -name '[!.]*' -print0 |
xargs -r0 mv --target='/some/other/directory'

The traditional Unix operating system kernel has a fixed limit of memory available for program environment and argument list combined. Into this combined program environment and argument space all of the environment and all of the command line arguments must be fit. If they don’t fit then the external program execution will fail. In the above example the * is expanded by the shell into a literal list of filenames to be passed as command line arguments to the mv program. If the number of files in the current directory to be expanded is larger than this fixed buffer space including the environment space then they won’t fit. In that case it is impossible to execute the program with the specified list of arguments and the current environment size.

Note that the “Argument list too long” message came from the bash shell command line interpreter. Its job is to expand command line wildcard characters that match filenames. It expands them before any program can access them. This is common to all programs on most Unix-like operating systems. It cannot exceed the OS limit of ARG_MAX and if it tries to do so the error “Argument list too long” is returned to the shell and the shell returns it to you.

This is not a bug in mv or other utilities nor is it a bug in bash or any other shell. It is an architecture limitation of Unix-like operating systems. The mv program was prevented by the OS from running and the shell is just the one in the middle reporting the problem. The shell tried to load the program but the OS ran out of space. However, this problem is one that is easily worked around using the supplied utilities. Please review the documentation on find and xargs for one possible combination of programs that work well.

The value can vary per operating system. POSIX only requires 4k as the minimum amount. Newer operating systems releases usually increase this to a somewhat larger value. On my Linux system (2.2.12) that amount is 128k bytes. On my HP-UX system (11.0) that amount is 2M bytes. HP-UX provided 20k for many years before increasing it to 2M. Sven Mascheck wrote to the mailing list that early systems such as Unix V7, System III, 3BSD used 5120 bytes and that early 4BSD and System V used 10240 bytes. More information about ARG_MAX from Sven Mascheck is available on his page http://www.in-ulm.de/~mascheck/various/argmax/.

The getconf command may be used to print the currently implemented limit.

getconf ARG_MAX
131072

It might occur to you to increase the value of ARG_MAX such as by recompiling the kernel with a larger value but I advise against it. Any limit, even if large, is still a limit. As long as it exists then it should be worked around for robust script operation. On the command line most of us ignore it unless we exceed it at which time we fall back to more robust methods.

Here is an example using chmod where exceeding ARG_MAX argument length is avoided by using find.

find htdocs -name '*.html' -exec chmod a+r {} +

Read the previous question for another facet of this problem.

Recent News: The Linux kernel has removed the classic ARG_MAX limitation. See the changeset on git.kernel.org for the change. This was released with the linux-2.6.23 kernel. It will eventually propagate into the release software distributions which include it. Note that glibc’s getconf ARG_MAX hasn’t caught up yet and still reports a fixed value.


Next: , Previous: , Up: Top  

20 I am trying to compile a C program ...

People trying to develop compiled software often ask questions that start with, “I have found a bug in the sleep() function.” Or ask a question similar to, “I am trying to use the chown() function in my program.” They confuse the library call with the program utility because they have the same name. They are having a problem using the library routine in their compiled program but are asking the question of the command line program developers.

Frequently the same name is used for both a program and a system call. Most of the programs got their names because that was what the system call was named. One caused the other. Which means that it is very likely that the sleep program was named because it used the sleep library routine.

This confusion between library routines and command line programs usually happens with chgrp, chmod, chown, mkdir, mknod, nice, rmdir, sleep, sync, uname, or one of the other programs that have a name matching a C programming routine. It can be confusing to realize that the program is a wrapper to the underlying library routine or system call. If you are compiling a program then you are NOT using the GNU command line utility in your C/C++ program.

You may be confusing the C library routine used in your program with the shell command line program. They are not related. Except that the program calls the library function just like your program does. If you have a real bug in the library routine then you need to determine who supplied that library and report the bug to them. If it is the GNU Lib C (a.k.a. glibc) provided library then the glibc mailing list at <bug-glibc@gnu.org> might be a useful reference.

The online man pages can also be confusing in regard to this. If you are trying to use chown() in a program and get a man page you will likely get the man page for the command line program and not the library routine you were looking for. You must specify the section of the manual that you want information.

man chown
man 2 chown
man sleep
man 3 sleep

Traditionally man pages are organized by section number. Section 1 are command line programs. Section 2 are operating system calls, aka kernel calls. Section 3 are library routines. Library routines may or may not use system calls. Etc. Other sections are used for other purposes that are not germane here.

Many GNU programs have deprecated the man pages and have moved entirely to info documentation. But man pages still have a loyal following and are quite useful for reference. But they make poor user guides.


Next: , Previous: , Up: Top  

21 I am having a problem with kill, nice, pwd, sleep, or test.

This is frequently a variant of the previous question. Go read it first and then come back here. This time the same name is used by a utility and is also built into a command line shell. In which case you might need to report your problem to your shell maintainers.

Many commands are built into your command shell so as to speed up shell scripts. People frequently get confused over whether they are running the external program or the internal shell builtin. This usually happens with kill, nice, pwd, printf, sleep, test, and some others that exist both as internal and external commands. Double check that the problem you are having is with the external program. If it is the internal version then contemplate reporting it to <bug-bash@gnu.org>.

Many times a program is required by circumstances to exist both as a builtin and as an external. Usually this is because of the need to be exec’d directly from another program without a shell to implement it as a builtin. Therefore it must exist as an external, standalone utility. However, those utilities are also built into most shells in order to improve the execution speed.


Next: , Previous: , Up: Top  

22 echo test printf kill

Q: echo/test/printf/test/[/kill do not work as documented. Commonly, one of these doesn’t work (yet expected to work):

echo --help
echo --version
echo -e '\x45'
printf --version
printf '\u263a\n'
kill --help

This is another variation of the previous two items above.

A: These commands are often implemented as shell builtin functions, supporting less features than GNU coreutils’ versions.

Use ’type’ (bash builtin) to determine if the command is a shell builtin:

$ type printf
printf is a shell builtin

Versus:

$ type printf
printf is /usr/bin/printf

To use the external version of these programs, prefix with ’env’ or specify a full path.

Shell builtin functions on do not support –version:

$ echo --version
--version
$ printf --version
bash: printf: --: invalid option
printf: usage: printf [-v var] format [arguments]

While GNU coreutils programs do support –version (and other features mentioned in the documentation):

$ env echo --version | head -n1
echo (GNU coreutils) 8.21
$ env printf --version | head -n1
printf (GNU coreutils) 8.21
$ /bin/echo --version | head -n1
echo (GNU coreutils) 8.21
$ env echo -e '==\x3D=='
=====
$ env printf '\u263a\n'
[prints unicode smily face, if supported by your terminal]

These features often extends POSIX, and are not portable. To restore strict POSIX compliance, use POSIXLY_CORRECT environment variable as explained by Eric Blake: http://lists.gnu.org/archive/html/bug-coreutils/2013-03/msg00016.html

This entry submitted by Assaf Gordon.


Next: , Previous: , Up: Top  

23 Sort does not sort in normal order!

Why is sort not sorting correctly?

Some years ago at the time of transition to locale aware libc routines this question arose very often. With the passage of time most people have learned about this behavior and workarounds are quite well known. But the question is still raised often. It is most often due to having LANG set to ‘en_US.UTF-8’ set in your user environment. At that point sort appears broken because case is folded and punctuation is ignored because ‘en_US.UTF-8’ specifies this behavior. Once specifically requested by LANG and other LC_* variables, sort and other locale knowledgeable programs must respect that setting and sort according to the operating system locale tables. That is why this is not a bug in sort, sort is doing the right thing, even if the behavior is generally undesirable. Most of the language specific locales have tables that specify the sort behavior to ignore punctuation and to fold case. This is counter intuitive to most long time computer users!

Of course locales are a blessing to non-english speaking computer users. Many languages use non-ASCII fonts and character sets. The POSIX standards and the GNU utilities support those by using the installed system library sorting routines. By using locales languages can specify a specific ordering. This can be done by a translator well after the program has been written and by translators not known by the program author. It is a dynamic change to the program. However, when those tables exhibit an undesired behavior then it can also break an otherwise correct program. If locale tables force a locale specific collation sequence when a standard collation sequence was desired then this is most noticeable with the sort command and so it bears the full force of the problem.

This locale sorting behavior is configured in your environment. This is due to the fact that you or your vendor have set environment variables that direct the program to use locale specific sorting tables which do not sort as you expect. You or your vendor have probably set environment variables such as LANG, LC_COLLATE, or LC_ALL to ‘en_US.UTF-8’. The locale collating table selected by the locale setting is not part of the GNU program but part of your vendor’s system release.

To select a standard sort ordering simply set LC_ALL to ‘C’ or ‘POSIX’.

# If you use bash or some other Bourne-based shell,
export LC_ALL=C

# If you use a C-shell,
setenv LC_ALL C

and it will then work the way you expect because it will use the standard set of tables. The standard set of tables is the traditional one for US-ASCII. However this will probably not sort characters outside of that character set correctly. So far there is still no fully satisfactory solution to this problem. If you find one then please contact me so that this information can be listed.

The locale affects almost every command that operates on characters. It has very deep reaching effects in addition to ordering. For instance it causes regular expressions to operate more slowly. Many commands execute more slowly in multibyte locales. Often the normal grep, sed, awk commands are all very much slower when operating in a UTF locale than when operating in the C locale. For this and related reasons a very commonly seen workaround in scripts is to set LC_ALL=C so as to force standard locale behavior for both sorting and for regular expressions.

See the standards documentation for more information on the locale variables with regards to sort. Also see http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html#tag_08_02 for more information on environment variables that control localization.


Next: , Previous: , Up: Top  

24 The ls command is not listing files in a normal order!

This is just a variant of the previous question. Any program that is compliant with the standards and implements locale based collating sequences to support non-ASCII languages will be affected.

See the standards documentation for more information on the locale variables with regards to ls.


Next: , Previous: , Up: Top  

25 The date command is not working right.

$ date -d "2006-04-02 02:30:00"
date: invalid date `2006-04-02 02:30:00'

There are several reasons why date might report a date as invalid. The usual reason is that the exact time indicated doesn’t exist because the local daylight saving time changed and skipped over that moment in time. The answer to the above was due to the timezone switched to DST just at that moment in time. This can be seen by using zdump to display the time zone data.

$ zdump -v US/Mountain | grep 2006
US/Mountain  Sun Apr  2 08:59:59 2006 UTC = Sun Apr  2 01:59:59 2006 MST isdst=0 gmtoff=-25200
US/Mountain  Sun Apr  2 09:00:00 2006 UTC = Sun Apr  2 03:00:00 2006 MDT isdst=1 gmtoff=-21600
US/Mountain  Sun Oct 29 07:59:59 2006 UTC = Sun Oct 29 01:59:59 2006 MDT isdst=1 gmtoff=-21600
US/Mountain  Sun Oct 29 08:00:00 2006 UTC = Sun Oct 29 01:00:00 2006 MST isdst=0 gmtoff=-25200

This shows that on my system daylight time begins on Apr 2 03:00:00 2006 MDT with the last second before being Apr 2 01:59:59 2006 MST and therefore "2006-04-02 02:30:00" cannot be a valid time in the US. It may be a valid time elsewhere in another timezone.

When debugging these problems including the timezone information is very important to understanding the problem. When reporting bugs please include the time zone in question. Also note that localized timezone names such as “EST” are ambiguous. For portability the numeric timezone offsets such as produced by date -R are best.

The parsing of dates with date --date=STRING is a GNU extension and not covered by any standards beyond those to which GNU holds itself. In which case whatever GNU decides to do is what it decides to do. However touch -d STRING is defined by POSIX and is implemented with the same date string parsing code. Therefore you can expect that similar rules apply to both. Currently the date parsing code makes a best guess at understanding the string in a variety of formats but isn’t always able to understand all possible string variations. See the date manual for an extended description of the syntax. But in short summary it is intended to be a human date string.

$ date --date="Wed, 07 Jan 2009 14:03:59 -0700" +"%F %T %z"
2009-01-07 14:03:59 -0700

$ date --date="2009-01-07 14:03:59 -0700" +"%F %T %z"
2009-01-07 14:03:59 -0700

$ date --date="Wed Jan  7 14:03:59 MST 2009" +"%F %T %z"
2009-01-07 14:03:59 -0700

$ date -d today +"%F %T %z"
2009-01-07 14:06:23 -0700

$ date -d tomorrow +"%F %T %z"
2009-01-08 14:06:38 -0700

Caution: Named timezones may not be unambiguous. In particular the locale-dependent string from the traditional legacy default date output may produce a string that can’t be parsed unambiguously. It is better to use the date -R format when producing output from date that is meant to be parsed again by it.

$ date
Mon Jul  2 13:10:11 MDT 2012  # <-- Warning: Text timezone may not be unique.

$ date -R
Mon, 02 Jul 2012 13:10:23 -0600  # <-- Info: Unambiguous time string.

$ date -R -d "$(date -R) + 2 days"
Wed, 04 Jul 2012 13:11:05 -0600

However the date -R format resolves time to the second. That is usually okay. But if resolution smaller than one second is needed, such as for setting a clock, then a high resolution and unambiguous time should be used. In addition to other formats the date --date=STRING may take the time in Unix time format. This is the number of seconds since midnight UTC of January 1, 1970. See Wikipidia’s Unix time page for a good description of Unix time. Precede the number with an @ character to indicate that the time string is a Unix time in seconds since the epoch.

$ date +%s
1243720606

$ date -R --date=@1243720606
Sat, 30 May 2009 15:56:46 -0600

$ date +%s.%N
1243720606.134242460

$ date --date=@1243720606.134242460 +"%F %T.%N %z"
2009-05-30 15:56:46.134242460 -0600

$ date --date=@1243720606 +"%F %T.%N %z"
2009-05-30 15:56:46.000000000 -0600

$ date --date=@1243720606.134242460 +"@%s.%N"
@1243720606.134242460

As noted in the info documentation fuzzy relative items in date strings may cause the result to end up at an undesired time. Such as at an invalid time due to a time change or in the same month instead of a different month. Therefore it is more robust to anchor the time at a particular safer starting point before applying a relative time modifier. Such as operating at noon to avoid time changes that happen at night. Such as operating at the middle of the month to avoid different lengths of months.

If you are calculating relative days from today then you can simply use today as the day reference point and then specify 12:00 to work at noon.

$ date -d "12:00 today -1 days" +"Yesterday was %A."
Yesterday was Saturday.

$ date -d "12:00 yesterday" +"Yesterday was %A."
Yesterday was Saturday.

$ date -d "12:00 tomorrow" +"Tomorrow will be %A."
Tomorrow will be Monday.

If you are working with months or years then it is more robust to operate at the middle of the month. Unfortunately that isn’t today and so needs a little bit of calculation. Use date +%Y-%m-15 to generate the date at the middle of the month.

$ date --date="$(date +%Y-%m-15) -1 month" +'Last month was %B.'
Last month was June.

$ date --date="$(date +%Y-%m-15) +1 month" +'Next month will be %B.'
Next month will be May.

$ date --date="$(date +%Y-%m-15) 1 year ago" +'Last year was %Y.'
Last year was 2008.

$ date --date="$(date +%Y-%m-15) +1 year" +'Next year will be %Y.'
Next year will be 2010.

If doing several calculations then it is possible to cross midnight between the multiple calls and produce confusing results. To avoid this take the time into a variable and then always use that variable for the calculations.

$ now=$(date +%s)
$ today=$(date --date="@$now" "+%F 12:00")

$ echo "Yesterday=$(date -d "$today 1 day ago" +%A), Tomorrow=$(date -d "$today 1 day" +%A)."
Yesterday: Saturday, Tomorrow: Monday.

$ thismonth=$(date --date="@$now" "+%Y-%m-15 12:00")
$ echo "LastMonth=$(date -d "$thismonth 1 month ago" +%B), NextMonth=$(date -d "$thismonth 1 month" +%B)."
LastMonth=March, NextMonth=May.

Tip: GNU date itself doesn’t include any direct support for finding days between dates. But it can be used as part of a script to find this information. The technique is to convert each date to an integer value such as a Julian Day Number or seconds since the Unix epoch 1970-01-01 00:00:00 UTC and then do take the difference and then convert the seconds to days. Use Unix seconds is very convenient due to the %s format. Here is an example.

date1="2008-01-01"
date2="2010-06-21"
date1seconds=$(date -d "$date1 12:00" +%s)
date2seconds=$(date -d "$date2 12:00" +%s)
totalseconds=$(( $date2seconds - $date1seconds ))
secondsperday=86400
days=$(( $totalseconds / $secondsperday ))
echo "There were $days days between $date1 and $date2"

And of course that can be considerably compacted by applying the constructs inline but this is not as illustrative and so was expanded out in the above example. Here is a compact version.

$ echo Days between dates: $(( ( $(date -d "2010-06-21 12:00" +%s) - $(date -d "2008-01-01 12:00" +%s) ) / 86400 ))

The NEWS shows the following snippet which is useful and seemingly not well known.

Tip: Date values can now have leading TZ=”...” assignments that override the environment only while that date is being processed. For example, the following shell command converts from Paris to New York time:

TZ="America/New_York" date --date='TZ="Europe/Paris" 2004-10-31 06:30'

At the change of the year there are always many bug reports concerning date behaving incorrectly. This is usually due to people using a mismatch of ISO and legacy date format specifiers.

$ date -d 2008-12-31 +%Y%V
200801

$ date -d 2008-12-31 "+%G-%m-%d"
2009-12-31

The %Y and %U or %W options work in combination. (Use %U for weeks starting with Sunday or %W for weeks starting with Monday.) The ISO %G and %V options work in combination. Mixing them up creates confusion. Instead use %Y and %U/%W together or use %G and %V together.

$ date -d 2008-12-31 +%G%V
200901

$ date -d 2009-01-01 +%G%V
200901

$ date -d 2008-12-31 +%Y%U
200852

$ date -d 2009-01-01 +%Y%U
200900

Use of ISO week numbers tends to create confusion. The ISO week numbering scheme is somewhat different from calendar week numbering. ISO week numbers start on Monday of the week with the year’s first Thursday in it. See Wikipidia’s ISO 8601 page or Wikipidia’s ISO week date page for a good summary.

ISO Week Dates can be created using the following format.

$ date -d "2009-01-07 12:00:00 +0000" "+%G-W%V-%u"
2009-W02-3

See the standards documentation for more information with regards to date.

Caution: Recently the appearance of different glyphs for the dash character in typsetting systems has caused some confusing when cutting and pasting examples. This can cause errors parsing the different and unexpected unicode characters that look like a dash but are not. (e.g. The dashes in 2009-01-07 might be unicode hyphens.) If you think the date looks okay but still produces a syntax error try inspecting the character set and replacing the dashes with ’-’ characters.

Urs Thuermann wrote a nice mailing list posting describing related concepts in some detail:

http://lists.debian.org/ygf624jkmyo.fsf@janus.isnogud.escape.de

Documentation on specifying the timezone with TZ may be found on a GNU system in the Texinfo documentation system in the “TZ Variable” section under the “Calendar Time” chapter. On a GNU system you can get there directly with this command line.

$ info libc 'TZ Variable'

This most recent version is also available online.

http://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html

See also the online POSIX documentation.

http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html

Next: , Previous: , Up: Top  

26 ln -s did not link my files together

Symbolic links are created with ln -s. That creates a name redirection. When a symlink is accessed the filesystem will take the contents of the symlink as a redirection to another file, where the process may recursively be continued many times. If you were meaning ln -s a /c/b then that would create ‘/c/b’ which would be a relative symlink to file ‘/c/a’. If ‘/c/a’ did not exist then this would be dangling until the file was created.

The owner, group and mode of a symlink are not significant to file access through it.

Symbolic links may use either absolute or relative paths but there are trade offs. Generally I advocate making only relative links so that the location is network independent and will work desirably across NFS mounted filesystems.

The this FreeBSD man page symlink(7) on the web includes a nice description of symlinks.

http://www.freebsd.org/cgi/man.cgi?query=symlink&sektion=7&manpath=FreeBSD+8.2-RELEASE

See also these mailing list threads for more discussion of this issue.

http://lists.gnu.org/archive/html/bug-coreutils/2007-12/msg00211.html

http://lists.gnu.org/archive/html/bug-coreutils/2007-11/msg00006.html

http://lists.gnu.org/archive/html/bug-coreutils/2011-12/msg00086.html

Next: , Previous: , Up: Top  

27 How do I change the ownership or permissions of a symlink?

The owner, group, and permissions of a symlink are not in any way significant. Only the value of the symlink is meaningful. Regardless of that some operating systems will allow you to change the owner, group or mode of a symlink and other operating systems will not. Do not worry about it as it does not matter in any case.


Next: , Previous: , Up: Top  

28 Value too large for defined data type

It means that your version of the utilities were not compiled with large file support enabled. The GNU utilities do support large files if they are compiled to do so. You may want to compile them again and make sure that large file support is enabled. This support is automatically configured by autoconf on most systems. But it is possible that on your particular system it could not determine how to do that and therefore autoconf concluded that your system did not support large files.

The message "Value too large for defined data type" is a system error message reported when an operation on a large file is attempted using a non-large file data type. Large files are defined as anything larger than a signed 32-bit integer, or stated differently, larger than 2GB.

Many system calls that deal with files return values in a "long int" data type. On 32-bit hardware a long int is 32-bits and therefore this imposes a 2GB limit on the size of files. When this was invented that was HUGE and it was hard to conceive of needing anything that large. Time has passed and files can be much larger today. On native 64-bit systems the file size limit is usually 2GB * 2GB. Which we will again think is huge.

On a 32-bit system with a 32-bit "long int" you find that you can’t make it any bigger and also maintain compatibility with previous programs. Changing that would break many things! But many systems make it possible to switch into a new program mode which rewrites all of the file operations into a 64-bit program model. Instead of "long" they use a new data type called "off_t" which is constructed to be 64-bits in size. Program source code must be written to use the off_t data type instead of the long data type. This is typically done by defining -D_FILE_OFFSET_BITS=64 or some such. It is system dependent. Once done and once switched into this new mode most programs will support large files just fine.

See the next question if you have inadvertently created a large file and now need some way to deal with it.


Next: , Previous: , Up: Top  

29 Tar created a Large File but I can’t remove it.

I created a file with tar cvf backup.tar. Trying to "rm" this file is not possible. The error message is:

rm: cannot remove `backup.tar': Value too large for defined data

What could I do to remove that file ?

Sometimes one utility such as tar will be compiled with large file support while another utility like rm will be compiled without. It happens. Which means you might find yourself with a large file created by one utility but unable to work with it with another.

At this point we need to be clever. Find a utility that can operate on a large file and use it to truncate the file. Here are several examples of how to work around this problem. Of course in a perfect world you would recompile the utilities to support large files and not worry about needing a workaround.

This example again requires perl to be configured for large files.

perl -e 'unlink("backup.tar");'

So let’s try to hit it more directly. Truncate the file first. That will make it small and then you can remove it. The shell will do this when redirecting the output of commands.

  true > backup.tar
  rm backup.tar

However, if your shell was not compiled for large files then the redirection will fail. In that case we have to resort to more subtle methods. Since tar created the file then tar must be configured to support large files. Use that to your advantage to truncate the file.

tar cvf backup.tar /dev/null

Next: , Previous: , Up: Top  

30 The ’od -x’ command prints bytes in the wrong order.

That is the required behavior depending upon the endianess of the underlying machine. You are probably mixing up words and bytes.

The -x option outputs short integers, note integers which is a word and not a byte, in the machine’s short integer format. If you are operating on a little endian machine such as an x86 then the bytes appear in ’backwords’ order. Here is what the od documentation says.

`-x'
     Output as hexadecimal two-byte units.  Equivalent to `-t x2'.

If you require a specific byte ordering, note bytes not words, then you need to supply a byte specification such as ’od -t x1’.

echo abcdefgh > /tmp/letters
od -cx /tmp/letters
0000000   a   b   c   d   e   f   g   h  \n  \0
        6261 6463 6665 6867 000a
0000011
od -t cx1 /tmp/letters
0000000   a   b   c   d   e   f   g   h  \n
        61 62 63 64 65 66 67 68 0a
0000011

Alternatively since version 8.23 (2014), od supports the --endian option, allowing selection of a specific byte ordering.

If you search the web for “little endian big endian” you should turn up many hits for various documentation on this subject. But for sure you should read “On Holy Wars and a Plea for Peace” written by Danny Cohen and published in IEEE Computer years ago as it is a classic treatise on the subject.

http://www.ietf.org/rfc/ien/ien137.txt

Next: , Previous: , Up: Top  

31 expr 2 * 3 does not work

The expr program appears to be broken because expr 2 * 3 produces a syntax error.

expr 2 * 3
expr: syntax error

The case shown is not quoted correctly. As such it provides a good example of incorrectly quoting shell metacharacters. The “*” is being expanded by the shell and the expr is seeing filenames creating the syntax error.

The “*” character is a special character to the command shell. It is called a glob character because the shell expands it to match filenames. It matches a glob of characters. The shell then passes the result to the command. The expr program does not see the star character in “2 * 3” but instead sees a “2” followed by every filename matched from the current directory and then finally the “3”.

Your command is really something completely different than you thought it was going to be. Use echo to see what your command line really is:

echo expr 2 * 3

You will see it matching filenames and so create the syntax error.

There are many entries in this FAQ related to command line “*” glob expansion. If you are having trouble with this concept then read through the other entries for a fresh perspective.

You need to quote shell metacharacters to prevent them from being modified by the shell. Here are some possible ways of quoting this example.

expr 2 \* 3
expr 2 '*' 3
expr 2 "*" 3

The man page for expr carries this text:

Beware that many operators need to be escaped or quoted for shells.


Next: , Previous: , Up: Top  

32 df and du report different information

Why don’t df and du report the same values?

This is a sometimes confusing topic. The df will report a certain value for the amount of disk space used in a filesystem. The user then tries to add up the amount of disk spaced consumed in the directory structure using du. The numbers are rarely exactly the same and are sometimes quite a large amount different! Why is this?

Fundamentally it is because df and du report different information. The df command reports overall filesystem information. The du command reports information in directory hierarchies. Those are two different things and they may report different information. In particular data that is not recorded within a directory hierarchy will not be shown by du.

A very common case where this occurs is when a program writes a very large log file. What often happens is that the user notices this large growing file and removes (unlinks) the file from the directory. If the program that was writing the file continues running and continues to maintain a handle to that file then the file will continue to exist in the filesystem. This will continue to be reported by df but will no longer be accessible through the directory hierarchy and du.

James Youngman contributed this nice explanation to the mailing list.

http://lists.gnu.org/archive/html/bug-coreutils/2007-09/msg00119.html

Another explanation written to the mailing list.

http://lists.gnu.org/archive/html/bug-coreutils/2007-07/msg00107.html

A related question is, “Why does simply trying to add up all of the disk space used by files not result in the amount of disk space consumed in the filesystem?” Because some disk space is lost in filesystem overhead such as in block fragments and filesystem inodes. This is especially noticeable in some filesystems with large block sizes when there are many very small files.


Next: , Previous: , Up: Top  

33 df Size and Used and Available do not add up

The df report simply does not add up? Why not?

$ df /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             40559188  35847132   2651768  94% /
$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              39G   35G  2.6G  94% /

The most natural thing in the world is to add the values of Used plus Available and expect to have a result that equals Size. But as we see here 35847132 plus 2651768 is not equal to 40559188 and is missing aproximately 2G of disk. Where did it go?

This data is in the minfree percentage of reserved filesystem disk blocks. A typical filesystem value for minfree is 5% reserved to superuser processes. Root can make use of all of the disk space but non-root processes will be restricted by the minfree value. If a user or user process fills up a partition the root user can still create files within the provided space.

Additionally modern filesystems attempt to control the amount of disk fragmentation automatically. This requires that there is sufficient free disk space available. When the filesystem is operated very close to 100% full then undesirable fragmentation is increased. This may significantly decrease disk performance. Keeping a minfree reserved is one way to ensure a sufficient amount of disk space for the filesystem to operate at high efficiency.

In this example 5% of 39G is reserved and not included in the Available value. 39G * 5% is about 1.9G minfree. 35G used plus 2.6G available plus 1.9G minfree is aproximately 39G and equal to the size of the filesystem.

The tunefs command using the tunefs -m NUM option is the traditional command to adjust the filesystem minfree value. More information may be found in the manual pages and documentation for that command.


Next: , Previous: , Up: Top  

34 Old tail plus N syntax now fails

Old tail syntax tail +N FILE now fails.

$ tail +5 FILENAME
tail: cannot open `+5' for reading: No such file or directory
==> FILENAME <==

The problem is that “+” is a valid filename character and is not an expected option character. Options are supposed to start with a “-” character. It has always been a bad idea to eat up an additional character as an additional option specifier. This is now no longer allowed by the standards. The tail command is now required to respect that “+” is not an option and treat it as a filename.

This is is documented in the NEWS file.

A few usages still have behavior that depends on which POSIX standard is
being conformed to, and portable applications should beware these
problematic usages.  These include:

  Problematic       Standard-conforming replacement, depending on
     usage            whether you prefer the behavior of:
                    POSIX 1003.2-1992    POSIX 1003.1-2001
  sort +4           sort -k 5            sort ./+4
  tail +4           tail -n +4           tail ./+4
  tail - f          tail f               [see (*) below]
  tail -c 4         tail -c 10 ./4       tail -c4
  touch 12312359 f  touch -t 12312359 f  touch ./12312359 f
  uniq +4           uniq -s 4            uniq ./+4

  (*) "tail - f" does not conform to POSIX 1003.1-2001; to read
  standard input and then "f", use the command "tail -- - f".

These changes are in response to decisions taken in the January 2005
Austin Group standardization meeting.  For more details, please see
"Utility Syntax Guidelines" in the Minutes of the January 2005
Meeting http://www.opengroup.org/austin/docs/austin_239.html.

Also the coreutils info manual has the following documentation.

The GNU utilities normally conform to the version of POSIX that is standard for your system. To cause them to conform to a different version of POSIX, define the _POSIX2_VERSION environment variable to a value of the form yyyymm specifying the year and month the standard was adopted. Two values are currently supported for _POSIX2_VERSION: ‘199209’ stands for POSIX 1003.2-1992, and ‘200112’ stands for POSIX 1003.1-2001. For example, if you have a newer system but are running software that assumes an older version of POSIX and uses ‘sort +1’ or ‘tail +10’, you can work around any compatibility problems by setting ‘_POSIX2_VERSION=199209’ in your environment.

What does this mean? It means that you should avoid tail +4, since it might mean either tail ./+4 or tail -n +4. It also means that there are several ways of dealing with this problem.

tail -n +5 SOMEFILE

Use the “-n” option to specify the number of lines.

_POSIX2_VERSION=199209 tail +5

Use the _POSIX2_VERSION environment variable to specify the level of standard conformance that is desired.

_POSIX2_VERSION=200112 tail +5

Use the _POSIX2_VERSION environment variable to specify the level of standard conformance that is desired.

The environment variable may be set in the environment and the effect will be global. Note that setting _POSIX2_VERSION will also affect the behavior of sort, uniq, touch and perhaps other programs.

_POSIX2_VERSION=199209
export _POSIX2_VERSION
tail +5

In many ways for portable scripts it is better to avoid using tail at all and use other standard utilities. For example use sed instead of tail. Use of sed is very portable.

Instead of tail +5 use either:

sed '1,4d'

Or use:

sed -n '5,$p'

Next: , Previous: , Up: Top  

35 join requires sorted input files

A common misconception is that join works on arbitrary input files. But join requires that the input files be sorted. The join --help documentation says:

Important: FILE1 and FILE2 must be sorted on the join fields.
E.g., use ` sort -k 1b,1 ' if `join' has no options,
or use ` join -t '' ' if `sort' has no options.
Note, comparisons honor the rules specified by `LC_COLLATE'.

Since coreutils release 6.11 (2008-04-19), if the input is not sorted and some lines cannot be joined, a warning message will be given.


Next: , Previous: , Up: Top  

36 cp and mv: the –reply=X option is deprecated

UPDATE: The –reply option has been removed in release 7.1.

In coreutils release 5.90 (2005-09-29) the cp and mv --reply=X option was deprecated. There was a discussion of the problems and issues on the bug-coreutils mailing list. As a result of the discussion it was then offically deprecated. The discussion may be found here in the mailing list archives:

http://lists.gnu.org/archive/html/bug-coreutils/2005-06/msg00160.html

http://lists.gnu.org/archive/html/bug-coreutils/2005-07/msg00009.html

The ‘--reply’ options only controlled the action when prompts would have been asked. It did not have any effect for when the program would not have prompted at all. Meaning that many users called –reply=no in a non-interactive use such as through a script by cron and since it wouldn’t prompt in that case it never got the “no” answer and would still overwrite the target file! Clearly that was not what was intended and very unfortunate behavior. This was so confusing to people that the ‘--reply’ option has been deprecated and the option is scheduled for removal in the future. It did not do what you want it to do anyway.

In the 7.1 release the ‘-n’, ‘--no-clobber’ option was introduced. It does what you wanted the other to do but didn’t.

Here is the NEWS entry.

cp and mv: the --reply={yes,no,query} option has been removed.
Using it has elicited a warning for the last three years.

cp and mv accept a new option, --no-clobber (-n): silently refrain
from overwriting any existing destination file

A good alternative is to use the rsync command using the --ignore-existing option.

$ rsync -a --ignore-existing file1 file2

Previous: , Up: Top  

37 uname is system specific

Due to the peculiarities of the Linux kernel there are often questions like this one concerning the uname -r -v output.

The man page for uname says:

-r, --kernel-release     print the kernel release
-v, --kernel-version     print the kernel version

But the Linux kernel seems to return the opposite value for each.

$ uname -r
2.6.32-431.1.2.0.1.el6.x86_64

$ uname -v
#1 SMP Fri Dec 13 13:06:13 UTC 2013

Isn’t that a bug?

This is not a bug in the uname command. The uname(3) command line utility is simply a thin wrapper around the uname(2) system call. It displays the kernel information stored in the kernel’s struct utsname data structure. That information is determined by the operating system kernel not the uname(3) command line display tool.

The short answer is that the kernel has chosen to store that specific data and it isn’t anything that the uname display program can change. If you want it changed then it must be changed in the kernel.

The long answer is that the uname command is a very old command and has been around a very long time. The utility of the command itself dates way back into the history of Unix and the origins of communication between different hosts. (If anyone can point me to a good reference on the original history of this please contact me.) The system name and host names are limited to eight characters which is smaller than the full field width available to those fields today. Longer strings will be truncated. The command has always been very system specific. The output of uname is not portable across different systems.

The only portable way to use uname(1) is to call it first without arguments to see which system name it returns and then after knowing the system type then call it again with whatever options make sense on that system.

A recent log of the discussion in the bug tracker includes more information and links to duplicate bug reports on the same topic.

http://debbugs.gnu.org/14388

At the time of this writing someone at Wikipedia has documented a list of uname outputs from various systems. I think that is unmanageable and at some point in the future I expect a deletionist will delete it. But in the meantime a table of various system outputs is there.

http://en.wikipedia.org/wiki/Uname

Documentation on the uname(2) system call and the struct utsname data structure may be found on a GNU system in the Texinfo documentation system in the “Platform Type” section under the “System Management” chapter. On a GNU system you can get there directly with this command line.

$ info libc 'Platform Type'

This most recent version is also available online.

http://www.gnu.org/software/libc/manual/html_node/Platform-Type.html