Here are a few tips I’ve learned over several years of heavy use of the bash shell. Almost none are critical, need-to-know shell techniques, but all have become essential parts of my day-to-day work.

hacks, programming, unix

^Replace^andExecute

I often find myself executing a “test” of a command before actually executing it. For example, the -n flag of rsync and cvs give you the output you would see were you to actually execute the command. Rather than retyping or manually deleting the flag from the command, I can quickly remove the -n flag and execute the command for real using this simple bash trick: ^-n^^. For example:

$ rsync -n ..... (test to see what will happen)
$ ^-n^^ (This strips -n from the previous command in the history and executes it)

The more general use of this trick is of the form ^this^that^, where “this” is the search pattern and “that” is the search pattern’s replacement on the previous command:

$ cp this.txt mydir/
$ ^this^that^
(executes: cp that.txt mydir/)

Keep in mind that this only replaces the first occurrence of the pattern in the previous command.

Finding Multiple Files

find is a must-have in the bash arsenal. Typing man find will give you the reference documentation you need once you forget all the options available to find. I won’t go into much detail (find is a powerful tool), but here are some examples I use often.

Find all files (-type f) ending with .html (-name) in the htdocs/ directory:

find htdocs/ -type f -name *.html

Find all files that have been modified within the last day (-mtime -1) starting in the current directory (represented by the period “.” character) which contain the case-insensitive phrase “index” (-iname *index*):

find . -type f -mtime -1 -iname *index*

Complicated unix commands like find are made more complicated by the problem of escaping in the bash shell. Characters like *, (, ), !, and ; have a special meaning for bash. To remove the meaning, we “escape” them with the backslash “ character.

Executing Commands on Multiple Files

The find command can be used as the argument of another command that expects a file or list of files. This can be done by placing the find command in a sub-shell by wrapping the command in a pair of back-ticks (the key above the TAB on most keyboards). Here’s an example using the bash for loop:

for i in `find . -iname *.html`; do
    cp $i $i.bak
done

find also has a built in method for executing a command on every file it finds: -exec. The command following the -exec flag is run on every file, and every instance of a brace pair {} is replaced by the absolute path of the file. This example recursively removes all jpeg files from the current directory tree:

find . -iname *.jpeg -exec rm {} \;

Searching your History

If you use a shell regularly, you will find yourself typing in similar commands. Long commands are no fun to type twice—thus the convenience of Bash’s history commands.

The most common of these is the up and down arrow, which allows you to move incrementally through your history. There are a slew of others, all useful in their own way. None, however, equals the simplicity and utility of the search.

Simply type <CTRL>-r and a search phrase, and bash will search backwards through your history, and find the first match. Hitting <CTRL>-r again in this mode finds the second match, and so forth. Hit <ENTER> to execute the command, and escape to edit it before execution.

It’s a thing of beauty.

History Replacement

The history variable !$ contains the last argument of the last command run. This is a definite time-saver. So a series of commands that once looked like:

$ mkdir long/directory/name
$ cd long/directory/name

Now becomes:

$ mkdir long/directory/name
$ cd !$

Simple, I know, but when your friends are seventy and crippled with arthritis, you’ll thank me for the extra second I saved you by not having to copy and paste from the previous line.

Setting Options In Your Scripts

If you ever want to add options (also called flags) to a script you’ve written, consider using the getopts command. Before I explain the getopts command, some explanation on scripts in general. Assume, for example, that I am writing a program called backup. It takes one file as an argument, and creates a copy of that file with the .bak extension. To run the program, I would type the command backup my_test_file and it would copy that file to my_test_file.bak. The program itself might look something like this:

#!/bin/bash

#make sure we have a file
if [[ ! -f “$1” ]]; then
    echo “Please specify a valid filename as an argument!”
    exit 1
fi

cp $1 $1.bak

$1 in this case is a positional parameter: the first argument given to the command is $1, the second is $2, etc. Collectively, you can refer to all of the positional parameters as $@, which allows you to operate on an arbitrary number of files. I’ll save that problem as an exercise for the reader.

Apart from being a complete waste of time, this program backup has no customization. What if I wanted to add a different backup extension? Say .backup instead of .bak? What if I didn’t want to backup a directory, just the files inside it? Unix solves this by adding flags or options to your scripts. These are arguments preceded by a dash (-). Some options are boolean: its either set or it isn’t. Others require an argument.

For example, we could add two options to our script: -h to display some help text, and -n extension for specifying a different extension than .bak. Thus, you might call your command like so:

backup -h -n “.newext” mytest_file

To put this in practice, your program would look like this:

#!/bin/bash

extension=“.bak”

#parse out options first
while getopts “hn:” my_opt; do
    case $my_opt in 
        h  ) display_help=1;;
        n  ) extension=$OPTARG;;
    esac
done
#this next fancy command makes sure
#that $1, $2, etc. are references to everything after your flags
shift $(($OPTIND - 1))

if [[ -n $display_help ]]; then
   echo “Help!”
   exit 1
fi

#check to make sure we have a file
if [[ ! -f “$1” ]]; then
    echo “Please specify a valid filename as an argument!”
    exit 1
fi

cp $1 $1$extension

The important part of this script is while getopts “hn:” .... This causes bash to loop through your options one at a time, saving the option to $my_opt each time through the loop. Any flag argument that follows (if specified) is stored in $OPTARG. You also have to specify which options you want getopts to look for. After the while getopts each flag is listed, with an optional colon (:) after the letter to specify if it has an argument or not. In our case, -n includes an extension.

There are (of course) other options to getopts. This should be enough to get you started.

Perl PIE

I tend not to use PERL much anymore. It continues, however, to be invaluable for its original design as a Practical Extraction and Reporting Language. Any time I need to do a global search and replace, for example, I turn to this command:

perl -pi -e 's{the_pattern}{the_replacement}g' <list of files>

I won’t go into detail on using regular expressions; that is the subject for a different post. Combined with the earlier hint on find, you can quickly do a global search and replace on any list of files. To replace all opening <b> tags with <strong> tags in html files only, the following works:

perl -pi -e 's{<b>}{<strong>}g' `find . -type -f -iname *.html`