The Unix Shell

Community

Software Carpentry

Author: agoucher

Introduction

Command-line user interfaces (CLUIs) still have a play an important role amongst today's graphical user interfaces (GUIs).

CLUIs are easier to create
Small tools, combined in many ways, can be very powerful
Remote interaction with Unix-based machines is often done through a CLUI

The most important command-line tool is the shell. It is responsibile for:

Reading commands from the keyboard
Executing those commands
Displaying the output

While Windows-based machines do have a shell (called the Windows PowerShell, this lesson focuses entirely on to not only survive, but thrive in the Unix Shell.

In order to complete the examples, you will need access to a BASH shell either in on a Unix machine or through something like Cygwin on Windows.

Navigation

A path is a description of how to find something on the filesystem. There are a couple characteristics of the Unix filesystem one must keep in mind when trying to navigate it:

The root directory is called / (slash)
Files and directories are case-sensitive
/home/hpotter is different than /home/HPotter is different than /home/HpOtTeR
Paths are absolute when they reference the location of a file starting from /
/home/hpotter/thesis/dada.txt
Paths are relative when they reference the location of a file from some other location
from /home/hpotter, the path to the 'Defense Against the Dark Arts' thesis is thesis/dada.txt (note the lack of the starting /)
The . (dot) directory is the current one
The .. (dot dot) directory is the directory one level up (also called the parent directory)

In order to successfully navigate the filesystem, you need to learn 3 commands:

pwd - (short for print working directory tells you where you currently are
ls - (for listing) without an optional directory lets you see what is in the current directory. With a directory it will give you the listing of that directory.
cd - (for change directory) will change your current directory to the one given in the command

These commands are shown here as executed at the unix prompt ($).
$ pwd
/home/hpotter/
$ ls
thesis quiddich griffendore
$ cd quiddich
$ ls
rules.txt fouls rankings.txt
$ ls fouls
blagging.txt blatching.txt blurting.txt bumphing.txt
cobbig.txt flacking.txt haversacking.txt quaffle-pocking.txt
snitchnip.txt stooging.txt
$ cd ..
$ pwd
/home/hpotter

Providing Program Options

Most programs either require extra information to run, or can have their behavior modified through the use of flags . By convention, flags for Unix tools being with a "-" followed by a single letter, however some newer programs accept "--" followed by a descriptive word.

To show directories with a trailing /

$ ls -F quiddich
rules.txt fouls/ rankings.txt

By default, ls does not show thing whose name begins with ., but it can be told to with the -a flag

$ ls -F quiddich
. .. rules.txt fouls rankings.txt

Flags can also be combined (-F and --classify are equivilant)

$ ls --classify quiddich
rules.txt fouls/ rankings.txt

Basic File and Directory Manipulation

Now that we know how to navigate directories and find out what files are available, we are now ready to start doing stuff to those files. Unix systems provide a number of useful tools which provide most of what you will need to do to files or directories.

cat - Concatenate and display text files
cp - Copy files and/or directories
diff - Show the difference between two text files
head - Displays the first few lines of a file
mkdir - Make directories
more - Page through a text file
mv - Move (rename) files and/or directories
rm - Remove files rmdir - Remove directories
sort - Sorts the lines of a text file and displays them
tail - Displays the last few lines of a file (opposite of head)
wc - Count lines, words or characters in a file

All these tools have a number of different flags which may vary from OS to OS. To see which are available to you and what they do you can use the man (short for manual) tool. It will display the man(ual) page for the tool given as an argument. For example

$ man ls

would display the man page for the ls command.

One thing that is beyond the scope of this lesson is how to actually edit a file. There are literally thousands of editors available, however only vi (or one of its derivatives) is universally available on all default unix installs.

When dealing with files and directories it is often handy to use a wildcard which are characters which have special meaning to the shell.

* matches zero or more characters

$ ls fouls/bl*ing
blagging.txt blatching.txt blurting.txt bumphing.txt

? matches any single character

$ ls stoogin?.txt
stooging.txt

Redirection

Every process (a running program) automatically has three connections to the outside world

Standard input (stdin) - connected to the keyboard
Standard output (stdout) - connected to the screen
Standard error (stderr) - also connected to the screen

You can, through what is called redirection, connect the shell to a file instead of it's default location for input and output. Note that redirection is done command-by-command, not permenantly.

command < input_file will read from the input file instead of the keyboard
command > output_file will write to a file instea of the screen
command < input_file > output_file will do both

A | (pipe, above the \ on the keyboard) is used when you want to connect more than one applications directly together. This allows you to merge a number of individual commands that might have used stdin or stdout redirection. Using pipes is faster to type and is more secure because of the lack of temporary files.

$ wc -w b*.txt | sort -n
204 blurting.txt
409 blagging.txt
942 bumphing.txt
1438 blatching.txt
2993 total

The Environment

The OS stores environment variables for every process. These environment variables

have a name which by convention is all upper case
have values which are always strings

To see what environment variables are current set and what their values are, you use the set command
$ set
BASH=/usr/bin/bash
BASH_VERSION='2.05b.0(1)-release'
COLUMNS=120
HISTFILE=/home/.bash_history
HISTFILESIZE=500
(and so on)

To get the value of a specific variable, you put a $ in front of the name. When used with the echo command you can see the value on the screen.
$ echo $USER
hpotter

Note: This next part is BASH specific. If using a shell other than BASH, consult the shell's documentation
Setting an environment variable is as easy as
$ SPELL1=Aparecium

$ SPELL2="Finite Incantatum"
Notice how the value needs to be in quotes if there is a space in it.

Even though we have set the value only the current shell process can access it.
$ SPELL=Aparecium
$ bash
$ echo $SPELL

$ exit

In order to make it available to other processes we must <b>export</b> it
$ SPELL=Aparecium
$ export SPELL
$ bash
$ echo $SPELL
Aparecium
$ exit

Typically, the setting and exporting of environment variables are combined into a single command
$ export SPELL=Aparecium
$ bash
$ echo $SPELL
Aparecium
$ exit

$PATH

The PATH environment variable is one of the more important ones set for each user; it is how the shell decides what program to run. The value of PATH is always a sequence of directories seperated by a :
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/opt/bin

The order of the directories matter. A lot. When the shell is looking for a program, it starts at the beginning of the list and looks in each directory for the program. When it finds one, it will run it. Even if it was not the specific instance you were looking for. For example, if the program wand was in both /usr/bin and /opt/bin
$ wand
would execute /usr/bin/wand, not /opt/bin/wand.

If you are ever confused about which version is being called, you can use the which ommand to determine it
$ which want
/usr/bin/wand

Should you really want to run the other version, you can specify the full path in the shell
$ /opt/bin/wand
POOF! A rabbit!

To run things in the current directory you can put . as the first item in your PATH
$ export PATH=.:$PATH

Ownership and Permissions

In a Unix environemnt, every user belongs to one or more groups. The groups command will show you which ones you are in. Every file is owned by both a particular user and a particular group. Permissions, read (r), write (w), and execute (x) can be independently set to a user, group, and others.

Read - can look at contents, but not modify them
Write - can modify contents
Execute - can run the file (e.g., it's a program)

Permissions are displayed as three rwx triples and can be seen by ls -l (along with other information about the files). So rw-rw-r-- means:

User and group can read and write
Everyone else can read, but not write
No one can execute

Permissions for directories follow the same pattern, with one exception. When a directory has the --x (not set, not set, execute) permission, a user can go into the directory, but they cannot see which files are in there nor can they create any new ones.

Eventually you will want to change the permissions of a files. To do this you use chmod command. chmod is best explained through example:

chmod u+x broom allows broom's owner to run it
chmod o-r notes.txt takes away the world's read permission for notes.txt

Advanced Tools

At this point you can not only survive but be productive in the Unix Shell. Here are other programs available from the shell which will further increase what you can do. For more information on each of these, check it's man page

du - Print the disk space used by files and directories
find - Find files with names that match patterns, that are of a certain age or size, etc.
grep - Print lines matching a pattern
gunzip - Uncompress a file
gzip - Compress a file
ps - Display running processes
tar - Archive files
who - See who is logged in
xargs - Execute a command for each line of input.