Shell Scripting

Overview

Scripts and automation are a critical part of system and network administration. It is much preferable to complete a task with a script as opposed to doing it manually:

1. It allows you to ensure the task is done the same way each time.
2. It decreases the chance of human error. If you repeat the same manual steps multiple times, you'll eventually make a mistake doing so.
3. They serve as documentation. If you want to know the steps taken for various administrative tasks, the script provides them.
4. They save time and effort.

A script is executed by some shell or interpreter. The default shell on modern Linux systems is Bash, but there are several others. Some of the more prominent or historically relevant of which are:

Scripts run in shells like this are called shell scripts. We can also write scripts for programming language interpreters such as in Python, Perl, Ruby, etc. The biggest difference between shell scripts and scripts for these languages is that shell scripts can invoke commands (such as ls, cp, rm, du, etc). simply by putting the command on a line. Scripts in programming languages like Python can run system commands, but it's not as easy to do so.

For that reason, scripts which invoke many system commands are best written as shell scripts. However, scripts which involve more programming logic, and don't call upon as many commands are likely better written in a language like Python.

Bash Scripting

Bash scripting was covered in CPSC 225 in the following links:

1. Saving Commands in Shell Scripts
2. More Shell Scripting

These cover the most important aspects of shell scripting and should be reviewed.

Helpful Commands

There are a number of commands that are more useful when used in scripts then in interactive use, or at least are more useful when combined together with other commands. Some of these are covered here.

The head and tail commands take input and give as output only the first few, or last few, lines respectively. By default the number of lines is 10, but this can be changed:

$ ps aux | head
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  24456 15128 ?        Ss   Mar02   0:19 /sbin/init
root           2  0.0  0.0      0     0 ?        S    Mar02   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        S    Mar02   0:00 [pool_workqueue_release]
root           4  0.0  0.0      0     0 ?        I<   Mar02   0:00 [kworker/R-kvfree_rcu_reclaim]
root           5  0.0  0.0      0     0 ?        I<   Mar02   0:00 [kworker/R-rcu_gp]
root           6  0.0  0.0      0     0 ?        I<   Mar02   0:00 [kworker/R-sync_wq]
root           7  0.0  0.0      0     0 ?        I<   Mar02   0:00 [kworker/R-slub_flushwq]
root           8  0.0  0.0      0     0 ?        I<   Mar02   0:00 [kworker/R-netns]
root          10  0.0  0.0      0     0 ?        I<   Mar02   0:00 [kworker/0:0H-events_highpri]

$ ps aux | tail -3
ifinlay    91031  0.1  0.0   9028  5876 pts/2    Ss   11:57   0:00 /bin/bash
ifinlay    91048  0.0  0.0   9668  4540 pts/2    R+   11:58   0:00 ps aux
ifinlay    91049  0.0  0.0   5620  2076 pts/2    S+   11:58   0:00 tail -5

The command sort sorts the lines it is given. By default the sort is lexicographic. We can also pass the -n option to get it to sort numerically instead.

For instance, to find the largest directories with the following:

$ du -d 1 | sort -n
48      ./par-java
204     ./raw2gba
700     ./gba-patcher
1028    ./png2gba
5888    ./GBA-Games
16852   ./gba-tileeditor
74512   ./devkitadv
99236   .

This command is often combined with the command uniq, which removes duplicate lines from its input, but only if they are consecutive. So if we want to remove duplicates from a list, the way to do it is to pipe through sort and then through uniq.

For example, to get a list of unique users logged into a system (awk is explained below):

$ who | awk '{print $1;}' | sort | uniq
ifinlay
mbremer
sjohnson

The wc command can be used for counting characters, lines, and words in its input. We can use it to, for example, get the number of unique users logged into a system:

$ who | awk '{print $1;}' | sort | uniq | wc -l
3

The awk command is used above to print only the first column of text (the user names). Awk is actually quite a complex command, being a mini-language of its own. The most common usage is to print specific columns of text like this, however.

The xargs program is an interesting one. It takes the output of a program and runs another program on each line of output. This is often combined with the find command. For example, the find command can be used to look for files over a certain size:

$ find . -size +50k
./build/distributions/chai-0.1.0.zip
./build/distributions/chai-0.1.0.tar
./build/libs/chai-0.1.0.jar
./build/classes/java/main/net/ianfinlayson/chai/ChaiParser.class

But it doesn't actually tell us how big the files are. To get that to happen, we want to run another command on each line of output, which is most easily done with xargs:

$ find . -size +50k | xargs ls -lh
-rw-rw-r-- 1 ifinlay ifinlay  15M Mar 11 13:47 ./build/distributions/chai-0.1.0.zip
-rw-rw-r-- 1 ifinlay ifinlay  16M Mar 11 13:47 ./build/distributions/chai-0.1.0.tar
-rw-rw-r-- 1 ifinlay ifinlay 194K Mar 11 13:47 ./build/libs/chai-0.1.0.jar
-rw-rw-r-- 1 ifinlay ifinlay  67K Mar 11 13:30 ./build/classes/java/main/net/ianfinlayson/chai/ChaiParser.class