Luca's System Engineer Learning Path

Bash Scripting

  • Mar 08, 2023

Basics

What is Bash Scripting

Bash scripts tell bash what to do at what time. They are a simple text file containing a series of commands we want to automate running rather than running them, often with or to optional parameters provided when the script is launched.

Commands can be executed within a shell script just as they would on the command line (the script will also assume the permissions of the person executing the command)

We need to set the execute bit on first line of the file (shebang - #!):

  • It specifies which interpreter should be used for the commands that follow it.
  • Include the full path to the interpreter.

Script Locations

  • /usr/bin: System available scripts
  • /usr/local/bin: Scripts installed/created only for use on the local system
  • /home/user/bin: Scripts created by the user user and available only to that user

Note: When creating your own script, make sure it is set executable chmod 755 myscript.sh. This will make sure that everyone has execute permissions on the script.

Bash Configuration

.bash_profile .bashrc .bash_history .bash_logout
Only runs when a user logs in (configures the shell before you receive the initial command prompt) Only runs before the command prompt comes up or when a new bash instance is started What you have typed as this user in a bash instance Runs before exit command is used.
Used at user level Used at system level env | grep HISTCONTROL  
Runs .bashrc if it exists Runs /etc/bashrc to apply global bash config export HISTCONTROL=$HISTCONTROL:ignorespace to exclude commands starting with a space (useful for passwords).  

Parameters

These are strings, numbers, etc. that are passed to the environment, along with the command, that the script can use.

./myscript.sh /home/user/testfile.txt

Parameters are referred to by the number of the parameter that you are using: $[#].

Note: Parameter numbering starts with 1, parameters that do not exist will not error but will simply be interpreted as "empty".

Working with Special Characters

Common Special Characters Used in Bash

Character Function
" " or ' ' Denotes whitespace. Single quotes preserve literal meaning; double quotes allow substitutions.
$ Denotes an expansion (for use with variables, command substitution, arithmetic substitution, etc.)
\ Escape character. Used to remove “specialness” from a special character.
# Comments. Anything after this character isn’t interpreted.
= Assignment
[ ] or [[ ]] Test; evaluates for either true or false
! Negation
>>, >, < IO redirection
\| Pipe. Sends the output of one command to the input of another.
* or ? Globs (aka, wildcards).? is a wildcard for a single character.

Note also take a look at environment variables.

Implementing and/or Lists

  • && And lists: string of commands where the next command is only executed is the previous command exited with a status of zero.
  • || Or lists: string of commands where the next command is only executed is the previous command exited with a non-zero status.

Exit status

Number returned to the shell to indicate if the command / shell / script exited correctly or something else failed:

  • zero: script or program ran to completion and everything is fine –> no guarantee for succes however.
  • non-zero: based on script or program –> no standard

In bash scripts you can manually set an exit status when certain conditions are met in the script or when throwing errors.

The exit command allows you to return a non-zero error code to the environment which could then be read/logged/reacted to by other applications.

For example:

exit 66

Will return the error code 66 to the environment and a subsequent echo $? would return that value.

Redirecting I/O, Utility Commands, and Pipes

I/O Redirection

  stdout stdin stderr
Description All UNIX-like operating systems have “a bucket” where all output goes. Usually comes from entries made on the keyboard. Files and standard output can also provide input to another command. Typically written to the screen and acts like a bucket for all errors.
File handle 1 0 2

Special characters >, >> are used to intercept information before it gets to stdout or stdin.

The most common use of output redirection is collect the output of a command in a file. I/O redirection may be used to feed input to a command from a file or to send the output of a command to a file.

  • [COMMAND] < [FILE] - Read input from a file
  • [COMMAND] > [FILE] - Send output to a file
  • [COMMAND] >> [FILE] - Append output to a file

The tee command reads data from stdin, and writes that data to stdout and files.

[COMMAND] | tee <FILE_NAME>

Note: This command is useful for chaining together long commands and viewing output at various stages.

  [COMMAND] | tee phase1.log | [COMMAND] | tee phase2.log | [COMMAND] | tee phase3.log

The xargs command accepts input from stdin and other commands. Commonly used with the find command (but can be used with other commands as well - grep -l).

[COMMAND] | xargs [COMMAND]

read

Useful for reading input from a shell (and/or file); similar to the cat command.

Can be used to take input from the user.

For example:

read FIRSTNAME

Will prompt user for a FIRSTNAME and store it in that variable to be used later

Variables

What Do Variables Look Like and How Do We Use Them?

  1. Variables don’t have data types
  2. All variables start with $
  3. When setting the vars –> no $ is used

Flow Control

Conditionals - if / then / else / fi

Conditionals allows you the ability to control the conditions that the commands in your script will run.

Structure of an if conditional statement, in plain English: “if this then that else then that fi (end)”

Fall down of if/then/else if/else if/else/fi - Structure of a multiple statement and multiple test.

Used to test variable values and then act on them.

For example:

if [[ "$?" == "0" ]]

This will determine if the preceding command succeeded or failed.

Note: The variable $? is a special variable that can be tested immediately after any command. If the value is 0, then the command succeeded (did not error). Any other value, the command failed.

If statements can also be used to test commands and values directly.

For example:

if pgrep sshd

Will be true if the sshd process returns a PID (i.e., it is running).

test is a command to “test” various conditions with a parameter and value.

For example:

test -f /home/user/testfile.txt

Will test for the existence of a file called /home/user/testfile.txt

Combined with a conditional like if, can be an effective method to test for various conditions on strings, numbers and files:

  • -d [file]: file exists and is a directory
  • -f [file]: file exists and is a regular file
  • -h [file]: file exists and is a symbolic link
  • -r/w/x [file]: file exists and the user has read, write or execute permissions (as indicated by letter)
  • -s [file]: file exists and is larger than 0 bytes in size

For example:

if test -d /home/user

Will test to determine if the /home/user file is a directory or not.

Note: The test keyword can be omitted completely using the “square brackets” method if [ -f /home/user/testfile.txt ].

Note: A space must be after the opening bracket and before the closing bracket or you will receive "command not found" errors.

Testing strings:

  • Testing string lengths
    • -n [stringname]: The string is non-zero length
    • -z [stringname]: The string is zero length
  • Equality (=) tested with a single equal sign
    • For example: if [[ "$MYSTRING" = "Some Value" ]] Tests to see if the $MYSTRING variable was “Some Value”
  • Non-equality (!=)
    • For example: if [[ "$MYSTRING" != "Some Value" ]] Tests to see if the $MYSTRING variable was NOT “Some Value”

Testing integers:

  • -eq: Test for equality
  • -ne: Test for non-equality
  • -gt: Greater than
  • -ge: Greater than or equal to
  • -lt: Less than
  • -le: Less than or equal to

For example:

if [[ $MYAGE -gt 50 ]]

Will test to see if the variable $MYAGE is greater than 50

Multiple tests: Can be combined via the && (and) operator or the || (or) operator

For example:

if [[ $MYAGE -gt 50 && $MYAGE -lt 100 ]]

Will test to see if the variable $MYAGE is greater than 50 but less than 100 (which means I am old, but maybe not dead)

!

Often used to test whether something is NOT something

For example:

if !pgrep sshd

Will be true if the sshd process is NOT running

Looping

Allows you to determine how many times a particular part of your script will run

for/do/done - Loops over a fixed number of items

For example:

for city in Cincinnati Cleveland Columbus; do

Will execute the subsequent commands over the three cities in the list

while/done - Loops until the indicated condition is false, loops while true

For example:

while [[ $MYLOOP -lt 10 ]]

Will loop as long as the $MYLOOP variable is less than 10

until - Loops like while was written with a ! at the end (continues while false, stops when true)

seq - Helpful when you need to loop a known number of times

  • -w: Pad the output with leading zeros as the length of the number changes
  • [#] [#] [#]: beginning number, counting number, ending number

Note: When only two parameters are provided, they are beginning/ending

Scripting in Python

We’ve talked about why scripting is useful, and I’ve shown some simple examples in Bash.

But what happens when things get too complex for Bash?

Python is a very common scripting language. On RPM- based operating systems, yum is written in Python. The sosreport utility, which does a lot of complex information gathering, is written in Python as well.

You can write scripts in any language you want. Some languages work better than others for certain situations, bu for everyday automation tasks, it doesn’t really matter which language you use

Because hwclock.sh is an init script, and most init scripts do a lot of the same work, it’s easier to have all of those functions in a single file that gets sourced into all of the scripts.

Search