Oracle 8i | Oracle 9i | Oracle 10g | Oracle 11g | Oracle 12c | Miscellaneous | PL/SQL | SQL | Oracle RAC | Oracle Apps | Linux

Linux Redirection and File Processing

This article covers basic input and output redirection and file processing on Linux, with specific reference to the information needed for the RHCSA EX200 and RHCE EX300 certification exams.

Remember, the exams are hands-on, so it doesn't matter which method you use to achieve the result, so long as the end product is correct.

Redirection

Many Linux commands produce output that is pushed to the screen, they are said to write to standard output (stdout). Standard output can be redirected to a file, other commands or even a device.

The ">" operator redirects standard output to a new file. If the specified file already exists it is overwritten.

# ls -al /tmp > /root/output.log

The ">>" operator is similar, creating a new file if one doesn't exist, or appending to the exist file if it already exists.

# ls -al /usr >> /root/output.log

In addition to standard output, standard error (stderr) can also be redirected. The following table shows variations of redirection for standard out and standard error.

Redirection Action
> filename Redirect standard out to a new file.
>> filename Append standard out to an existing file.
1> filename Redirect standard out to a new file.
1>> filename Append standard out to an existing file.
2> filename Redirect standard error to a new file.
2>> filename Append standard error to an existing file.
&> filename Redirect standard out and standard error to a new file. (Requires bash 3+)
&>> filename Append standard out and standard error to an existing file. (Requires bash 4+)
> filename 2>&1 Redirect standard out and standard error to a new file.
>> filename 2>&1 Append standard out and standard error to an existing file.

To throw away all standard output and standard error, redirect it to "/dev/null".

The "<" operator is used to redirect standard input. This is typically used to pass a file as standard input for a command. It is more common to see the "|" operator used for this.

command < filename

The "|" operator is used to link (pipe) commands together by passing the standard output (and optionally standard error) from one command and passing it as the standard input for the next command. You will see examples of this throughout this and many other Linux articles.

# ps -ef | grep root

cat

The cat command pushes the contents of a file out to standard output.

# cat /etc/group

less and more

The less and more commands are a way to deal with displaying large amounts of text on a command line screen. The less command displays one screens worth of text and allows you to scroll up and down using the arrow keys. The "q" key quits.

# ps -ef | less
# cat /var/log/messages | less

The more command shows one page of text at a time, then waits for the user to prompt for more. The return key scrolls down one line, while the space bar scrolls down one page.

# ps -ef | more
# cat /var/log/messages | more

head and tail

The head and tail commands allow you to focus on that start and end of large amounts of text by specifying the number of lines to display. The following command displays the first 5 lines of the text.

# head -5 /etc/passwd
# ps -ef | head -5

The following command display the last 5 lines of test.

# tail -5 /etc/passwd
# ps -ef | tail -5

The "tail -f" command is extremely useful for watching real-time writes to a file. When you run it, the shell will wait on writes to the specified file and display them as they happen. This is great for watching processes write to log files. Use "CTRL+C" to release control back to the shell.

# tail -f /var/log/messages

grep

The grep command allows you to search through a file or stream for a specific pattern. Any line of data containing that pattern is returned. The following commands use ps to return the current processes, then filter them using grep to return only those lines containing the word "root". The "-i" option makes the filter case insensitive.

# ps -ef | grep root
# ps -ef | grep -i ROOT

The grep command can also use regular expressions to search for more complex patterns. The following example searches for any process lines that contain both 'root' and 'sbin', separated by any number of characters.

# ps -ef | grep 'root.*sbin'

Both the man and info pages of the grep command contain explanations of regular expressions.

Check out the fgrep (grep -F) and egrep (grep -E) variants also.

sed

The sed command allows you to process the contents of a file or stream to produce amended output. Let's start with a simple search and replace.

# echo "This is my data." | sed 's/my/your/'
This is your data.
#

Use the "i" flag to make it case insensitive.

# echo "This is My data." | sed 's/my/your/'
This is My data.
# echo "This is My data." | sed 's/my/your/i'
This is your data.
#

Use the "g" flag to make the change global, for every occurrence of the word, not just the first.

# echo "This is My data. This is My data." | sed 's/my/your/i'
This is your data. This is My data.
# echo "This is My data. This is My data." | sed 's/my/your/gi'
This is your data. This is your data.
#

You can also use sed to omit lines. Some basic examples are shows below.

# # Remove the first line
# ps -ef | sed '1d'

# # Remove the first 5 lines (1-5)
# ps -ef | sed '1,5d'

# # Remove lines containing a string
# ps -ef | sed '/root/d'

# Only output lines containing the modified string.
# ps -ef | sed -n 's/root/banana/gp'

Whole books have been written about the sed command, so this is very much the tip of the iceberg, but these are the type of things I find myself using regularly.

awk

Like sed, awk can do incredibly complicated and powerful things, so a full explanation of it is well beyond the scope of this article. Instead I will focus on the one task I use it for all the time, which is selectively pulling columns from a file or stream.

The output from many commands are displayed in the form of columns. For example, the ps command.

# ps -ef | head -1
UID        PID  PPID  C STIME TTY          TIME CMD
#

Using awk we can selectively pull out columns from a file or stream and use them to build up a new string.

# ps -ef | head -1 | awk '{print $1 "," $2 "," $8}'
UID,PID,CMD
# ps -ef | head -1 | awk '{print "The user " $1 " has the process ID of " $2 " and is running command: " $8}'
The user UID has the process ID of PID and is running command: CMD
#

In the above example, I was limiting the input to awk to a single row, but removing the "| head -1" would produce output for all rows returned by the ps command.

To for better example, let's imagine we might want to create a new CSV file containing the output from the ps command, minus the first row.

# ps -ef | sed '1d' | awk '{print $1 "," $2 "," $3 "," $4 "," $5 "," $6 "," $7 "," $8 }' > /tmp/output.txt

The "/tmp/output.txt" file now contains the required information in CSV format.

wc

The wc command outputs the newline, word and byte count for the specified file or files. The output can be limited using the appropriate flag.

# wc /var/log/messages
  1227  14163 103672 /var/log/messages

# wc -l /var/log/messages
1227 /var/log/messages

# wc -w /var/log/messages
14163 /var/log/messages

# wc -c /var/log/messages
103672 /var/log/messages
#

For more information see:

Hope this helps. Regards Tim...

Back to the Top.