AwkUsageIn Bash Scripting

Page 1 of 67

How To Use awk In Bash Scripting How do I use awk pattern scanning and processing language under bash scripts? Can you provide a few examples? Awk is an excellent tool for building UNIX/Linux shell scripts. AWK is a programming language that is designed for processing text-based data, either in files or data streams, or using shell pipes. In other words you can combine awk with shell scripts or directly use at a shell prompt. Print a Text File awk '{ print }' /etc/passwd OR awk '{ print $0 }' /etc/passwd Print Specific Field Use : as the input field separator and print first field only i.e. usernames (will print the the first field. all other fields are ignored): awk -F':' '{ print $1 }' /etc/passwd Send output to sort command using a shell pipe: awk -F':' '{ print $1 }' /etc/passwd | sort Pattern Matching You can only print line of the file if pattern matched. For e.g. display all lines from Apache log file if HTTP error code is 500 (9th field logs status error code for each http request): awk '$9 == 500 { print $0}' /var/log/httpd/access.log The part outside the curly braces is called the "pattern", and the part inside is the "action". The comparison operators include the ones from C: == != < > = ?: If no pattern is given, then the action applies to all lines. If no action is given, then the entire line is printed. If "print" is used all by itself, the entire line is printed. Thus, the following are equivalent: awk '$9 == 500 ' /var/log/httpd/access.log awk '$9 == 500 {print} ' /var/log/httpd/access.log awk '$9 == 500 {print $0} ' /var/log/httpd/access.log Print Lines Containing tom, jerry AND vivek Print pattern possibly on separate lines: awk '/tom|jerry|vivek/' /etc/passwd Print 1st Line From File awk "NR==1{print;exit}" /etc/resolv.conf awk "NR==$line{print;exit}" /etc/resolv.conf Simply Arithmetic

Page 2 of 67

You get the sum of all the numbers in a column: awk '{total += $1} END {print total}' earnings.txt Shell cannot calculate with floating point numbers, but awk can: awk 'BEGIN {printf "%.3f\n", 2005.50 / 3}' Call AWK From Shell Script A shell script to list all IP addresses that accessing your website. This script use awk for processing log file and verification is done using shell script commands. #!/bin/bash d=$1 OUT=/tmp/spam.ip.$$ HTTPDLOG="/www/$d/var/log/httpd/access.log" [ $# -eq 0 ] && { echo "Usage: $0 domain-name"; exit 999; } if [ -f $HTTPDLOG ]; then awk '{print}' $HTTPDLOG >$OUT awk '{ print $1}' $OUT | sort -n | uniq -c | sort -n else echo "$HTTPDLOG not found. Make sure domain exists and setup correctly." fi /bin/rm -f $OUT AWK and Shell Functions Here is another example. chrootCpSupportFiles() find out the shared libraries required by each program (such as perl / php-cgi) or shared library specified on the command line and copy them to destination. This code calls awk to print selected fields from the ldd output: chrootCpSupportFiles() { # Set CHROOT directory name local BASE="$1" # JAIL ROOT local pFILE="$2" # copy bin file libs [ ! -d $BASE ] && mkdir -p $BASE || : FILES="$(ldd $pFILE | awk '{ print $3 }' |egrep -v ^'\(')" for i in $FILES do dcc="$(dirname $i)" [ ! -d $BASE$dcc ] && mkdir -p $BASE$dcc || : /bin/cp $i $BASE$dcc done sldl="$(ldd $pFILE | grep 'ld-linux' | awk '{ print $1}')" sldlsubdir="$(dirname $sldl)" if [ ! -f $BASE$sldl ]; then /bin/cp $sldl $BASE$sldlsubdir

Page 3 of 67

else : fi } This function can be called as follows: chrootCpSupportFiles /lighttpd-jail /usr/local/bin/php-cgi AWK and Shell Pipes List your top 10 favorite commands: history | awk '{print $2}' | sort | uniq -c | sort -rn | head Sample Output: 172 ls 144 cd 69 vi 62 grep 41 dsu 36 yum 29 tail 28 netstat 21 mysql 20 cat whois cyberciti.com | awk '/Domain Expiration Date:/ { print $6"-"$5"-"$9 }' Awk Program File You can put all awk commands in a file and call the same from a shell script using the following syntax: awk -f mypgoram.awk input.txt Awk in Shell Scripts - Passing Shell Variables TO Awk You can pass shell variables to awk using the -v option: n1=5 n2=10 echo | awk -v x=$n1 -v y=$n2 -f program.awk Assign the value n1 to the variable x, before execution of the program begins. Such variable values are available to the BEGIN block of an AWK program: BEGIN{ans=x+y} {print ans} END{}

simple awk tutorial

Page 4 of 67

why awk? awk is small, fast, and simple, unlike, say, perl. awk also has a clean comprehensible C-like input language, unlike, say, perl. And while it can't do everything you can do in perl, it can do most things that are actually text processing, and it's much easier to work with. what do you do? In its simplest usage awk is meant for processing column-oriented text data, such as tables, presented to it on standard input. The variables $1, $2, and so forth are the contents of the first, second, etc. column of the current input line. For example, to print the second column of a file, you might use the following simple awk script: awk < file '{ print $2 }' This means "on every line, print the second field". To print the second and third columns, you might use awk < file '{ print $2, $3 }' Input separator By default awk splits input lines into fields based on whitespace, that is, spaces and tabs. You can change this by using the -F option to awk and supplying another character. For instance, to print the home directories of all users on the system, you might do awk < /etc/passwd -F: '{ print $6 }' since the password file has fields delimited by colons and the home directory is the 6th field. Arithmetic Awk is a weakly typed language; variables can be either strings or numbers, depending on how they're referenced. All numbers are floating-point. So to implement the fahrenheit-to-celsius calculator, you might write awk '{ print ($1-32)*(5/9) }' which will convert fahrenheit temperatures provided on standard input to celsius until it gets an end-of-file. The selection of operators is basically the same as in C, although some of C's wilder constructs do not work. String concatenation is accomplished simply by writing two string expressions next to each other. '+' is always addition. Thus echo 5 4 | awk '{ print $1 + $2 }' prints 9, while echo 5 4 | awk '{ print $1 $2 }' prints 54. Note that echo 5 4 | awk '{ print $1, $2 }' prints "5 4".

Page 5 of 67

Variables awk has some built-in variables that are automatically set; $1 and so on are examples of these. The other builtin variables that are useful for beginners are generally NF, which holds the number of fields in the current input line ($NF gives the last field), and $0, which holds the entire current input line. You can make your own variables, with whatever names you like (except for reserved words in the awk language) just by using them. You do not have to declare variables. Variables that haven't been explicitly set to anything have the value "" as strings and 0 as numbers. For example, the following code prints the average of all the numbers on each line: awk '{ tot=0; for (i=1; i>:" printf("string\n") >> "/tmp/file"; Like the shell, the double angle brackets indicates output is appended to the file, instead of written to an empty file. Appending to the file does not delete the old contents. However, there is a subtle difference between AWK and the shell. Consider the shell program: #!/bin/sh while x=`line` do echo got $x >>/tmp/a echo got $x >/tmp/b done This will read standard input, and copy the standard input to files "/tmp/a" and "/tmp/b." File "/tmp/a" will grow larger, as information is always appended to the file. File "/tmp/b," however, will only contain one line. This happens because each time the shell see the ">" or ">>" characters, it opens the file for writing, choosing the truncate/create or appending option at that time. Now consider the equivalent AWK program: #!/usr/bin/awk -f { print $0 >>"/tmp/a" print $0 >"/tmp/b" } This behaves differently. AWK chooses the create/append option the first time a file is opened for writing. Afterwards, the use of ">" or ">>" is ignored. Unlike the shell, AWK copies all of standard input to file "/tmp/b." Instead of a string, some versions of AWK allow you to specify an expression: # [note to self] check this one - it might not work printf("string\n") > FILENAME ".out"; The following uses a string concatenation expression to illustrate this: #!/usr/bin/awk -f

Page 45 of 67

END { for (i=0;i "/tmp/a" i; } }

Click here to get file: awk_example12.awk This script never finishes, because AWK can have 10 additional files open, and NAWK can have 20. If you find this to be a problem, look into PERL. I hope this gives you the skill to make your AWK output picture perfect. AWK Numerical Functions In previous tutorials, I have shown how useful AWK is in manipulating information, and generating reports. When you add a few functions, AWK becomes even more, mmm, functional. There are three types of functions: numeric, string and whatever's left. Table9 lists all of the numeric functions: +----------------------------------+ | AWK Table 9 | +----------------------------------+ | Numeric Functions | |Name Function Variant | +----------------------------------+ |cos cosine AWK | |exp Exponent AWK | |int Integer AWK | |log Logarithm AWK | |sin Sine AWK | |sqrt Square Root AWK | |atan2 Arctangent NAWK |rand Random NAWK | |srand Seed Random NAWK | +----------------------------------+

|

Trigonometric Functions Oh joy. I bet millions, if not dozens, of my readers have been waiting for me to discuss trigonometry. Personally, I don't use trigonometry much at work, except when I go off on a tangent. Sorry about that. I don't know what came over me. I don't usually resort to puns. I'll write a note to myself, and after I sine the note, I'll have my boss cosine it. Now stop that! I hate arguing with myself. I always lose. Thinking about math I learned in the year 2 B.C. (Before Computers) seems to cause flashbacks of high school, pimples, and (shudder) times best left forgotten. The stress of remembering those days must have made me forget the standards I normally set for myself. Besides, no-one appreciates obtuse humor anyway, even if I find acute way to say it. I better change the subject fast. Combining humor and computers is a very serious matter.

Page 46 of 67

Here is a NAWK script that calculates the trigonometric functions for all degrees between 0 and 360. It also shows why there is no tangent, secant or cosecant function. (They aren't necessary). If you read the script, you will learn of some subtle differences between AWK and NAWK. All this in a thin veneer of demonstrating why we learned trigonometry in the first place. What more can you ask for? Oh, in case you are wondering, I wrote this in the month of December. #!/usr/bin/nawk -f # # A smattering of trigonometry... # # This AWK script plots the values from 0 to 360 # for the basic trigonometry functions # but first - a review: # # (Note to the editor - the following diagram assumes # a fixed width font, like Courier. # otherwise, the diagram looks very stupid, instead of slightly stupid) # # Assume the following right triangle # # Angle Y # # | # | # | # a | c # | # | # +------Angle X # b # # since the triangle is a right angle, then # X+Y=90 # # Basic Trigonometric Functions. If you know the length # of 2 sides, and the angles, you can find the length of the third side. # Also - if you know the length of the sides, you can calculate # the angles. # # The formulas are # # sine(X) = a/c # cosine(X) = b/c # tangent(X) = a/b # # reciprocal functions # cotangent(X) = b/a # secant(X) = c/b # cosecant(X) = c/a # # Example 1) # if an angle is 30, and the hypotenuse (c) is 10, then # a = sine(30) * 10 = 5

Page 47 of 67

# b = cosine(30) * 10 = 8.66 # # The second example will be more realistic: # # Suppose you are looking for a Christmas tree, and # while talking to your family, you smack into a tree # because your head was turned, and your kids were arguing over who # was going to put the first ornament on the tree. # # As you come to, you realize your feet are touching the trunk of the tree, # and your eyes are 6 feet from the bottom of your frostbitten toes. # While counting the stars that spin around your head, you also realize # the top of the tree is located at a 65 degree angle, relative to your eyes. # You suddenly realize the tree is 12.84 feet high! After all, # tangent(65 degrees) * 6 feet = 12.84 feet # All right, it isn't realistic. Not many people memorize the # tangent table, or can estimate angles that accurately. # I was telling the truth about the stars spinning around the head, however. # BEGIN { # assign a value for pi. PI=3.14159; # select an "Ed Sullivan" number - really really big BIG=999999; # pick two formats # Keep them close together, so when one column is made larger # the other column can be adjusted to be the same width fmt1="%7s %8s %8s %8s %10s %10s %10s %10sn"; # print out the title of each column fmt2="%7d %8.2f %8.2f %8.2f %10.2f %10.2f %10.2f %10.2fn"; # old AWK wants a backslash at the end of the next line # to continue the print statement # new AWK allows you to break the line into two, after a comma printf(fmt1,"Degrees","Radians","Cosine","Sine", "Tangent","Cotangent","Secant", "Cosecant"); for (i=0;i