Bash Guide
March 19, 2017 | Author: mxx_2012 | Category: N/A
Short Description
Download Bash Guide...
Description
Linux
Server Administration
Tech Tips
Web Development
Pentesting
Contact
SEARCH Search …
LINUX
Bash Script Step By Step, You Will Love It February 7, 2017
admin
3 Comments
Today we are going to talk about bash script or shell scripting actually, they are called shell scripts in general but we are going to call them bash scripts because we are going to use bash among the other Linux shells. There
Search
Linux
Server Administration
Tech Tips
Contact
Bash shebang Setting permission Print messages using variables Environment variables User variables Command substitution Math calculation if-then statement Nested if- Statement Numeric Comparisons String Comparisons File Comparisons
Web Development
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The key to bash script is the ability to enter multiple Commands and deal with the results from each command, even possibly passing the results of one command to another. The shell allows you to run multiple commands in a single step. If you want to run two commands together or more, you can type them on the same line, separated with a semicolon pwd ; whoami Actually what you have typed is a bash script!! This simple script uses just two bash shell commands. The pwd command runs first, displaying the current working directory followed by the output of the whoami command, showing who is currently logged in user. Using this technique, you can string together as many commands as you wish, maximum
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Well, that’s fine but it has a problem that you must enter the command at the command prompt every time you want to run it ok, what about we combine the commands into a file. And when we need to run those commands we run that file only. This is called the bash script Now create an empty file using touch command as we discussed that in a previous post about Linux commands. The first line we should define which shell we will use as we know there are many shells on Linux bash is one of them
Bash Script Shebang Here we will write bash script and in order to do that we will type #!/bin/bash In a normal bash script line, the pound sign (#) is used as a comment line which is not processed by the shell. However, the first line
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
And the shell commands are entered one per line followed by enter and you can write a comment by adding the pound sign at the beginning of the file like this
#!/bin/bash # This is a comment pwd whoami You can use the semicolon and put multiple commands on the same Line if you want to, but in bash script, you can list commands on separate lines, and this to make it simpler to read them later. The shell will process them any way.
Set Script Permission Save the file, you are almost finished. All you need to do now is to set that file to be executable in order to be able to run it
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
chmod +x ./myscript Then try run it by just type it in the shell ./myscript And Yes it is executed
Print Messages As we know from other posts, printing text is done by echo command So take your knowledge to bash script and apply it Now we edit our file and type this #!/bin/bash
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
whoami Look at the output
Perfect! Now we can run commands and display text using echo command If you don’t know echo command or how to edit a file I recommend you to view previous articles about basic Linux commands
Using Variables Variables allow you to store information in the bash script for use with other commands in the script. Running individual commands from the bash script is good, but this has its limitations. There are two types of variables you can use in your bash script
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
in your shell commands to process information. You can do this by using environment variables. We talked about environment variables on another post you can check it. #!/bin/bash # display user home echo "Home for the current user is: $HOME"
Notice that we were able to place the $HOME system variable in the double quotation marks in the first string, and the shell script still know it
What if we want to print the dollar sign itself? echo "I have $1 in my pocket" The script sees a dollar sign within quotes; it
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
By using escape characters back slash \ before the dollar sign like this echo "I have \$1 in my pocket" Now the bash script will print the dollar sign as it is
User variables In addition to the environment variables, a bash script allows you to set and use your own variables in the script. Variables defined in the shell script maintain their values till bash script execution finished. Like system variables, user variables can be referenced using the dollar sign #!/bin/bash # testing variables
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
chmod +x myscript ./myscript
Command substitution One of the best features of bash scripts is the ability to extract information from the output of a command and assign it to a variable so you can use that value anywhere in your script There are two ways to do that The backtick character ` The $() format Make sure when you type backtick character it is not the single quotation mark. You must surround the entire command line command with two backtick characters like this mydir=`pwd`
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
#!/bin/bash mydir=$(pwd) echo $mydir The output of the command will be stored in that variable called mydir.
Math calculation You can perform basic math calculations using $[] format #!/bin/bash var1=$(( 5 + 5 )) echo $var1 var2=$(( $var1 * 2 )) echo $var2 Just that easy
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Bash script requires some sort of logic flow control between the commands in the script. Like if the value is greater than 5 do that else do whatever you can imagine any login you want. The most basic structure of if-then statement is like this if command then commands fi and here is an example #!/bin/bash if pwd then echo "It works" fi
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
May be searching for a specific user in the users /etc/passwd and if it exists it prints that the user is present #!/bin/bash user=likegeeks if grep $user /etc/passwd then echo "The user $user Exists" fi
We use grep command to search for the user in /etc/passwd file. You can check our tutorial about basic Linux commands if you don’t know grep command. If the user exists the bash script will print the message.
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
if-then-else Statement The if-then-else statement looks like this if command then commands else commands fi if the first command runs and returns with a zero which means success it will not hit the commands after the else statement, otherwise, if the if statement returns non-zero which means the statement condition not correct, At this case the shell will hit the commands after else statement. #!/bin/bash user=anotherUser
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
echo "The user $user doesn’t exist" fi
We are doing good till now, keep moving Now, what if we have more else statements, like if user correct print this else if it is some other print this else print another thing? Well that is easy also we can achieve by nesting if statements like this if command1 then commands elif command2 then commands
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Pentesting
Contact
commands after it else if none of those return zero it will execute the last commands only. #!/bin/bash user=anotherUser if grep $user /etc/passwd then echo "The user $user Exists" elif ls /home echo "The user doesn’t exist but anyway there is a direc fi You can imagine any scenario here may be if the user doesn’t exist create it using the useradd command or do anything else.
Numeric Comparisons You can perform numeric comparison between two numeric values using numeric comparison checks as on this list n1 -eq n2 Checks if n1 is equal to n2
Linux
Server Administration
Tech Tips
Web Development
Contact
n1 -le n2 Checks if n1 is less than or equal to n2 n1 -lt n2 Checks if n1 is less than n2 n1 -ne n2 Checks if n1 is not equal to n2 As an example, we will try one of them and the rest is the same Note that the comparison statement is in square brackets as shown. #!/bin/bash val1=6 if [ $val1 -gt 5 ] then echo "The test value $value1 is greater than 5" else echo "The test value $value1 is not greater than 5" fi
The val1 is greater than 5 so it will run the first
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The comparison functions you can use to evaluate two string values are str1 = str2 Checks if str1 is the same as string str2 str1 != str2 Checks if str1 is not the same as str2 str1 < str2 Checks if str1 is less than str2 str1 > str2 Checks if str1 is greater than str2 -n str1 Checks if str1 has a length greater than zero -z str1 Checks if str1 has a length of zero We can apply string comparison on our example #!/bin/bash user ="likegeeks" if [$user = $USER] then
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
One tricky note about the greater than and less than for string comparisons MUST be escaped with the back slash because by just using the greater-than symbol itself in the script, no errors are generated, but the results are wrong. The script interpreted the greater-than symbol as an output redirection. So you should do it like that #!/bin/bash val1=text val2="another text" if [ $val1 \> "$val2" ] then echo "$val1 is greater than $val2" else echo "$val1 is less than $val2" fi
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Just little fix is to wrap the $vals with double quotation, forcing it to stay as one string like this #!/bin/bash val1=text val2="another text" if [ $val1 \> "$val2" ] then echo "$val1 is greater than $val2" else echo "$val1 is less than $val2" fi
Last tricky note about greater than and less than for string comparisons is when working with uppercase and lowercase letters. The sort command handles uppercase letters opposite to the way the test conditions consider them
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
then echo "$val1 is greater than $val2" else echo "$val1 is less than $val2" fi
sort myfile likegeeks Likegeeks
Capitalized letters are treated as less than lowercase letters in test comparisons. However, the sort command does exactly the opposite. String test comparisons use standard ASCII
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
defined for the system locale language settings.
File Comparisons This is the best and most powerful and most used comparison in bash scripting there are many file comparisons that you can do in bash script -d file Checks if file exists and is a directory -e file Checks if file exists -f file Checks if file exists and is a file -r file Checks if file exists and is readable -s file Checks if file exists and is not empty -w file Checks if file exists and is writable -x file Checks if file exists and is executable file1 -nt file2 Checks if file1 is newer than file2
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
-G file Checks if file exists and the default group is the same as the current user As they imply, you will never forget them. Let’s pick one of them and take it as an example
#!/bin/bash mydir=/home/likegeeks if [ -d $mydir ] then echo "The $mydir directory exists" cd $ mydir ls else echo "The $mydir directory does not exist" fi
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
There are some other advanced if-then features but let’s make it on another post. That’s for now. I hope you enjoy it and keep practicing more and more. Wait for another tutorial about bash scripting so stay tuned.
LINUX
Bash Scripting The Awesome Guide Part2 February 9, 2017
admin
4 Comments
In the previous post, we talked about how to
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
demonstrates for loop, while in bash scripts we will discuss the following: for command Iterating over simple values Iterating over complex values Reading values from a command The field separator Iterating over directory files for Command C-Style The while Command Nesting Loops Looping on File Data Controlling the Loop The break command
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
for Command The bash shell provides the for command to allow you to create a loop that iterates through a series of values. This is the basic format of the bash shell for command for var in list do commands done In each iteration, the variable var contains the current value in the list. The first iteration uses the first item in the list; the second iteration contains the second item, and so on until the end of the list items
Iterating over simple values The most basic use of the for command in bash
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
do echo The
$var item
done As you can see from the output the $var variable is changed on every loop cycle till the last item on the list.
Notice that the $var variable retained its value and allowed us to change the value and use it outside of the for command loop, like any variable.
Iterating over complex values Your list maybe contains some comma or two words but you want to deal with them as one item on the list.
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
echo "This is: $var" done We play nice till now, always we do. Just keep reading and practicing.
Reading values from a command Another way to a list is to use the output of a command. You use command substitution to execute any command that produces output. #!/bin/bash file="myfile" for var in $(cat $file) do echo " $var" done
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Notice that our file contains one word per line, not separated by spaces. The for command still iterates through the output of the cat command one line at a time, assuming that each line has one word. However, this doesn’t solve the problem of having spaces in data. If you list that contains words with spaces in it, the for command still takes each word as a separate value. There’s a reason for this, which we look at now.
The field separator The cause of this problem is the special environment variable IFS, called the internal field separator. By default, the bash shell considers the following characters as field
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
the data, it assumes that you’re starting a new data field in the list. To solve this problem, you can temporarily change the IFS environment variable values in your bash script suppose that you want to separate by new lines so it will be like this IFS=$'\n' So after you add this to your bash script it will ignore spaces and tabs and consider new lines as a separator. #!/bin/bash file="/etc/passwd" IFS=$'\n' for var in $(cat $file) do echo " $var" done You got it. Bash scripting is easy just little attention
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
In this case, the separator is colon like the case of /etc/passwd file which contains the user’s information you can assign it like this IFS=: How bash scripting is awesome?
Iterating over directory files One of the most common things when using for loop in bash scripting is iterate over files in a directory and deal with them. For example, we want to list the file inside /home directory so the code will be like this #!/bin/bash for file in /home/likegeeks/* do if [ -d "$file" ]
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
echo "$file is a file" fi done From the previous post, you should know the if statement and how to differentiate between files and folders, so if you don’t know I recommend you to review it bash script step by step.
Here we use wildcard character which is asterisk * and this is called in bash scripting file globbing which is a process of producing filenames automatically that matches the wildcard character in our case asterisk means
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Sure enough, it lists all files and directories in that folder
for Command C-Style If you know c language you may found that the for loop here is weird because you are familiar with this syntax for (i = 0; i < 10; i++) { printf(“number is %d\n”, i); } The bash scripting also supports a version of the for loop that looks similar to the C-style for loop with little difference here’s the syntax. for (( variable assignment ; condition ; iteration process )) So it looks like this
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
for (( i=1; i > myfile The output generated by pwd is appended to myfile without deleting the existed content.
Fine but if you try to redirect something and that command run into a problem
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Here there is no file called xfile on my PC and it generates error and the shell doesn’t redirect the error message to the output redirection file but the error message appeared on the screen and here is the third type of file descriptors
STDERR This file descriptor standard error output for the shell By default, the STDERR file descriptor points to the same place as the STDOUT file descriptor that’s why when an error occurs you see the error on the screen. So you need to redirect the errors to maybe log file or any else instead of printing it on the screen
Redirecting errors As we see the STDERR file descriptor is set to the value 2. We can redirect the errors by
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
cat ./myfile
As you see the error now is in the file and nothing on the screen
Redirecting errors and normal output In shell scripting, if you want to redirect both errors and the normal output, you need to precede each with the appropriate file descriptor for the data you want to redirect like this ls –l myfile xfile anotherfile 2> errorcontent 1> correctcontent
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
using the 2> symbol. If you want, you can redirect both STDERR and STDOUT output to the same output file use &> symbol like this ls –l myfile xfile anotherfile &> content
All error and standard output are redirected to file named content.
Redirecting Output in Scripts There are two methods for redirecting output in shell scripting Temporarily redirection
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
You can redirect an individual output line to STDERR. You just need to use the output redirection symbol to redirect the output to the STDERR file descriptor and you must precede the file descriptor number with an ampersand (&) like this #!/bin/bash echo "This is an error" >&2 echo "This is normal output"
So if we run it we will see both lines printed normally because as we know STDERR output to STDOUT if you redirect STDERR when running the script we should do it like this ./myscript 2> myfile
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Permanent redirections If you have lots of data that you’re redirecting in your script, it would be hard to redirect every echo statement. Instead, you can redirect to a specific file descriptor for the duration of the script by using the exec command. #!/bin/bash exec 1>outfile echo "This is a test of redirecting all output" echo "from a shell script to another file." echo "without having to redirect every line"
If we look at the file called outfile we will see the output of echo lines. You can also redirect the STDOUT in the middle
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
echo "now redirecting all output to another location" exec 1>myfile echo "This should go to the myfile file" echo "and this should go to the myerror file"
The exec command redirects any output going to STDERR to the file myerror. Then, the script uses the echo statement to display a few lines to STDOUT which is the screen. After that, the exec command is used again to redirect STDOUT to the myfile file and finally, we redirect the error from within the echo statement to go to STDERR which in this case is myerror file. Now you have all the ability to redirect all of your output to whatever you want Excellent!
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
exec command allows you to redirect STDIN from a file. exec 0< myfile This command tell the shell to take the input from the file called myfile instead of STDIN and here is an example #!/bin/bash exec 0< testfile count=1 while read line do echo "Line #$count: $line" count=$(( $count + 1 )) done
Shell scripting is easy.
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
file instead of the keyboard. Some Linux system administrators use this technique to read the log files for processing and we will discuss more ways to read the log on the upcoming posts more professionally.
Creating Your Own Redirection When you redirect input and output in your shell script, you’re not limited to the three default file descriptors. As I mentioned that you could have up to nine open file descriptors in the shell. The other six file descriptors from 3 through 8 and are available for you to use as either input or output redirection. You can assign any of these file descriptors to a file and then use them in your shell scripts You can assign a file descriptor for output by using the exec command and here’s an example how to do that
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
echo "And this should be back on the screen"
Creating input file descriptors You can redirect input file descriptors in shell scripting exactly the same way as output file descriptors. Save the STDIN file descriptor location to another file descriptor before redirecting it to a file. When you’re finished reading the file, you can restore STDIN to its original location #!/bin/bash exec 6&-
#!/bin/bash exec 3> myfile echo "This is a test line of data" >&3 exec 3>&echo "This won't work" >&3
As you can see it gives error bad file descriptor because it is no longer exist Note: careful in shell scripting when closing file descriptors. If you open the same output file later on in your shell script, the shell replaces
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
descriptors The lsof command lists all the open file descriptors on the entire Linux system On many Linux systems like Fedora, the lsof command is located in the /usr/sbin. This command is very useful actually it displays information about every file currently open on the Linux system. This includes all the processes running on background, as well as any user accounts logged into the system. This command has a lot of options so I think I will make a special post about it later but let’s take the important parameters we need -p, allows you to specify a process ID -d, allows you to specify the file descriptor numbers to display To get the current PID of the process, you can use the special environment variable $$, which
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
lsof -a -p $$ -d 0,1,2
The file type associated with STDIN, STDOUT, and STDERR is character mode. Because the STDIN, STDOUT, and STDERR file descriptors all point to the terminal, the name of the output file is the device name of the terminal. All three standard files are available for both reading and writing. Now, let’s look at the results of the lsof command from inside a script that’s opened a couple of alternative file descriptors #!/bin/bash exec 3> myfile1 exec 6> myfile2 exec 7< myfile3 lsof -a -p $$ -d 0,1,2,3,6,7
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
two for output (3 and 6) and one for input (7). And you can see the pathname for the files used in the file descriptors.
Suppressing Command Output Sometimes you don’t want to see any output this often occurs if you’re running a script as a background process (we will discuss how to make you shell script run in the background in the next posts) We redirect the output to the hole which is /dev/null For example, we can suppress errors like this ls -al badfile anotherfile 2> /dev/null And this idea is also used when you want to truncate a file without deleting it completely cat /dev/null > myfile
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
I hope you enjoy it; the next post will be how to control our running script and how to run your shell script in the background without interruption and how to pause them while they are running and some other cool stuff, Stay tuned. Thanks
LINUX
Linux Bash Scripting The Awesome Guide Part5 February 15, 2017
admin
0 Comments
On the last post, we’ve talked about input and
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
command line interface in real-time mode. This isn’t the only way to run Linux bash scripts in Linux. Our main points are: Linux signals Stop a process Pause a process Trapping signals Trapping the script exit Modifying or removing a trap Running scripts in background mode Running Scripts without a Hang-Up Viewing jobs Restarting stopped jobs
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Starting scripts with a new shell There are various control methods include sending signals to your script, modifying a script’s priority, and switching the run mode while a script is running. This post describes the different ways you can control your Linux bash scripts.
Linux signals There are more than 30 Linux signals that can be generated by the system and applications and this is the most common Linux system signals that you’ll run across in your Linux bash script writing Signal Value Description 1 SIGHUP Hangs up the process 2 SIGINT Interrupts the process
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
15 SIGTERM Terminates the process if possible 17 SIGSTOP Unconditionally stops, but doesn’t terminate, the process 18 SIGTSTP Stops or pauses the process, but doesn’t terminate 19 SIGCONT Continues a stopped process If the bash shell receives a SIGHUP signal, such as when you leave an interactive shell, it exits. Before it exits, it passes the SIGHUP signal to any processes started by the shell, including any running shell scripts With a SIGINT signal, the shell is just interrupted. The Linux kernel stops giving the shell processing time on the CPU. When this happens, the shell passes the SIGINT signal to any processes started by the shell to notify them.
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Generating signals The bash shell allows you to generate two basic Linux signals using key combinations on the keyboard. This feature comes in handy if you need to stop or pause a running bash script
Stop a process The Ctrl+C key combination generates a SIGINT signal and sends it to any processes currently running in the shell which simply stops the current process running in the shell. $ sleep 100 Ctrl+C
Pause a process The Ctrl+Z key combination generates a
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
$ sleep 100 Ctrl+Z
The number in the square brackets is the job number assigned by the shell. The shell refers to each process running in the shell as a job which is unique. It assigns the first started process job number 1, the second job number 2, and so on If you have a stopped job assigned to your shell the bash warns you if you try to exit the shell. You can view the stopped jobs using the ps command ps –l
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
In the S column (process state), shows the stopped job’s state as T. This indicates the command is either being traced or is stopped If you want to terminate a stopped job you can kill its process by using kill command I recommend you to review the basic Linux commands if you need more info about kill command kill processID
Trapping signals The trap command allows you to specify which Linux signals your shell script can watch for and intercept from the shell. If the script receives a signal listed in the trap command, it prevents it from being processed by the shell and instead handles it locally. So instead of allowing your Linux bash script to leave signals ungoverned, you can use trap command to do that.
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
#!/bin/bash trap "echo ' Trapped Ctrl-C'" SIGINT echo This is a test script count=1 while [ $count -le 10 ] do echo "Loop #$count" sleep 1 count=$(( $count + 1 )) done The trap command used in this example displays a simple text message each time it detects the SIGINT signal when hitting Ctrl+C.
Each time the Ctrl+C key combination was
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
You can trap them when the shell script exits; just add the EXIT signal to the trap command #!/bin/bash trap "echo Goodbye..." EXIT count=1 while [ $count -le 5 ] do echo "Loop #$count" sleep 1 count=$(( $count + 1 )) done
When the Linux bash script gets exit, the trap is triggered and the shell executes the echo command specified
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
trap You can reissue the trap command with new options #!/bin/bash trap "echo 'Ctrl-C is trapped.'" SIGINT count=1 while [ $count -le 5 ] do echo "Loop #$count" sleep 1 count=$(( $count + 1 )) done trap "echo ' I modified the trap!'" SIGINT count=1 while [ $count -le 5 ] do echo "Second Loop #$count" sleep 1 count=$(( $count + 1 )) done
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
After the signal trap is modified, the bash script manages the signal or signals differently. You can also remove a set trap. Simply add two dashes after the trap command #!/bin/bash trap "echo 'Ctrl-C is trapped.'" SIGINT count=1 while [ $count -le 5 ] do echo "Loop #$count" sleep 1 count=$(( $count + 1 )) done trap -- SIGINT echo "I just removed the trap" count=1 while [ $count -le 5 ] do
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
If a signal is received before the trap is removed, the script processes it per the original trap command $ ./myscript Crtl+C
The first Ctrl+C were used to attempt to terminate the script. Because the signal was received before the trap was removed, the script executed the echo command specified in the trap. After the script executed the trap removal, then Ctrl+C could terminate the bash script
Running Linux bash scripts in background
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
can’t do anything else in your terminal session. Fortunately, there’s a simple solution to that problem. If you see the output of the ps command you will see all the running processes in the background and not tied to the terminal. We can do the same just place ampersand symbol after the command #!/bin/bash count=1 while [ $count -le 10 ] do sleep 1 count=$(( $count + 1 )) done $ ./myscipt &
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
done. Notice that while the background process is running, it still uses your terminal monitor for STDOUT and STDERR messages so if the error occurs you will see the error message and normal output also.
If the terminal session exit, the background process also exit So what if you want to continue running even if you close the terminal?
Running Scripts without a Hang-Up You can run your Linux bash scripts in the background process even if you exit the
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
blocking any SIGHUP signals that are sent to the process. This prevents the process from exiting when you exit your terminal. $ nohup ./myscript &
The nohup command disassociates the process from the terminal, the process loses the STDOUT and STDERR output links. To accommodate any output generated by the command, the nohup command automatically redirects STDOUT and STDERR messages to a file, called nohup.out Note when running multiple commands from the same directory, because all the output is sent to the same nohup.out file
Viewing jobs The jobs command allows you to view the
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
do echo "Loop #$count" sleep 10 count=$(( $count + 1 )) done Then run it $ ./myscript Then stop it using the Ctrl+Z
Run the same bash script but in background using the ampersand symbol and to make life a little easier, I’m going to make the output of that script is redirected to a file so it doesn’t appear on the screen $ ./myscript > outfile &
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The jobs command shows both the stopped and the running jobs jobs –l -l parameter to view the process ID
Restarting stopped jobs To restart a job in background mode, use the bg command $ ./myscript Ten press Ctrl+z Now it is stopped $ bg
As you can see it is now running in background
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
To restart a job in foreground mode, use the fg command. $ fg 1
Scheduling a job The Linux system provides a couple of ways to run a bash script at a preselected time: the at command and the cron table The at command This is the format of the command at [-f filename] time The at command recognizes lots of different time formats A standard hour and minute, such as 10:15 An AM/PM indicator, such as 10:15PM A specific named time, such as now, noon, midnight
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
MM/DD/YY, or DD.MM.YY A text date, such as Jul 4 or Dec 25, with or without the year Now + 25 minutes 10:15PM tomorrow 10:15 + 7 days We don’t want to dig deep into the at command but for now, just make it simple and we will discuss it in detail in future posts. $ at -f ./myscript now
The –M parameter is to send output to e-mail if the system has e-mail and if not this will suppress the output of the at command To list the pending jobs use atq command $ atq
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
you can use the atrm command to remove a pending job by specifying the job number $ atrm 18
Scheduling scripts Using the at command to schedule a script to run at a preset time is great, but what if you need that script to run at the same time every day or once a week or once a month. The Linux system uses the crontab command to allow you to schedule jobs that need to run regularly. The crontab program runs in the background and checks special tables, called cron tables, for jobs that are scheduled to run To list an existing cron table, use the -l
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
minute,Hour, dayofmonth, month, and dayofweek So if you want to run a command at 10:30 on every day, you would use this cron table entry 30 10 * * * command The wildcard character (*) used in the dayofmonth, month, and dayofweek fields indicates that cron will execute the command every day of every month at 10:30. To specify a command to run at 4:30 PM every Monday, you would use the following 30 16 * * 1 command The day of the week start from 0 to 6 where 0 is Sunday and 6 is Saturday Here’s another example: to execute a command at 12 noon on the first day of every month, you would use the following format
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
the cron in great detail in future posts. To add entries to your cron table, use the -e parameter like this crontab –e Then type your command like the following 30 10 * * * /home/likegeeks/Desktop /myscript This will schedule our script to run at 10:30 every day Note sometimes you see error says Resource temporarily unavailable. All you have to do is this $ rm -f /var/run/crond.pid You should be root user Just that simple! You can use one of the pre-configured cron
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
/etc/cron.daily /etc/cron.weekly /etc/cron.monthly Just put your bash script file on any of those directories and it will run periodically.
Starting scripts with a new shell Remember from the previous posts we’ve talked about startup files I recommend you to review the previous posts to get the point. $HOME/.bash_profile $HOME/.bash_login $HOME/.profile Just place any scripts you want to run at login time in the first file listed. Ok but what about running our bash script
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
execute that command This for now, I hope you find the post useful. Thanks.
LINUX
Bash Scripting The Awesome Guide Part6 Bash Functions February 17, 2017
admin
2 Comments
Before we talk about bash functions let’s discuss this situation. When writing bash scripts, you’ll find yourself that you are using
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
refer to that block of code anywhere in your bash script without having to rewrite it. The bash shell provides a feature allowing you to do just that called Functions. Bash functions are blocks of script code that you assign a name to and reuse anywhere in your code. Anytime you need to use that block of code in your script, you simply use the function name you assigned it. We are going to talk about how to create your own bash functions and how to use them in other shell scripts. Our main points are: Creating a function Using functions Using the return command Using function output
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Recursive function Creating libraries Use bash functions from command line
Creating a function You can create a function like this functionName { } Or like this functionName() { } The parenthesis on the second way is used to pass values to the function from outside of it so these values can be used inside the function.
Using functions #!/bin/bash
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
while [ $count -le 3 ] do myfunc count=$(( $count + 1 )) done echo "This is the end of the loop" myfunc echo "End of the script" Here we’ve created a function called myfunc and in order to call it, we just type it’s name.
The function can be called many times as you want. Notice: If you attempt to use a function before it’s defined, you’ll get an error message #!/bin/bash
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
count=$(( $count + 1 )) done echo "This is the end of the loop" function myfunc { echo "This is an example of using a function" } echo "End of the script"
Another notice: bash function name must be unique, or you’ll have a problem. If you redefine a function, the new definition overrides the original function definition without any errors #!/bin/bash function myfunc { echo "The first function definition" } myfunc function myfunc {
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
As you see the second function definition takes control from the first one without any error so take care when defining functions.
Using the return command The return command allows you to specify a single integer value to define the function exit status. There are two ways of using return command; the first way is like this #!/bin/bash function myfunc { read -p "Enter a value: " value echo "adding value" return $(( $value + 10 )) }
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The myfunc function adds 10 to the value contained in the $value variable provided by the user input. It then returns the result using the return command, which the script displays using the $? Variable. If you execute any other commands before retrieving the value of the function, using the $? variable, the return value from the function is lost. Remember that the $? Variable returns the exit status of the last executed command. You cannot use this return value technique if you need to return either larger integer values or a string value.
Using function output The second way of returning a value from a bash function is to capture the output of a command to a shell variable; you can also capture the output of a function to a shell
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
function myfunc { read -p "Enter a value: " value echo $(( $value + 10 )) } result=$( myfunc) echo "The value is $result"
Passing parameters to a function We can deal with bash functions like small snippets that we can reuse and that’s ok but we need to make the function like an engine, we give it something and it returns another thing based on what we gave. Functions can use the standard parameter environment variables to represent any parameters passed to the function on the command line. For example, the name of the
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
parameters passed to the function. I recommend you to review the previous posts to empower your knowledge about them on Linux bash scripting. We pass parameters to functions on the same command line as the function, like this myfunc $val1 10 20 The following example shows you how to retrieve the parameter values using the parameter environment variables #!/bin/bash function addnum { if [ $# -eq 0 ] || [ $# -gt 2 ] then echo -1 elif [ $# -eq 1 ] then echo $(( $1 + $1 )) else
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
value=$(addnum 10 15) echo $value echo -n "Adding one number: " value=$(addnum 10) echo $value echo -n "Adding no numbers: " value=$(addnum) echo $value echo -n "Adding three numbers: " value=$(addnum 10 15 20) echo $value
The addnum function checks the number of parameters passed to it by the script. If there are no parameters, or if there are more than two parameters, addnum returns a value of -1. If there’s one parameter, addnum adds the parameter to itself for the result. If there are
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
values from the command line of the script. The following example fails #!/bin/bash function myfunc { echo $(( $1 + $2 )) } if [ $# -eq 2 ] then value=$( myfunc) echo "The result is $value" else echo "Usage: myfunc a b" fi
Instead, if you want to use those values in your bash function, you have to manually pass them when you call the function like this
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
if [ $# -eq 2 ] then value=$(myfunc $1 $2) echo "The result is $value" else echo "Usage: myfunc a b" fi
Now they are available for the function to use, just like any other parameter
Handling variables in bash functions Every variable we use has a scope, the scope is where the variable is visible Variables defined inside functions can have a different scope than regular variables.
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Local
Global variables Global variables are variables that are visible and valid anywhere in the bash script. If you define a global variable in the main section of a script, you can retrieve its value inside a function. The same, if you define a global variable inside a function, you can retrieve its value in the main section of the script. By default, any variables you define in the script are global variables. Variables defined outside of a function can be accessed inside the function without problems #!/bin/bash function myfunc { value=$(( $value + 10 )) } read -p "Enter a value: " value myfunc
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
When the variable is assigned a new value inside the function, that new value is still valid when the script references the variable as the above example the variable $value is changed inside the function. So how to overcome something like this; Use local variables
Local variables Any variables that the bash function uses internally can be declared as local variables. To do that, just use the local keyword in front of the variable like this local temp=$(( $value + 5 )) If a variable with the same name appears outside the function in the script, the shell keeps the two variable values separate. Now you can easily keep your function variables separate from your script variables #!/bin/bash
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
temp=4 myfunc echo "The temp from outside is $temp"
Now when you use the $temp variable inside the myfunc function, it doesn’t affect the value assigned to the $temp variable in the main script.
Passing arrays to functions The art of passing an array variable to a bash function can be confusing. If you try to pass the array variable as a single parameter, it doesn’t work #!/bin/bash function myfunc { echo "The parameters are: $@"
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
echo "The original array is: ${myarray[*]}" myfunc $myarray
If you try using the array variable as a function parameter, the function only picks up the first value of the array variable To solve this problem, you must disassemble the array variable into its individual values and use the values as function parameters. Inside the function, you can reassemble all the parameters into a new array variable like this. #!/bin/bash function myfunc { local newarray newarray=("$@") echo "The new array value is: ${newarray[*]}" } myarray=(1 2 3 4 5)
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The function rebuilds the array variable from the command line parameters
Recursive function This feature enables the function to call itself from within the function itself The classic example of a recursive function is calculating factorials. A factorial of a number is the value of the preceding numbers multiplied with the number. Thus, to find the factorial of 5, you’d perform the following equation. Thus the factorial of 5 is: 5! = 1 * 2 * 3 * 4 * 5 Using recursion, the equation is reduced down to the following format x! = x * (x-1)! So to write factorial function using bash scripting it will be like this
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
echo 1 else local temp=$(( $1 - 1 )) local result=$(factorial $temp) echo $(( $result * $1 )) fi } read -p "Enter value: " value result=$(factorial $value) echo "The factorial of $value is: $result"
Using recursive bash functions is so easy!
Creating libraries Now we know how to write functions and how to call them but what if you want to use these bash functions or blocks of code on different bash script files without copying and pasting it over your files.
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The key to using function libraries is the source command. You use the source command to run the library file script inside of your shell script. This makes the functions available to the script, without it, the function will not be visible in the scope of the bash script The source command has a shortcut alias, called the dot operator. To source a file in a shell script, you just need to add the following line: . ./myscript Let’s assume that we have a file called myfuncs and contains the following function addnum { echo $(( $1 + $2 )) } Now we will use it inside another bash script file like this #!/bin/bash
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Awesome!! We’ve used the bash functions inside myfuncs file inside our bash script file but what if we want to use those functions from our bash shell directly?
Use bash functions from command line Well, that is easy if you read the previous post which was about signals and jobs you will have idea that we can source our functions file in .bashrc file and hence we can use the functions directly from the bash shell. Cool Edit .bashrc file and add this line . /home/likegeeks/Desktop/myfuncs Just make sure you type the correct path and now from the shell when we type the following $ addnum 10 20
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
functions are automatically available for any bash scripts without sourcing wow!! That really cool right Note: you may need to logout and login to use the bash functions from the shell Another note: if you make your function name like the any of the built-in commands you will overwrite the default command so you should take care of that. With these examples, I finish my post today hope you like it Thank you.
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
LINUX
31+ Examples For Sed Linux Command In Text Manipulation February 19, 2017
admin
2 Comments
On the previous post we’ve talked about bash functions and how to use it from the command line and we’ve seen some other cool stuff I recommend you to review it, Today we will talk about a very useful tool for string manipulation called sed, sed Linux command is one of the most common tools that people use to work with text files like log files, configuration files, and other text files. If you perform any type of data manipulation in your bash scripts, you want to become familiar with the sed and gawk tools in this post we are going to focus on sed Linux command and see its ability to
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
extract some text, delete, modify or whatever. The Linux system provides some common tools for doing just that one of those tools is sed. We will discuss the 31+ examples with pictures to show the output of everyone. Our main points are: Understand sed command Using multiple sed commands in the command line Reading commands from a file Substituting flags Replacing characters Limiting sed Deleting lines Inserting and appending text
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Reading data from a file Useful examples
Understand sed Linux command The sed command is called a stream editor, it is an interactive text editor, such as nano, you interactively use keyboard commands to insert, delete, or replace text in the data. Sed Linux command edits a stream of data based on a set of rules you provide and this is the format of the sed command $ sed options file By default, the sed Linux command applies the specified commands to the STDIN. This allows you to pipe data directly to the sed editor for processing like this $ echo "This is a test" | sed 's/test /another test/'
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
between the forward slashes. In this example, the words another test were substituted for the word test so the result will be like this That’s the power of using the sed Linux command. The above example was a very basic example to demonstrate the tool. We can use sed Linux command to manipulate files as well. This is our file
$ sed 's/test/another test' ./myfile You’ll start seeing results before the sed editor completes processing the entire file because sed returns the data instantaneously awesome!
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
text file itself. It only sends the modified text to STDOUT. If you look at the text file, it still contains the original data. You can overwrite the file with the new content very easy if you follow our previous posts we talk about redirections
Using multiple sed Linux commands in the command line To execute more than one command from the sed command line, just use the -e option like this $ sed -e 's/This/That/; s/test/another test/' ./myfile
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Also, you can use single quotation to separate commands like this $ sed -e ' > s/This/That/ > s/test/another test/' myfile
The same result no big deal.
Reading commands from a file If you have lots of sed commands you want to process, it is often easier to just store them in a separate file. Use the -f option to specify the file in the sed command like this $ cat mycommands
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Substituting flags Look at the following example carefully $ cat myfile $ sed 's/test/another test/' myfile
The substitute command works fine in replacing text in multiple lines, but it replaces only the first occurrence in each line. To get the substitute command to work on different occurrences of the text, you must use a
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
There are four types of substitutions A number, indicating the pattern occurrence for which new text should be substituted g, indicating that new text should be substituted for all occurrences of the existing text p, indicating that the contents of the original line should be printed w file, which means to write the results of the substitution to a file The first type of substitution, you can specify which occurrence of the matching pattern the sed Linux command should substitute new text for $ sed 's/test/another test/2' myfile
As a result of specifying a 2 as the substitution flag, the sed Linux command replaces the
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
text $ sed 's/test/another test/g' myfile
The p substitution flag prints a line that contains a matching pattern in the substitute command which used with -n option suppresses output from the sed command so it produces output only for lines that have been modified by the substitute command $ cat myfile $ sed -n 's/test/another test/p' myfile
The w substitution flag produces the same
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The output of the sed command appears in STDOUT, but only the lines that include the matching pattern are stored in output file.
Replacing characters Suppose that you want to substitute the C shell for the bash shell in the /etc/passwd file, you’d have to do this $ sed 's/\/bin\/bash/\/bin\/csh/' /etc/passwd That looks confusing for some people because the forward slash is used as the string delimiter, you must use a backslash to escape it. Luckily there is another way to achieve that. The sed Linux command allows you to select a different character for the string delimiter in the
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The exclamation point is used for the string delimiter. It is now easier to read.
Limiting sed The commands you use in the sed command apply to all lines of the text data. If you want to apply a command only to a specific line or a group of lines, there are two forms A numeric range of lines A text pattern that filters out a line The address you specify in the command can be a single line number or a range of lines specified by a starting line number, a comma, and an ending line number $ sed '2s/test/another test/' myfile
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Also, we can start from a line to the end of the file $ sed '2,$s/test/another test/' myfile
You can specify a text pattern to filter lines for the sed command. The pattern is written like this $ sed '/likegeeks/s/bash/csh/' /etc/passwd
The command was applied only to the line with the matching text pattern.
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Deleting lines If you need to delete specific lines of text in a text stream, you can use the delete command Be careful with the delete command, because if you forget to include an addressing scheme, all the lines are deleted from the stream $ sed '3d' myfile
Here we delete the third line only from myfile $ sed '2,3d' myfile
Here we delete a range of lines the second and
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Here we delete from the third line to the end of the file Note that the sed command doesn’t touch the original file. Any lines you delete are only gone from the output only. $ sed '/test 1/d' myfile
Here we use pattern to delete the line if matched on the first line You can also delete a range of lines using two text patterns like the following $ sed '/second/,/fourth/d' myfile
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The first to the third line deleted.
Inserting and appending text The sed Linux command allows you to insert and append text lines to the data stream using the following commands The insert command (i) adds a new line before the specified line The append command (a) adds a new line after the specified line You must specify the line to insert or append the line to insert on a separate line by itself So you can’t use these commands on a single command line. $ echo "Another test" | sed 'i\First test '
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
While in append mode the text appears after the data stream text. This works well for adding text before or after the text in the data stream, but what about adding text in the middle? To insert or append data inside the data stream lines, you must specify where the sed command where you want the data to appear. You can match either a numeric line number or a text pattern, and of course, you cannot use a range of addresses. $ sed '2i\This is the inserted line.' myfile
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
And the appending goes the same way but look at the position of the appended text $ sed '2a\This is the appended line.' myfile
The difference is it places the new text line after the specified line number.
Modifying lines The change command allows you to change the contents of an entire line of text in the data stream. All you have to do is to specify the line that you want to change. $ sed '3c\This is a modified line.'
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
You can also use a text pattern or a regular expression and all lines match that pattern will be modified. $ sed '/This is/c This is a changed line of text.' myfile
Transforming characters The transform command (y) works on a single character like this. $ sed 'y/123/567/' myfile
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
You can’t limit the transformation to a specific occurrence of the character.
Printing line numbers The equal sign command prints the current line number for the line within the data stream. $ sed '=' myfile
The sed editor prints the line number before the actual line of text nothing fancy here. However by using –n combined with the equal sign the sed command display the line number that contains the matching text pattern only. $ sed -n '/test/=' myfile
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
We’ve seen how to insert and append data to the data stream. Now we will read data from a file. The read command (r) allows you to insert data contained in a separate file. You can only specify a single line number or text pattern address. The sed Linux command inserts the text from the file after the address specified. $ cat newfile $ sed '3r newfile' myfile
The file called newfile content is just inserted after the third line as expected.
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Cool right?
Useful examples We can use the read command is to use it in conjunction with a delete command to replace a placeholder in a file with data from another file. Suppose we have the following file called newfile The word DATA in the file is a placeholder for a real content which is stored on another file called data. We will replace it with the actual content $ Sed '/DATA>/ {
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Now the placeholder text is replaced with the data file content. Super cool!! This is just a very small intro about sed command. Actually, sed Linux command is another world by itself. We can spend weeks to cover sed and its uses but you can use your mind with our previous posts to produce something great. As I said before the only limitation is your imagination. I hope you enjoy what’ve introduced today about the string manipulation using sed Linux command. This is just a beginning for sed command I will
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
LINUX
30 Examples For Awk Command In Text Processing February 21, 2017
admin
3 Comments
On the previous post we’ve talked about sed Linux command and we’ve seen many examples of using it in text processing and how it is good in this, nobody can deny that sed is very handy tool but it has some limitations, sometimes you need a more advanced tool for manipulating data, one that provides a more programming-like environment giving you more control to modify data in a file more robust. This
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
than the sed editor by providing a programming language instead of just editor commands. Within the awk programming language, you can do the following Define variables to store data. Use arithmetic and string operators to operate on data. Use structured programming concepts and control flow, such as if-then statements and loops, to add logic to your text processing. Generate formatted reports Actually generating formatted reports comes very handy when working with log files contain hundreds or maybe millions of lines and output a readable report that you can benefit from. our main points are: command options Reading the program script from the command line
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Running scripts before processing data Running scripts after processing data Built-in variables Data variables User defined variables Structured Commands Formatted Printing Built-In Functions User Defined Functions
awk command options The awk command has a basic format as follows $ awk options program file And those are some of the options for awk
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
-f file specifies a file name to read the program from -v var=value Defines a variable and default value used in the awk command –mf N specifies the maximum number of fields to process in the data file –mr N Specifies the maximum record size in the data file -W keyword Specifies the compatibility mode or warning level for awk The real power of awk is in the program script. You can write scripts to read the data within a text line and then manipulate and display the data to create any type of output report.
Reading the program script from the command line awk program script is defined by opening and
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
like this $ awk '{print "Welcome to awk command tutorial"}' If you run this command nothing will happen!! And that because no filename was defined in the command line. The awk command retrieves data from STDIN. When you run the program, it just waits for text to come in via STDIN If you type a line of text and press the Enter, the awk command runs the text through the program script. Just like the sed editor, the awk command executes the program script on each line of text available in the data stream. Because the program script is set to display a fixed text string, you get the same text output. $ awk '{print "Welcome to awk command tutorial "}'
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
string we provide. To terminate the program we have to send End-of-File (EOF) character. The Ctrl+D key combination generates an EOF character in bash. Maybe you disappointed by this example but wait for the awesomeness.
Using data field variables One of the primary features of awk is its ability to manipulate data in the text file. It does this by automatically assigning a variable to each data element in a line. By default, awk assigns the following variables to each data field it detects in the line of text $0 represents the entire line of text. $1 represents the first data field in the line of text. $2 represents the second data field in the line of text. $n represents the nth data field in the line of text.
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Look at the following file and see how awk deal with it $ awk '{print $1}' myfile
This command uses the $1 field variable to display only the first data field for each line of text. Sometimes the separator in some files is not space or a tab but something else. You can specify it using –F option $ awk -F: '{print $1}' /etc/passwd
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
This command displays the first data field in the passwd file. Because the /etc/passwd file uses a colon to separate the data fields.
Using multiple commands Any programming language wouldn’t be very useful if you could only execute one command. The awk programming language allows you to combine commands into a normal program. To use multiple commands on the command line, just place a semicolon between each command $ echo "My name is Tom" | awk '{$4="Adam"; print $0}'
The first command assigns a value to the $4 field variable. The second command then prints
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
As with sed command, the awk command allows you to store your scripts in a file and refer to them in the command line with the –f option Our file contains this script {print $1 " has a home directory at " $6} $ awk -F: -f testfile /etc/passwd
Here we print the username which is the first field $1 and the home path which is the sixth field $6 from /etc/passwd and we specify the file that contains that script which is called myscipt with -f option and surely the separator
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
This is our file { text = " has a home directory at " print $1 text $6 } $ awk -F: -f testfile /etc/passwd
Here we define a variable that holds a text string used in the print command.
Running scripts before processing data Sometimes, you may need to run a script
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
It forces awk to execute the script specified after the BEGIN keyword and before awk reads the data $ awk 'BEGIN {print "Hello World!"}' Let’s apply it to something we can see the result $ awk 'BEGIN {print "The File Contents:"} {print $0}' myfile
Now after awk command executes the BEGIN script, it uses the second script to process any file data. Be careful when doing this; both of the scripts are still considered one text string on the awk command line. You need to place your single quotation marks accordingly.
Running scripts after
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
$ awk 'BEGIN {print "The File Contents:"} {print $0} END {print "End of File"}' myfile
When the awk command is finished printing the file contents, it executes the commands in the END script. This is useful to use to add the footer as an example. We can put all these elements together into a nice little script file BEGIN { print "The latest list of users and shells" print " UserName \t HomePath" print "-------- \t -------" FS=":" }
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
print "The end" } This script uses the BEGIN script to create a header section for the report. It also defines the file separator FS and prints the footer at the end. Then we use the file $ awk -f myscript
/etc/passwd
This gives you a small taste of the power available when you use simple awk scripts.
Built-in variables The awk command uses built-in variables to reference specific features within the program data
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
But those are not the only variables, there are more built-in variables The following list is some of the built-in variables that awk command use: FIELDWIDTHS A space-separated list of numbers defining the exact width (in spaces) of each data field FS Input field separator character RS Input record separator character OFS Output field separator character ORS Output record separator character By default, awk sets the OFS variable to a space, By setting the OFS variable, you can use any string to separate data fields in the output $ awk 'BEGIN{FS=":"; OFS="-"} {print $1,$6,$7}' /etc/passwd
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The FIELDWIDTHS variable allows you to read records without using a field separator character. In some situations, instead of using a field separator, data is placed in specific columns within the record. In these instances, you must set the FIELDWIDTHS variable to match the layout of the data in the records After you set the FIELDWIDTHS variable, awk ignores the FS and calculates data fields based on the provided field width sizes Suppose we have this content 1235.9652147.91 927-8.365217.27 36257.8157492.5 $ awk 'BEGIN{FIELDWIDTHS="3 5 2 5"}{print $1,$2,$3,$4}'
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
defines four data fields, and awk command parses the data record accordingly. The string of numbers in each record is split based on the defined field width values The RS and ORS variables define how your awk command handles records in the data. By default, awk sets the RS and ORS variables to the newline character which means that each new line of text in the input data stream is a new record Sometimes, you run into situations where data fields are spread across multiple lines in the data stream Like an address and phone number, each on a separate line Person Name 123 High Street (222) 466-1234
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
and RS variable values, awk reads each line as a separate record and interprets each space in the record as a field separator. This is not what you want To solve this problem, you need to set the FS variable to the newline character. This indicates that each line in the data is a separate field and all the data on a line belongs to the data field and set the RS variable to an empty string. The awk command interprets each blank line as a record separator $ awk 'BEGIN{FS="\n"; RS=""} {print $1,$3}' addresses
Awesome! The awk command interpreted each line in the file as a data field and the blank lines as record separators.
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
extract information from the shell environment ARGC The number of command line parameters present ARGIND The index in ARGV of the current file being processed ARGV An array of command line parameters ENVIRON An associative array of the current shell environment variables and their values ERRNO The system error if an error occurs when reading or closing input files FILENAME The filename of the data file used for input to the awk command FNR The current record number in the data file IGNORECASE If set to a non-zero value ignores the case of characters in strings used in the awk command
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
You should recognize a few of these variables from previous posts about shell scripting series The ARGC and ARGV variables allow you to retrieve the number of command line parameters This can be a little tricky because awk command doesn’t count the script as part of the command line parameters $ awk 'BEGIN{print ARGC,ARGV[1]}' myfile
The ENVIRON variable uses an associative array to retrieve shell environment variables like this. $ awk ' BEGIN{ print ENVIRON["HOME"] print ENVIRON["PATH"]
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
You can use shell variables without ENVIRON variables like this $ echo | awk -v home=$HOME '{print "My home is " home}'
The NF variable allows you to specify the last data field in the record without having to know its position $ awk 'BEGIN{FS=":"; OFS=":"} {print $1,$NF}' /etc/passwd
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The FNR and NR variables are similar to each other but slightly different. The FNR variable contains the number of records processed in the current data file. The NR variable contains the total number of records processed. Let’s take a look at those two examples to illustrate the difference $ awk 'BEGIN{FS=","}{print $1,"FNR="FNR}' myfile myfile
In this example, the awk command defines two input files. (It specifies the same input file twice.) The script prints the first data field value and the current value of the FNR variable. Notice that the FNR value was reset to 1 when the awk command processed the second data
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
$ BEGIN {FS=","} {print $1,"FNR="FNR,"NR="NR} END{print "There were",NR,"records processed"}'
The FNR variable value was reset when awk processed the second data file, but the NR variable maintained its count into the second data file.
User defined variables Like any other programming language, awk allows you to define your own variables for use within the script. awk user-defined variable name can be any number of letters, digits, and underscores, but
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
$ BEGIN{ test="This is a test" print test }'
Structured Commands The awk programming language supports the standard if-then-else format of the if statement. You must define a condition for the if statement to evaluate, enclosed in parentheses. Here is an example testfile contains the following 10 15
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
$ awk '{if ($1 > 20) print $1}' testfile
Just that simple You can execute multiple statements in the if statement, you must enclose them with braces $ awk '{ if ($1 > 20) { x = $1 * 2 print x } }' testfile
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
{ x = $1 * 2 print x } else { x = $1 / 2 print x }}' testfile
You can use the else clause on a single line, but you must use a semicolon after the if statement
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The while loop allows you to iterate over a set of data, checking a condition that stops the iteration cat myfile 124 127 130 112 142 135 175 158 245 $ awk '{ total = 0 i = 1 while (i < 4) { total += $i i++ } avg = total / 3 print "Average:",avg }' testfile
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The while statement iterates through the data fields in the record, adding each value to the total variable and incrementing the counter variable i. When the counter value is equal to 4, the while condition becomes FALSE, and the loop terminates, dropping through to the next statement in the script. That statement calculates the average and prints the average. The awk programming language supports using the break and continue statements in while loops, allowing you to jump out of the middle of the loop $ awk '{ total = 0 i = 1 while (i < 4) {
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
} avg = total / 2 print "The average of the first two elements is:",avg }' testfile
The for loop The for loop is a common method used in many programming languages for looping. The awk programming language supports for loops $ awk '{ total = 0 for (i = 1; i < 4; i++) {
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
}' testfile
By defining the iteration counter in the for loop, you don’t have to worry about incrementing it yourself as you did when using the while statement.
Formatted Printing The printf command in awk allows you to specify detailed instructions on how to display data. It specifies exactly how the formatted output should appear, using both text elements and format specifiers. A format specifier is a special code that indicates what type of variable is displayed and how to display it. The awk command uses each format specifier as a
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The format specifiers use the following format %[modifier]control-letter This list is the format specifiers you can use with printf: c Displays a number as an ASCII character d Displays an integer value i Displays an integer value (same as d) e Displays a number in scientific notation f Displays a floating-point value g Displays either scientific notation or floating point, whichever is shorter o Displays an octal value s Displays a text string
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
}'
Here as an example, we display a large value using scientific notation %e. We are not going to try every format specifier. You know the concept
Built-In Functions The awk programming language provides quite a few built-in functions that perform mathematical, string, and time functions. You can utilize these functions in your awk scripts
Mathematical functions If you love math, those are some of the mathematical functions you can use with awk cos(x) The cosine of x, with x specified in
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
toward 0 log(x) The natural logarithm of x rand() A random floating point value larger than 0 and less than 1 sin(x) The sine of x, with x specified in radians sqrt(x) The square root of x and they can be used normally $ awk 'BEGIN{x=exp(5); print x}'
String functions There are many string functions you can check the list but we will examine one of them as an example and the rest is the same $ awk 'BEGIN{x = "likegeeks"; print
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
The function toupper convert case to upper case for the string passed.
User Defined Functions You can create your own functions for use in awk scripts, just define them and use them $ awk ' function myprint() { printf "The user %s has home path at %s\n", $1,$6 } BEGIN{FS=":"} { myprint() }' /etc/passwd
Pentesting
Linux
Server Administration
Tech Tips
Web Development
Contact
Here we define a function called myprint then we use it in out script to print output using printf built-in function. With this example, I finish my post today hope you like it. Thank you
Admin https://likegeeks.com
RELATED ARTICLES
LINUX
LINUX
LINUX
Pentesting
View more...
Comments
