grep – finding patterns in your log files

grep is a very useful utility when working with the Linux file system.

It is one of many common Linux tools that support regular expressions such as vim, sed and awk, allowing you to home in on results of a particular string pattern contained within a file.

However even searching for simple patterns can prove to be very useful too. So don’t let the term “regular expression” put you off making use of it.

The syntax can be as simple as: grep patterntomatch filename

Here are some simple examples that I find particularly useful/interesting.

Having a look at the web server access logs for file types I know I don’t use, but might indicate signs of possible attack.

/var/log/apache2# grep .php access.log
104.224.15.126 – – [17/Dec/2014:02:22:53 +0400] “GET /dddd/ddd/dd.php HTTP/1.1” 404 527 “-” “-”
104.224.15.126 – – [17/Dec/2014:02:22:54 +0400] “GET /phpMyAdmin/scripts/setup.php HTTP/1.1” 404 527 “-” “-”
104.224.15.126 – – [17/Dec/2014:02:22:54 +0400] “GET /pma/scripts/setup.php HTTP/1.1” 404 527 “-” “-”
104.224.15.126 – – [17/Dec/2014:02:22:55 +0400] “GET /myadmin/scripts/setup.php HTTP/1.1” 404 527 “-” “-”
195.154.42.218 – – [17/Dec/2014:08:39:44 +0400] “GET /rgrg/rgr/rg.php HTTP/1.1” 404 527 “-” “-”
195.154.42.218 – – [17/Dec/2014:08:39:44 +0400] “GET /phpMyAdmin/scripts/setup.php HTTP/1.1” 404 527 “-” “-”
195.154.42.218 – – [17/Dec/2014:08:39:44 +0400] “GET /pma/scripts/setup.php HTTP/1.1” 404 527 “-” “-”
195.154.42.218 – – [17/Dec/2014:08:39:44 +0400] “GET /myadmin/scripts/setup.php HTTP/1.1” 404 527 “-” “-“

Above we can see a few results indicating that likely automated attempts are being made for detecting mis-configured or vulnerable versions of phpMyAdmin.

Again taking a look at the server’s logs for instances of requests for content that doesn’t exist can provide an interesting picture. Normal user activity will rarely include 404 errors if your content is broken link free, even if not, as a site administrator you should have a decent idea of what constitutes an unusual pattern and could be used anyway to identify/fix broken links in any case 🙂

/var/log/apache2# grep -v robots.txt *| grep 404

21.41.58.199 – – [15/Dec/2014:02:57:32 +0400] “GET /bvbv/bvb/bv.php HTTP/1.1” 404 527 “-” “-”
121.41.58.199 – – [15/Dec/2014:02:57:33 +0400] “GET /phpMyAdmin/scripts/setup.php HTTP/1.1” 404 527 “-” “-”
121.41.58.199 – – [15/Dec/2014:02:57:34 +0400] “GET /pma/scripts/setup.php HTTP/1.1” 404 527 “-” “-”
121.41.58.199 – – [15/Dec/2014:02:57:34 +0400] “GET /myadmin/scripts/setup.php HTTP/1.1” 404 527 “-” “-”
66.249.75.24 – – [15/Dec/2014:03:17:04 +0400] “GET /rvmgthgqv.html HTTP/1.1” 404 546 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
61.160.247.7 – – [15/Dec/2014:09:42:46 +0400] “GET /manager/html HTTP/1.1″ 404 508 “-” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET4.0C; .NET4.0E; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)”

The -v robots.txt has been used to remove all of the requests that include robots.txt as I know these will be in there and I don’t want to see them as I expect a 404 for any of these requests.

I have highlighted above an attempt to detect a Tomcat login page. Misconfigured Tomcat servers are often responsible for compromises in many environments due to the ability to deploy your own code if successfully compromised, extra win if configured to run as system or root as can sometimes be the case.

See the related metasploit modules for more info on these attacks.

Suppose your interested in determining where the IP that made the suspicious connection to your server is located:

whois 61.160.247.7 | grep country | sort -u
country: CN

Combining whois to lookup the IP record information, filtering that output with grep to return only lines with country in the response and performing a unique sort to get rid of the duplicates we can see its our friends from china who are apparently paying us a visit.

Particular flags I find most useful when combined with grep:

-v — Invert search, i.e. patterns that don’t match

-E — Regular expression

-i — Ignore case

-l — Print the name of the file that matches instead of the whole output

-n — include line number

Vim – VI improved

I think it was around 1998, that I was introduced to Linux for the very first time.

I am pretty sure it was SuSE 6.2, I had a college class at the time that involved installing two operating systems and making some pretty basic system configuration changes to secure a pass for the class.

Learning about various commands and tools like YAST, and thinking how funny are these Linux guys. Yet another setup tool, still makes me chuckle to myself.

I had really only ever installed Win 95 on the old Pentium MMX 200 at home and was really pleased to be learning something new, that hopefully might give me some skills to get a job.

We only had to install Windows 3.1 and SuSE in our groups of 3 and then make some changes and record the steps.

There wasn’t the hardware for a physical machine each and the idea of  virtual machines was certainly not something we had ever discussed. Not like now, when spinning up a VM is as normal a task as sending an email.

I recall, being intrigued at the time by vi.

VIM(1) VIM(1)

NAME
vim – Vi IMproved, a programmers text editor

I was amazed that people actually worked with such a tool, having to use various keys to navigate the text and insert, copy, paste etc. I could imagine that servers over the Internet with only shell access would all be configured in such a way.

Not realizing at the time how powerful an editor it actual was. I was struggling to just do some basic tasks with it.

Some 16 years later. I am still amazed at vi, I have become more comfortable with it, the more you use it, the easier it becomes, although I do still refer at times to my cheat sheets 😉

These days vi has been improved to vim, with various excellent new features. Using vi, is actually usually using vim.

Some very basics to get you going:

create a file:

vi filename

We insert some text by leaving command mode and entering INSERT mode i:

i

~
~
~
~
~
~
— INSERT — 0,1 All

We can then type in some text such as:

Monday Tuesday Wednesday Thursday Friday

We want to save, but can not do this in insert mode, so we need to return to command mode, we press esc key to do this. I sometimes just press it a couple of times, just to be sure I am not about to start inserting colons into my text, however you do only need to press it 1 time.

We then save the changes by entering :w (don’t forget the colon)

Monday
Tuesday
Wednesday
Thursday
Friday
~
~
“filename” [New] 5L, 41C written 5,1 All

We can do things like remove lines of text by using dd.

Place your cursor at the first line of Monday. Command mode gg will take you there quickly.

Press dd to remove the first line:

Tuesday
Wednesday
Thursday
Friday
~
~
1,1 All

If you’ve  made a mistake or decided you like Mondays you can press u (undo)

Monday
Tuesday
Wednesday
Thursday
Friday
~
~
~
1,1 All

We can search for strings within the file using /string

Monday
Tuesday
Wednesday
Thursday
Friday
~
~
~
/day 5,4 All

The cursor will move to the first instance of the string from were the position of the cursor is at the time you start the search. I suggest starting at the first line with gg then /day

We can replace all strings quickly that match a particular pattern, which is pretty useful feature:

Monday
Tuesday
Wednesday
Thursday
Friday
~
~
~
:%s/day/day evening/g

Changing the strings to the best time of any week day:

Monday evening
Tuesday evening
Wednesday evening
Thursday evening
Friday evening
~
~
~
5 substitutions on 5 lines 5,1 All

For more info and further examples of vim:

Monday evening
Tuesday evening
Wednesday evening
Thursday evening
Friday evening
~
~
:q!

To quit back to the command line.

And check out the vimtutor:

#man vimtutor

VIMTUTOR(1) VIMTUTOR(1)

NAME
vimtutor – the Vim tutor

SYNOPSIS
vimtutor [-g] [language]

DESCRIPTION
Vimtutor starts the Vim tutor. It copies the tutor file first, so that it can be modified without changing the original file.

The Vimtutor is useful for people that want to learn their first Vim commands.

Linux Man-Pages

Linux is great, but at times you can become a bit frustrated. Especially if you cant remember what you are doing at the command line or you don’t work with it that often.

For example:

You find yourself sitting at the command prompt and you want to make some changes to your Apache web server, but you have not done the task very often or it’s just been a while and as you’re getting a bit older, your mobile phone likely has greater memory than you.

You reach for the favorite search engine … WAIT.

All you should need is 1 command. After all this is Linux.

# man man

MAN(1) Manual pager utils MAN(1)

NAME
man – an interface to the on-line reference manuals

*** Snipped

For example, suppose I know I want to do something with Apache, you can search using keywords with the -k switch:

# man -k apache

a2dismod (8) – enable or disable an apache2 module
a2dissite (8) – enable or disable an apache2 site / virtual host
a2enmod (8) – enable or disable an apache2 module
a2ensite (8) – enable or disable an apache2 site / virtual host
ab (1) – Apache HTTP server benchmarking tool
apache2 (8) – Apache Hypertext Transfer Protocol Server
apache2ctl (8) – Apache HTTP server control interface
apachectl (8) – Apache HTTP server control interface
check_forensic (8) – tool to extract mod_log_forensic output from apache log files
DBI::ProfileDumper::Apache (3pm) – capture DBI profiling data from Apache/mod_perl
logresolve (1) – Resolve IP-addresses to hostnames in Apache log files
rotatelogs (8) – Piped logging program to rotate Apache logs

Ok, we can see the list of commands related to Apache. We can read the desciption and maybe that is enough of a reminder. But what do the numbers mean?

Ask man:

# man man-pages

**Snipped

Sections of the Manual Pages
The manual Sections are traditionally defined as follows:

1 Commands (Programs)
Those commands that can be executed by the user from within a shell.

2 System calls
Those functions which must be performed by the kernel.

3 Library calls
Most of the libc functions.

4 Special files (devices)
Files found in /dev.

5 File formats and conventions
The format for /etc/passwd and other human-readable files.

6 Games

7 Conventions and miscellaneous
Overviews of various topics, conventions and protocols, character set standards, and miscellaneous other things.

8 System management commands
Commands like mount(8), many of which only root can execute.

 

Turns out the numbers are sections of the man-pages. So this information might help you further to decide if the one you suspect is actually the command you are looking for. Great

Then you can dig deeper into the command, reading all of the useful information or just paying attention to the specific areas of interest:

# man a2dismod

***Snipped

DESCRIPTION
This manual page documents briefly the a2enmod and a2dismod commands.

a2enmod is a script that enables the specified module within the apache2 configuration. It does this by creating symlinks within /etc/apache2/mods-enabled. Likewise, a2dismod disables a module by
removing those symlinks. It is not an error to enable a module which is already enabled, or to disable one which is already disabled.

Note that many modules have, in addition to a .load file, an associated .conf file. Enabling the module puts the configuration directives in the .conf file as directives into the main server con‐
text of apache2

EXAMPLES
a2enmod imagemap
a2dismod mime_magic

Enables the mod_imagemap module, and disables the mod_mime_magic module.