Showing posts with label tooltime. Show all posts
Showing posts with label tooltime. Show all posts

Monday, October 31, 2011

Apache Output Rewrite/Filter

When performing migration or code upgrades on public facing website, it is a common practice to test the changes in a development environment.   Occasionally, due to various reasons, you are forced to reference the development instance of the site using the public domain name.

For example, if the public facing site in question is www.example.com, and you want to setup a development instance in your lab, but want to be able to reference the site using the public DNS record, you have essentially two options: (1) Modify your DNS resolver (2) Modify your 'host' file.

The DNS option requires more effort but it is a better option if you have multiple developers, all using the same DNS server.

Host file trick is often used when you do not have a split-horizon DNS option or you do not control your DNS server.  Host file trick is the easiest and the most commonly used trick for such needs, however it comes with a bit of risk.  There is always the risk that a developer forgot to add the entry in his/her host file, and accidentally made changes to the production instance.  One simple trick to address such accidents is to modify the development site's header image or something similar.  However, in some applications which make extensive use of caching, sometimes this trick fails or leads to confusion.

Recently, I found myself on one such project (Drupal CMS) where we required not just a development instance, but also a QA instance.  To make matters worse, the developers needed to jump between production, development and QA instances multiple times a day.  I wanted to figure out a way to modify the web content to reflect the environment (dev or QA) without modifying the application PHP code.  Drupal has 3rd party extensions which can address this need, but I wanted to find a solution which was independent of the application (Drupal) and the server-side scripting technology (PHP).  I needed to make the modification at Apache level.

After doing some research, I quickly discovered Apache mod_ext_filter, a standard Apache module.  mod_ext_filter met all my criteria:

  1. Web application independent
  2. Server-side scripting technology independent
  3. Flexible and simple
  4. No 3rd party Apache modules or modifications to existing modules
  5. Performance should be acceptable
If you know of a better way, than the one proposed below, to accomplish these goals, please post a comment.

mod_ext_filter presents a simple and familiar programming model for filters. With this module, a program which reads from stdin and writes to stdout (i.e., a Unix-style filter command) can be a filter for Apache. This filtering mechanism is much slower than using a filter which is specially written for the Apache API and runs inside of the Apache server process, but it does have the following benefits:
  • the programming model is much simpler
  • any programming/scripting language can be used, provided that it allows the program to read from standard input and write to standard output
  • existing programs can be used unmodified as Apache filters


To use mod_ext_filter, enable it in your httpd.conf file by adding the following line:
LoadModule setenvif_module modules/mod_setenvif.so
For my needs, my goal was to inject a line of text in the header and footer of each page to identify the environment (dev or QA).

Using ext_filter module, a custom stdout filter can be as simple as adding a call to 'sed' in the module configuration file (see "Using sed to replace text in the response" example).  The problem with this approach is that you need to restart Apache every time you made a change to your filter, which can get annoying very quickly.
A better approach is to call an external script, which can be written in any language of choice.  In addition to not having to restart Apache, this approach has the advantage of allowing for easier troubleshooting.

Here is my /etc/httpd/conf.d/ext_filter.conf calling an external filter.sh script.
ExtFilterDefine banner mode=output intype=text/html \
cmd="/bin/sh /var/www/filter.sh"
 
<Location />
        SetOutputFilter banner
</Location>
 
 In my filter.sh, I decided to perform a simple find/replace on the <body> and </body> tag.  To avoid any font and background color conflict issues, I decided to use uncommon colors for top and bottom banner.  Here is my filter.sh:

#!/bin/sh# Insert banner after and before the body opening and closing tags.
/bin/sed -r 's/(<body.*>$)/\1\<div align=center\>\<font size=4 color=#00FFFF\>Development Instance\<\/font\>\<div\>/1MI' | /bin/sed -r 's/\s*(<\/body.*>)/<div align=center\>\<font size=4 color=#00FF00\>Development Instance\<\/font\>\<div\&/1MI'


Notice, there are two sed find/replace happening, first one adds the header and second one adds the footer.  Here is a brief explanation of the script above:
-r: Use regular expressions

(<body.*>$)/\1: find the first instance of body tag.

\<div align=center\>\<font size=4 color=#00FFFF\>Development Instance\<\/font\>\<div\> : Fancy font work, with '\' for escaping special characters.

1MI: Stop after first find/replace, multi-line, and case insensitive


Escaping special characters makes the above script unreadable.  A better choice might be a Python script, especially if you plan to do something more elaborate, since Python can be compiled into object code.

The impact of page load performance is only noticeable on large pages using the proposed solution.  Beware, if the script has syntactic or other errors, the result is often a blank web page.  No amount of logging will reveal anything useful, and the only solution is the run the script independent of mod_ext_filter (e.g cat test.html | sh filter.sh ).

Hope this helps!
VVK

Wednesday, December 1, 2010

CLI Interactions w/SSL Enabled Website

I found it amusing that one of the major changes in the new PCI 2.0 regulation requires that any vulnerabilities with a CVSS score > 4 must be remediated (6.2).  It is amusing because what good does it do to require companies to perform vulnerability scanning, if remediation is not enforced, which was the case with the previous version of PCI DSS (11.2).

I digress.  Often I am required to confirm an identified vulnerability or validate a fix for a web server.  For example, checking to see if TRACK/TRACE is enabled/disabled or HOST header is set for name based virtual hosts.  These checks are easy to perform on a non-SSL web server (HTTP) using Telnet, but Telnet cannot be used against an SSL enabled web server (HTTPS).

# telnet ssl.somehost.com 443
Trying 10.10.3.93...
Connected to ssl.somehost.com.
Escape character is '^]'.
GET / HTTP/1.1
Connection closed by foreign host.

Telnet-SSL
The connection fails because telnet lacks SSL support.  You can verify this on Linux using the ‘ldd’ tool.
# ldd /usr/bin/telnet
        linux-gate.so.1 =>  (0xffffe000)
        libncurses.so.5 => /lib/libncurses.so.5 (0xb76c8000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb75da000)
        libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb75b3000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb75a4000)
        libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb744a000)
        libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7446000)
        /lib/ld-linux.so.2 (0xb7711000)

The good news is that there is an SSL enabled version of telnet, called  telnet-ssl (netkit-telnet-ssl).  In the following examples, I am using the BackTrack 4 distribution.

# apt-get install telnet-ssl
# ldd /usr/bin/telnet
        linux-gate.so.1 =>  (0xffffe000)
        libncurses.so.5 => /lib/libncurses.so.5 (0xb76cb000)
        libssl.so.0.9.8 => /usr/lib/i686/cmov/libssl.so.0.9.8 (0xb7684000)
        libcrypto.so.0.9.8 => /usr/lib/i686/cmov/libcrypto.so.0.9.8 (0xb7537000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb7449000)
       [snip]

# ls -l `which telnet`
lrwxrwxrwx 1 root root 24 Dec  1 10:48 /usr/bin/telnet -> /etc/alternatives/telnet
# ls -l /etc/alternatives/telnet
lrwxrwxrwx 1 root root 19 Dec  1 10:48 /etc/alternatives/telnet -> /usr/bin/telnet-ssl

# telnet -z ssl ssl.somehost.com 443
Trying 10.10.3.93...
Connected to ssl.somehost.com.
Escape character is '^]'.
GET / HTTP/1.1
HOST: ssl.somehost.com
HTTP/1.1 200 OK
Content-Type: text/html
Last-Modified: Thu, 11 Nov 2010 15:08:59 GMT
Accept-Ranges: bytes
Server: Microsoft-IIS/7.5
Date: Wed, 01 Dec 2010 17:07:40 GMT
Content-Length: 519

[snip]

Metasploit
Another alternative to installing netkit-telnet-ssl is to use Metasploit itself.  Of course Metasploit might be an overkill if all you want to do is perform simple tests like the one above.  In the case of BackTrack 4 distribution, Metasploit comes installed by default. 

msf > connect -s ssl.somehost.com 443
[*] Connected to ssl.somehost.com:443
GET / HTTP/1.1
HOST: ssl.somehost.com
HTTP/1.1 200 OK
Content-Type: text/html
Last-Modified: Thu, 11 Nov 2010 15:08:59 GMT
Accept-Ranges: bytes
Server: Microsoft-IIS/7.5
Date: Wed, 01 Dec 2010 17:07:40 GMT
Content-Length: 519

[snip]

If you know of any standard tools on Linux and Windows or any 3rd party tools for Windows, which can do the same, please leave a comment.

Cheers,
VVK

Friday, July 30, 2010

SSL/TLS Weak Cipher

While reviewing a Qualys report, I noticed the following "QID: 38140 SSL Server Supports Weak Encryption Vulnerability".  Of course one can verify Qualys findings one cipher at a time using openssl, but in order to verify all supported cipher-MAC combination, I needed to find an automated tool. Here are some of the useful ones I found:
  1. Qualys SSL Labs - Good choice if you need to generate a presentable report for management.
  2. CryptoNark - In addition to checking SSL Ciphers, it also does HTTP Track/Trace check and 'Unsafe' URL check.  You will need to install some custom Perl modules to get it working.
  3. SSLscan - Comes bundled with BackTrack4.
It is a lot easier to quickly verify your remediation using these tools, as opposed to submitting another Qualys scan.

Update (6/21/11): Here is a new tool from Leviathan Security to help test SSL Re-negotiation vulnerability. (also see "(Really) Testing for SSL/TLS Re-negotiation")

Cheers,
VVK

Thursday, July 29, 2010

Quick 'n' Dirty Bulk Reverse Lookup

Although you can do bulk forward look-ups using '-f' option in dig, for some reason this does not work for reverse look-ups. See: Dig HowTo

So, I whipped up a quick python script to get the job done. Script assumes you have a file with list of IP's, one per line.
input = open('ips.txt','r')
for i in input.readlines():
   try:
      result = socket.gethostbyaddr(i)
      print i.strip('\n'),result[0]
   except:
      continue
Please use the script at your own risk!

I got the following trick from my good friend Kevin to accomplish similar result using nmap:
nmap -sL -iL fileofips.txt | grep '('

Cheers,
VVK

Tuesday, May 4, 2010

Microsoft's Adobe PDF Woes

In recent months, Adobe's Acrobat reader has become an attractive target on Microsoft Windows platform.

I stopped using Acrobat reader on my Windows hosts a while back when I realized that Acrobat reader is 38 MB in size, compared to 1.2 MB SumatraPDF (Open Source), 6.7 MB Foxit Reader (Closed Source), and online Google Docs.

Here I am assuming that the size of the Acrobat Reader directly correlates with the number of lines of code, and studies have shown a direct relationship between the number of lines and number of defects.

"Commercial software typically has 20 to 30 bugs for every 1,000 lines of code, according to Carnegie Mellon University's CyLab Sustainable Computing Consortium. This would be equivalent to 114,000 to 171,000 bugs in 5.7 million lines of code" (Linux: Fewer Bugs Than Rivals )

I consider myself a very heavy PDF user, and it is extremely rare for me to need features such as Javascript functionality, Assistive technology, various third party plugins etc. built into Acrobat Reader.

F-Secure's Sean Sullivan recently urged Microsoft to integrate some type of PDF preview option and spare its users from having to install Adobe Acrobat Reader. I could not agree more with Mr. Sullivan. It is about time Microsoft learned a lesson from Gnome (Evince) and KDE (Okular) and provided a bundled basic PDF viewer as part of the OS.

My 2 cents.
VVK

Saturday, March 20, 2010

BackInTime on CentOS 5.x

Back In Time is a simple backup tool geared mostly toward Linux desktop. Only Debian/Ubuntu binary packages are available from the official website.

I like Back In Time because it uses standard Linux tools like rsync,diff and cp to create backups and heavily utilizes hard-links to save space on the backup media.

I wanted to install it on a CentOS 5.x host and decided to create an RPM so others can use it on RPM based distros.

Spec File: backintime-cli.spec
Source RPM: backintime-cli-0.9.26-1.src.rpm
x86 RPM: backintime-cli-0.9.26-1.rhel5.i386.rpm

Configuration
Configuration file is located in /etc/backintime/config. I haven't yet found any useful documentation. The included config file was generated by the KDE4 GUI client. I removed many of the GUI specific configuration lines. If you come across good documentation, please leave a comment.

If you run /usr/bin/backintime as root, it uses /etc/backintime/config for configuration.
If you wish to run backintime as a regular user, copy the /etc/backintime/config file to ~/.config/backintime/config.

Here is what I have managed to figure out from experimenting using the GUI client:

config.version=2
snapshots.automatic_backup_mode=20
snapshots.cron.nice=true

# Manually run snapshots are not removed automatically

snapshots.dont_remove_named_snapshots=true
snapshots.exclude_patterns=*.backup*:*~

# Useful if you have mixed snapshot frequencies

snapshots.expert.per_directory_schedule=true

#0-manual,10-hourly,20-daily,30-weekly,40-monthly
# [file/folder]|[snapshot frequency]

snapshots.include_folders=/etc|30:/home:|20:/root|30:/var/named|40
snapshots.min_free_space.enabled=true

# 10=Mb, 20=Gb

snapshots.min_free_space.unit=20

# Maintain minimum of 1G free space on destination

snapshots.min_free_space.value=1
snapshots.notify.enabled=true

# Backup destination. Create 'backintime' folder manually.

snapshots.path=/backup
snapshots.remove_old_snapshots.enabled=true

# 20=days,30=weeks,80=years

snapshots.remove_old_snapshots.unit=30

# Remove backups older than 27 weeks

snapshots.remove_old_snapshots.value=27

# keep all snapshots from today & yesterday
# keep one snapshot for the last week and one for two weeks ago
# keep one snapshot per month for all previous months of this year
# keep one snapshot per year for all previous years

snapshots.smart_remove=true

Sample run:

[root@lamp32 ~]# backintime -b /etc
Back In Time
Version: 0.9.26

Back In Time comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type `backintime --license' for details.

INFO: Lock
INFO: Include folders: ['/etc']
INFO: Ignore folders: []
INFO: Last snapshots: {}
INFO: Compare with old snapshot: 20100320-181730
INFO: Command "rsync -aEAX -i --dry-run --chmod=Fa-w,D+w --whole-file --delete --exclude="/backup" --exclude="/root/.
local/share/backintime" --include="/etc/" --exclude="*.backup*" --exclude="*~" --include="/etc/**" --exclude="*" / "/bac
kup/backintime/20100320-181730/backup/"" returns 0
INFO: Create hard-links
INFO: Command "cp -al "/backup/backintime/20100320-181730/backup/"* "/backup/backintime/new_snapshot/backup/"" returns 0
INFO: Call rsync to take the snapshot
INFO: Command "rsync -aEAX -v --delete-excluded --chmod=Fa-w,D+w --whole-file --delete --exclude="/backup" --exclude="/root/.local/share/backintime" --include="/etc/" --exclude="*.backup*" --exclude="*~" --include="/etc/**" --exclude="*" / "/backup/backintime/new_snapshot/backup/"" returns 0
INFO: Save permissions
INFO: Remove backups older than: 20090907-000000
INFO: [smart remove] keep all >= 20100319-000000
INFO: [smart remove] keep first >= 20100308-000000 and <>= 20100301-000000 and <>= 20100101-000000 and <>= 20100201-000000 and <>

Hope this helps someone looking to run backintime on RHEL/CentOS.

Cheers,
VVK

Thursday, February 18, 2010

Antivirus False Positive Validation

ClamAV running on a clients server flagged a packed javascript file as a virus. A co-worker recommended Virustotal website for testing and fortunately it appears to be a false positive based on the fact that only 1 (ClamAV) out of 41 anti-virus solutions flagged it as a virus.

Update: Javascript Unpacker tool.

Update 4/8/10: Jotti's Malware Scan
Also check out the following post from Kerbs:
Virus Scanners for Virus Authors - Part 1
Virus Scanners for Virus Authors - Part 2

Cheers,
VVK

Saturday, November 8, 2008

Enhance Windows CLI Experience

Background
My client uses Check Point VPN solution, which does not provide a native Linux client. My work around has been to run Windows XP as a VM for VPN purposes.

Problem

Windows lacks numerous Unix tools which I have grown accustomed to and to make matters worse, Windows XP cmd shell is a complete piece of crap.

Solution
Open Source to the rescue! By combining Cygwin and Console, I have achieved command line usability *almost* at par with my Linux desktop environment.

Once you find a fast,reliable mirror site, installing Cygwin is straight forward, . Here is a list of non-default packages which I find quite useful:
  1. Editors - vim
  2. Net - netcat, openssh, rsync, stunnel
  3. Security - pwgen
  4. Shells - bash-completion
  5. Utils - diffutils, keychain, screen
  6. Web - lynx
There is no installer for Console (version 2), it is available for download in zip format. Here are few tweaks I did to replace Windows cmd as the terminal program for Cygwin and configure Cygwin as the default shell for Console:
  1. Modify the Cygwin desktop shortcut and specify Console.exe as the Target.

    Shortcut >> Target >> D:\Console2\Console.exe

  2. Apply the following settings to your Console.exe preferences:

    Console:
    Shell >>D:\cygwin\bin\bash.exe --login -i
    Startup dir >> d:\cygwin\bin


    Behavior:
    Copy & Paste >> Copy on select
    Copy newline character >> Unix (LF)


    Hotkeys:
    New Tab 1 >> Ctr + Shift + T
    Rename tab >> Ctr + Shift + R

    Tabs:
    Main >> Shell >>D:\cygwin\bin\bash.exe --login -i
    Main >> Startup dir >> d:\cygwin\bin

Thats it! Enjoy the usability of a Linux style terminal and shell in Windows environment. So long putty, good-bye cmd.exe!

VVK

Sunday, November 2, 2008

strace to the rescue

Background
Client would like to see some rough site usage statistics using webalizer.

Problem
After some preliminary testing, I installed and configured the webalizer tool on the production server. However, all my attempts to run webalizer resulted in "segmentation fault". Even setting the "Quiet" and "ReallyQuiet" options to "no" did not help.

[root@host ~]# /usr/bin/webalizer -c /etc/webalizer/webalizer.conf
Segmentation fault

On the other hand running webalizer without the configuration file (-c), worked.

Solution
Installed strace and ran the following:

strace -o /var/tmp/webalizer.out /usr/bin/webalizer -c /etc/webalizer/webalizer.conf

Check out the last few lines in the strace output:

[root@host webalizer]# less /var/tmp/webalizer.out
[...]
write(1, "Webalizer V2.01-10 (Linux 2.6.9-"..., 54) = 54
open("/var/www/site.com/logs/site.com-access_log", O_RDONLY|O_LARGEFILE) = 3
write(1, "Using logfile /var/www/site.com"..., 81) = 81
chdir("/var/www/html/site") = -1 ENOENT (No such file or directory)
write(2, "Error: Can\'t change directory to"..., 61) = 61
exit_group(1) = ?


Sure enough I had a typo in the webalizer.conf file. The path I specified for the "OutputDir" option was incorrect.

VVK