Tenable SC4 Asset Groups From AD

Note (5/8/12): With the latest release of Tenable Security Center 4.4.x, you can create asset groups using DNS names.

If you are a Tenable Security Center 4 user, you might have noticed that unlike Qualys, you can only use IP addresses in SC4 asset groups**.  In normal production environments, IPs are static and this restriction does not pose a huge challenge.  However, in my case we are using SC4 to scan a large end-user desktop environment which is using AD Dynamic DHCP.  Due to this, I find myself having to update the asset groups before a scan and it can be tedious.

To make things easier, I decided to write a quick 'n' dirty shell script to get the computer names from AD and do a reverse lookup.

#!/bin/sh

LDAPURI="ldap://corp.ldap.server"
USER="user@domain.com"
echo "LDAP Password:"
read PASS
DEPT=([1]="Dept 1" [2]="Dept 2" [3]="Dept 3")

for ((i=1; i<=${#DEPT[@]}; i++ ));
do
        CMD="/usr/bin/ldapsearch -x -v -LLL -E pr=200/noprompt -H $LDAPURI -D \"$USER\" -w $PASS -b \"OU=${DEPT[$i]} Computers,OU=${DEPT[$i]},OU=Corporate,DC=domain,DC=com\" -s one \"(objectClass=computer)\" | grep name | awk '{print \$2}' | tr '[:upper:]' '[:lower:]' | sed 's/$/.domain.com/' | dig +short -f -"

        echo "Run $CMD:"

        eval $CMD > ${DEPT[$i]}_IPs.txt
done

The resulting output should be one file per department containing IP's of all the names which resolved.

Note: I am doing a simple LDAP bind (-x) and the password (-w) is sent clear-text.

** I have been told by my Tenable sales rep that in late Feb 2012 they *might* include the ability to add hosts using DNS name.

Thanks,
VVK

Windows XP Firewall All Ports IP Exception

Windows XP is no longer officially supported by Microsoft, but that does not mean it has cease to exist in the enterprise. With SP2, Microsoft introduced a built in firewall (and I use that term loosely) which improved security (compared to pre-SP2).

For whatever reason, Microsoft decided to not offer the ability to simply specify a port range. Each specified rule can be related to a service or a port/protocol combination. This might not seem like a major hurdle, until you come across a specific need such as vulnerability scanning.

I recently ran into this issue where I needed to make an exception for a specific vulnerability scanner IP for all port/protocol combinations on a Windows XP machine. Disabling the firewall was not an option because the target network of Windows XP hosts is in a high risk environment and to make matters worse, the users have administrative privileges.

There are two solutions to this problem, you can either run a for-loop and insert your firewall rules, or you can update the %windir%\Inf\Netfw.inf file.

For-loop Approach

Run the following using a script of command-line:

FOR /L %I IN (1,1,65535) DO netsh firewall add portopening protocol = ALL port = %I name = scanner mode = ENABLE scope = CUSTOM addresses = 10.10.12.101/32

Disclaimer:

The result of this command is 131070 lines of firewall rules, all named 'scanner'.  Also, it will take a long time for this command to run.

Based on my test on a VM (1 x 1.6 GHz, 512 MB), I was able to process 66 rules per minute.  So it will take approximately 16 hours to insert all 131070 rules!  :-O

Netfw.inf Approach 

Windows XP firewall has two modes, domain and standard.  Depending on your use scenario, add a variation of the following lines under the appropriate section:

HKLM,"SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\DomainProfile\GloballyOpenPorts\List","port number:protocol",0x00000000,"port number:protocol:scope:mode:port’s friendly name"

port number – A port is specified by the combination of a protocol and a port number. The port number must be between 1 and 65535 inclusive.

protocol – A port is specified by the combination of a protocol and a port number. The protocol must be either TCP or UDP. 

scope – Permitted values for scope are defined the “Defining the Scope for an Entry in the Windows Firewall INF File” section of this article.

mode – An entry can be added to Windows Firewall’s default exceptions lists as either enabled or disabled. The two permitted values for this element are enabled and disabled. If a port’s entry is enabled, the port will be statically opened in Windows Firewall. If a port’s entry is disabled, the port will not be statically opened in Windows Firewall.

port’s friendly name – This is the description that will be used to represent the entry in the Windows Firewall Control Panel applet. It should provide an indication of why the port is statically opened, such as "Web Server (TCP 80)" or "Telnet Server (TCP 23)".

 

For more information, refer to "Using the Windows Firewall INF File in Microsoft Windows XP Service Pack 2"

Here is a simple python script to help you generate your firewall rules.  Rules are written to 'fw.txt'.  Copy the lines and paste them under the appropriate profile in %windir%\Inf\Netfw.inf.

To apply the changes, run:

netsh firewall reset

Note: This will take a very long time.

You can verify your rules by running:

netsh firewall show config

Cheers,

VVK

Zenoss 3.0.3 Disappointments

Lured by the new sleek looking Zenoss interface in 3.x series, and availability of pre-compiled RHEL/CentOS rpm's, I decided to give it a go.  This turned out to be a terrible waste of time and effort.

Installation
The rpm by default created a user zenoss (UID 3117) and group zenoss with bash as shell.  I did not like the fact that the rpm did not add the user as the system user, and decided to change it.  This turned out to be a terrible mistake!  Zenoss developers for some very strange reason have decided to hard code the database name, db user, db password, username, zope password, username, UID, and group in various files (python & shell script).

I should have stopped when I found myself grep'ing through all the files looking for these hard coded values, but I was on a fool's mission.

Once I finished editing all the hard coded values, Zenoss refused to start with a very confusing error message "This account is currently not available."  It turned out that zenoss user must have a valid interactive login shell (bash in this case).

Once I got Zenoss started, it spit out a lot of errors about missing librrd.so.4.  I found the file in /opt/zenoss/lib and added it to my ld.so.conf file, which resolved the issues.  I am not sure why the rpm did not set this up.

As per the zenoss documentation, I proceeded to installing the zenoss-core-zenpacks.  It turned out to be another failure because it too had user and group hard coded.  It refused to install, unless I renamed the zenoss user and group from 'monitor' to 'zenoss'.

Next, all my attempts to access the portal resulted in very strange behavior.  The page kept reloading non-stop.  I was expecting some sort of a user setup wizard, but no such luck.  All attempts to guess the password failed.

Uninstallation
At this point, any normal person would call it a night and revisit the issue after getting some sleep.  Not I!  I decided to remove all my changes and accept the defaults set by the rpm.  I made the mistake of deleting the user and group (monitor:monitor) I had created prior to uninstalling the zenoss and zenoss-core-zenpacks rpm.  Adding the user turned out to be insufficient, to uninstall the rpm you have to add the user and group with the same name and uid/gid.  However, the ordeal was not over yet.  Turns out, to uninstall the zenoss-core-zenpacks, you must have zenoss running and a particular python process listening on TCP/8100.

Finally, I managed to uninstall the whole mess and decided to start fresh with all the rpm defaults.  Accepting all the rpm defaults resulted in success and I was finally greeted with a functioning monitoring portal.

Reflection
I decided to call it a (very unsatisfactory) night.  I woke up this morning and couldn't help wonder if Zenoss is worth using.  If the installation, customization and uninstallation could be so poorly engineered, I can only imagine what issues might creep up in the future.  I have decided to uninstall Zenoss and continue my search for a monitoring solution.

Feedback
This post won't be constructive criticism if I did not offer up some solutions, so here are my 2 cents:

1. Please don't hard code configuration values in multiple files.  Why not designate one python file which is referenced by all other python scripts (pre_init, post_init, upgrade etc.)?  Similarly, don't hard code configuration values in init.d script, a better option would be to create a /etc/sysconfig/zenoss file.
2. When adding a user, please add it as a system user since the role of the user is to run system services.  Also, there is no reason to assign this user a home directory in /home.  /opt/zenoss will be a more appropriate choice.
3. I am not a fan of running services with an interactive login shell such as bash due to security reasons.  I did not dig deep enough to determine if this was a necessity, but I am sure if daemons like Apache can run without an interactive shell, so can a simple monitoring application.  I think /sbin/nologin is an appropriate choice.
4. Please reference the user added by the rpm consistently.  I noticed some references were using UID, while others were using username 'zenoss'.
5. Newer PCI compliance standards require us to not accept vendor defaults when it comes to settings such as passwords.  So it is very important to easily allow a system administrator to change settings such as database user, database password etc.  I think web applications such as Drupal have done a fantastic job on installation wizard.
6. Please update the zenoss-core-pack rpm to check to see if the zenoss service is running, and if not, proceed with the uninstalltion.  If the service is necessary for uninstallation, then either consider starting it or adding such dependency to the '%preun' section in the rpm spec file.
7. Please add a zenoss library config to /etc/ld.so.conf.d folder.

Cheers & Happy Thanksgiving!,
VVK

Linux VM Kernel Tweaks

Based on "Oracle 10g Server on Red Hat Enterprise Linux 5 - Deployment Recommendations", I updated my VirtualBox Linux VM's to use the following kernel parameters:

64bit RHEL/CentOS VM: divider=10 notsc iommu=soft elevator=noop
32bit RHEL/CentOS VM: divider=10 clocksource=acpi_pm iommu=soft elevator=noop

Consult /usr/share/doc/kernel-doc-2.6.18/Documentation (kernel-doc-2.6.18-164.el5 rpm) for more info on these parameters.  The above mentioned Oracle document does a good job of explaining the parameters as well.

Although, I did not do any scientific measurements, just from the CPU usage live graph and pidstat (sysstat), I can confirm that my CPU usage dropped drastically.

Pre-tweak:

$ pidstat -u -p 22414 5 10  
Linux 2.6.30.8-64.fc11.i586 (moonshine) 10/06/2009

01:49:27 PM PID %user %system %CPU CPU Command
01:49:32 PM 22414 39.20 28.00 67.20 1 VBoxHeadless
01:49:37 PM 22414 40.80 27.60 68.40 1 VBoxHeadless
01:49:42 PM 22414 37.80 28.40 66.20 1 VBoxHeadless
01:49:47 PM 22414 38.40 27.80 66.20 1 VBoxHeadless
01:49:52 PM 22414 38.60 28.20 66.80 1 VBoxHeadless
01:49:57 PM 22414 36.40 28.20 64.60 1 VBoxHeadless
01:50:02 PM 22414 38.80 26.20 65.00 1 VBoxHeadless
01:50:07 PM 22414 38.40 28.80 67.20 1 VBoxHeadless
01:50:12 PM 22414 38.40 29.00 67.40 1 VBoxHeadless
01:50:17 PM 22414 38.00 27.80 65.80 1 VBoxHeadless
Average: 22414 38.48 28.00 66.48 - VBoxHeadless


Post-tweak:

$ pidstat -u -p 22414 5 10
Linux 2.6.30.8-64.fc11.i586 (moonshine) 10/06/2009

01:53:55 PM PID %user %system %CPU CPU Command
01:54:00 PM 22414 3.20 3.60 6.80 1 VBoxHeadless
01:54:05 PM 22414 3.20 3.60 6.80 1 VBoxHeadless
01:54:10 PM 22414 2.80 3.40 6.20 1 VBoxHeadless
01:54:15 PM 22414 2.80 2.80 5.60 1 VBoxHeadless
01:54:20 PM 22414 3.00 3.20 6.20 1 VBoxHeadless
01:54:25 PM 22414 3.60 3.80 7.40 1 VBoxHeadless
01:54:30 PM 22414 4.00 4.40 8.40 1 VBoxHeadless
01:54:35 PM 22414 3.20 3.00 6.20 1 VBoxHeadless
01:54:40 PM 22414 5.00 5.80 10.80 1 VBoxHeadless
01:54:45 PM 22414 6.40 4.00 10.40 1 VBoxHeadless
Average: 22414 3.72 3.76 7.48 - VBoxHeadless

Cheers,
VVK

EC2 instance meta-data via SNMP

Here is a simple script and a sample snmp.conf file which will enable you to query your EC2 instance meta-data via SNMP. For more information on meta-data, see (see page 49 EC2 Developers Guide ).

metadata.py

#!/usr/bin/python
# This script returns metadata specific to AWS instances
# See Page 49, Amazon Elastic Compute Cloud - Developer Guide

from httplib import HTTPConnection
from sys import argv
import socket

if len(argv) > 1:
     try:
          conn = HTTPConnection("169.254.169.254")
          url = "/latest/meta-data/" + str(argv[1])
          conn.request("GET",url)
          r1 = conn.getresponse()
          if r1.status == 200:
               print r1.read()
          else:
               print r1.status,url
     except socket.error,msg:
         print msg[1] 
else:
     print "data key missing"

snmp.conf

rwcommunity EDF9ErIfzNFb 127.0.0.1
rocommunity public

syscontact user@host.com
syslocation AWS

com2sec notConfigUser default public
group notConfigGroup v1 notConfigUser
group notConfigGroup v2c notConfigUser

# RFC1213-MIB.iso.org.dod.internet.mgmt.mib-2
view systemview included .1.3.6.1.2.1.1
view systemview included .1.3.6.1.2.1.2

# HOST-RESOURCES-MIB.iso.org.dod.internet.mgmt.mib-2.host
view systemview included .1.3.6.1.2.1.25 view systemview included .1.3.6.1.2.1.25.1.1

# UCD-SNMP-MIB.iso.org.dod.internet.private.enterprises.ucdavis
view systemview included .1.3.6.1.4.1.2021

access notConfigGroup "" any noauth exact systemview none none

# Added for support of bcm5820 cards.
pass .1.3.6.1.4.1.4413.4.1 /usr/bin/ucd5820stat

#agentaddress 192.168.8.103

# Executable Scripts
# AWS Specific
exec ami-id /etc/snmp/aws_metadata.sh ami-id
exec ami-launch-index /etc/snmp/aws_metadata.sh ami-launch-index
exec ami-manifest-path /etc/snmp/aws_metadata.sh ami-manifest-path
exec ancestor-ami-ids /etc/snmp/aws_metadata.sh ancestor-ami-ids
exec block-device-mapping /etc/snmp/aws_metadata.sh block-device-mapping
exec instance-id /etc/snmp/aws_metadata.sh instance-id
exec instance-type /etc/snmp/aws_metadata.sh instance-type
exec local-hostname /etc/snmp/aws_metadata.sh local-hostname
exec local-ipv /etc/snmp/aws_metadata.sh local-ipv4
exec kernel-id /etc/snmp/aws_metadata.sh kernel-id
exec availability-zone /etc/snmp/aws_metadata.sh availability-zone
#exec product-codes /etc/snmp/aws_metadata.sh product-codes
exec public-hostname /etc/snmp/aws_metadata.sh public-hostname
exec public-ipv4 /etc/snmp/aws_metadata.sh public-ipv4
#exec public-keys /etc/snmp/aws_metadata.sh public-keys
exec ramdisk-id /etc/snmp/aws_metadata.sh ramdisk-id
exec reservation-id /etc/snmp/aws_metadata.sh reservation-id
exec security-groups /etc/snmp/aws_metadata.sh security-groups

# Disk Checks
disk /

# Process Checks
proc crond
proc ntpd
proc rsyslogd
proc sshd

Sample Output

[root@ami1 ~]# snmpwalk -Os -c public -v 2c localhost .1.3.6.1.4.1.2021.8.1.101.1
extOutput.1 = STRING: ami-08f41161

[root@ami1 ~]# snmpwalk -Os -c public -v 2c localhost .1.3.6.1.4.1.2021.8.1.101.2
extOutput.2 = STRING: 101

Hope this helps!
VVK

Distributed monitoring with Zabbix - Part 1

After Red Hat announced their plans to discontinue their monitoring product, Red Hat Command Center, I was on the hunt for a good replacement. Our goals for a monitoring product was quite straight forward:

1. Free and Open Source. Most commercial products charge per IP, monitor, agent etc. and that could quickly become very expensive for us since we are planning to monitor many customers down the road.
2. Agent-less monitoring. Primary focus is on SNMP based monitoring and only use agent based monitoring as a last resort. SNMP is the least intrusive way to monitor a system and it is a mature technology supported by many development platforms.
   
3. Scalable solution. We want the product to be able to perform distributed monitoring (master-slaves).
4. Web based interface. Dealing with thick clients is a hassle when it comes to the management software or even the agent software.
5. More than up-down monitoring. We want to collect historic usage data for reporting and convenient analysis of trends.
6. Custom reports.
   

With these goals in mind, I started to scour the web for various monitoring solutions. Fortunately, my search was made easy by a wonderful comparison matrix of monitoring products on Wikipedia.

We tried Zenoss, Groundwork community, ManageEngine OpManager, and we already had experiences with Nagios, OpenNMS, WhatsUp Gold, and ProactiveNet. For the sake of brevity, I am not going to explain why we did not choose any of the above mentioned products. Although, I should mention that we really liked ManageEngine's OpManager 8, however, their per monitor license fee was a bit hard to swallow. Finally, we hesitently settled on Zabbix because it met all our goals and offered few other goodies such as:

1. Server, proxy and agent daemons run unpriveleged.
2. Ease of custom template creation.
3. Ability to create nice custom graphs.
4. Ability to graph just about any monitored numerical value.
5. LDAP authentication for web interface.
   

Zabbix does have plenty of flaws, and it is by no means a perfect product.

1. Although the Zabbix Manual is over 300 pages long, it is not very helpful. There is a lot of room for improvement when it comes to documentation pertaining to doing more advanced monitoring with Zabbix.
2. The php web interface is not very intutive. Considering this is the primary interface for interacting with the monitoring solution, it is also the primary source of much of my frustration. Hopefully, I will touch on many of them in my future posts.
3. Zabbix forum is not very helpful. I am not sure if this is due to the size of the community, my phrasing of the questions or something else.

My goal with this and the following blog entries would be to document how to setup distributed monitoring with Zabbix. I will try to point out the what I consider to be flaws or non-intutive features and how they (Zabbix devs) can possibly improve it.

Cheers,
VVK