Showing posts with label troubleshooting. Show all posts
Showing posts with label troubleshooting. Show all posts

Monday, October 31, 2011

Apache Output Rewrite/Filter

When performing migration or code upgrades on public facing website, it is a common practice to test the changes in a development environment.   Occasionally, due to various reasons, you are forced to reference the development instance of the site using the public domain name.

For example, if the public facing site in question is www.example.com, and you want to setup a development instance in your lab, but want to be able to reference the site using the public DNS record, you have essentially two options: (1) Modify your DNS resolver (2) Modify your 'host' file.

The DNS option requires more effort but it is a better option if you have multiple developers, all using the same DNS server.

Host file trick is often used when you do not have a split-horizon DNS option or you do not control your DNS server.  Host file trick is the easiest and the most commonly used trick for such needs, however it comes with a bit of risk.  There is always the risk that a developer forgot to add the entry in his/her host file, and accidentally made changes to the production instance.  One simple trick to address such accidents is to modify the development site's header image or something similar.  However, in some applications which make extensive use of caching, sometimes this trick fails or leads to confusion.

Recently, I found myself on one such project (Drupal CMS) where we required not just a development instance, but also a QA instance.  To make matters worse, the developers needed to jump between production, development and QA instances multiple times a day.  I wanted to figure out a way to modify the web content to reflect the environment (dev or QA) without modifying the application PHP code.  Drupal has 3rd party extensions which can address this need, but I wanted to find a solution which was independent of the application (Drupal) and the server-side scripting technology (PHP).  I needed to make the modification at Apache level.

After doing some research, I quickly discovered Apache mod_ext_filter, a standard Apache module.  mod_ext_filter met all my criteria:

  1. Web application independent
  2. Server-side scripting technology independent
  3. Flexible and simple
  4. No 3rd party Apache modules or modifications to existing modules
  5. Performance should be acceptable
If you know of a better way, than the one proposed below, to accomplish these goals, please post a comment.

mod_ext_filter presents a simple and familiar programming model for filters. With this module, a program which reads from stdin and writes to stdout (i.e., a Unix-style filter command) can be a filter for Apache. This filtering mechanism is much slower than using a filter which is specially written for the Apache API and runs inside of the Apache server process, but it does have the following benefits:
  • the programming model is much simpler
  • any programming/scripting language can be used, provided that it allows the program to read from standard input and write to standard output
  • existing programs can be used unmodified as Apache filters


To use mod_ext_filter, enable it in your httpd.conf file by adding the following line:
LoadModule setenvif_module modules/mod_setenvif.so
For my needs, my goal was to inject a line of text in the header and footer of each page to identify the environment (dev or QA).

Using ext_filter module, a custom stdout filter can be as simple as adding a call to 'sed' in the module configuration file (see "Using sed to replace text in the response" example).  The problem with this approach is that you need to restart Apache every time you made a change to your filter, which can get annoying very quickly.
A better approach is to call an external script, which can be written in any language of choice.  In addition to not having to restart Apache, this approach has the advantage of allowing for easier troubleshooting.

Here is my /etc/httpd/conf.d/ext_filter.conf calling an external filter.sh script.
ExtFilterDefine banner mode=output intype=text/html \
cmd="/bin/sh /var/www/filter.sh"
 
<Location />
        SetOutputFilter banner
</Location>
 
 In my filter.sh, I decided to perform a simple find/replace on the <body> and </body> tag.  To avoid any font and background color conflict issues, I decided to use uncommon colors for top and bottom banner.  Here is my filter.sh:

#!/bin/sh# Insert banner after and before the body opening and closing tags.
/bin/sed -r 's/(<body.*>$)/\1\<div align=center\>\<font size=4 color=#00FFFF\>Development Instance\<\/font\>\<div\>/1MI' | /bin/sed -r 's/\s*(<\/body.*>)/<div align=center\>\<font size=4 color=#00FF00\>Development Instance\<\/font\>\<div\&/1MI'


Notice, there are two sed find/replace happening, first one adds the header and second one adds the footer.  Here is a brief explanation of the script above:
-r: Use regular expressions

(<body.*>$)/\1: find the first instance of body tag.

\<div align=center\>\<font size=4 color=#00FFFF\>Development Instance\<\/font\>\<div\> : Fancy font work, with '\' for escaping special characters.

1MI: Stop after first find/replace, multi-line, and case insensitive


Escaping special characters makes the above script unreadable.  A better choice might be a Python script, especially if you plan to do something more elaborate, since Python can be compiled into object code.

The impact of page load performance is only noticeable on large pages using the proposed solution.  Beware, if the script has syntactic or other errors, the result is often a blank web page.  No amount of logging will reveal anything useful, and the only solution is the run the script independent of mod_ext_filter (e.g cat test.html | sh filter.sh ).

Hope this helps!
VVK

Thursday, November 25, 2010

Zenoss 3.0.3 Disappointments

Lured by the new sleek looking Zenoss interface in 3.x series, and availability of pre-compiled RHEL/CentOS rpm's, I decided to give it a go.  This turned out to be a terrible waste of time and effort.

Installation
The rpm by default created a user zenoss (UID 3117) and group zenoss with bash as shell.  I did not like the fact that the rpm did not add the user as the system user, and decided to change it.  This turned out to be a terrible mistake!  Zenoss developers for some very strange reason have decided to hard code the database name, db user, db password, username, zope password, username, UID, and group in various files (python & shell script).

I should have stopped when I found myself grep'ing through all the files looking for these hard coded values, but I was on a fool's mission.

Once I finished editing all the hard coded values, Zenoss refused to start with a very confusing error message "This account is currently not available."  It turned out that zenoss user must have a valid interactive login shell (bash in this case).

Once I got Zenoss started, it spit out a lot of errors about missing librrd.so.4.  I found the file in /opt/zenoss/lib and added it to my ld.so.conf file, which resolved the issues.  I am not sure why the rpm did not set this up.

As per the zenoss documentation, I proceeded to installing the zenoss-core-zenpacks.  It turned out to be another failure because it too had user and group hard coded.  It refused to install, unless I renamed the zenoss user and group from 'monitor' to 'zenoss'.

Next, all my attempts to access the portal resulted in very strange behavior.  The page kept reloading non-stop.  I was expecting some sort of a user setup wizard, but no such luck.  All attempts to guess the password failed.

Uninstallation
At this point, any normal person would call it a night and revisit the issue after getting some sleep.  Not I!  I decided to remove all my changes and accept the defaults set by the rpm.  I made the mistake of deleting the user and group (monitor:monitor) I had created prior to uninstalling the zenoss and zenoss-core-zenpacks rpm.  Adding the user turned out to be insufficient, to uninstall the rpm you have to add the user and group with the same name and uid/gid.  However, the ordeal was not over yet.  Turns out, to uninstall the zenoss-core-zenpacks, you must have zenoss running and a particular python process listening on TCP/8100.

Finally, I managed to uninstall the whole mess and decided to start fresh with all the rpm defaults.  Accepting all the rpm defaults resulted in success and I was finally greeted with a functioning monitoring portal.

Reflection
I decided to call it a (very unsatisfactory) night.  I woke up this morning and couldn't help wonder if Zenoss is worth using.  If the installation, customization and uninstallation could be so poorly engineered, I can only imagine what issues might creep up in the future.  I have decided to uninstall Zenoss and continue my search for a monitoring solution.

Feedback
This post won't be constructive criticism if I did not offer up some solutions, so here are my 2 cents:

  1. Please don't hard code configuration values in multiple files.  Why not designate one python file which is referenced by all other python scripts (pre_init, post_init, upgrade etc.)?  Similarly, don't hard code configuration values in init.d script, a better option would be to create a /etc/sysconfig/zenoss file.
  2. When adding a user, please add it as a system user since the role of the user is to run system services.  Also, there is no reason to assign this user a home directory in /home.  /opt/zenoss will be a more appropriate choice.
  3. I am not a fan of running services with an interactive login shell such as bash due to security reasons.  I did not dig deep enough to determine if this was a necessity, but I am sure if daemons like Apache can run without an interactive shell, so can a simple monitoring application.  I think /sbin/nologin is an appropriate choice.
  4. Please reference the user added by the rpm consistently.  I noticed some references were using UID, while others were using username 'zenoss'.
  5. Newer PCI compliance standards require us to not accept vendor defaults when it comes to settings such as passwords.  So it is very important to easily allow a system administrator to change settings such as database user, database password etc.  I think web applications such as Drupal have done a fantastic job on installation wizard.
  6. Please update the zenoss-core-pack rpm to check to see if the zenoss service is running, and if not, proceed with the uninstalltion.  If the service is necessary for uninstallation, then either consider starting it or adding such dependency to the '%preun' section in the rpm spec file.
  7. Please add a zenoss library config to /etc/ld.so.conf.d folder.

Cheers & Happy Thanksgiving!,
VVK

Saturday, December 5, 2009

Unintrusive RPM Distro Audit

As a consultant, I am often faced with an unfamiliar Linux system (usually RHEL). I always find it useful to understand which files that shipped with rpm packages have been modified, since it is usually a good indicator of what customizations have been performed on the system. To determine the modified files, I simply run:

% rpm -qa | xargs rpm --verify --nomtime | less

# Sample output:

missing /usr/local/src
.M...... /bin/ping6
.M...... /usr/bin/chage
.M...... /usr/bin/gpasswd
....L... c /etc/pam.d/system-auth
.M...... /usr/bin/chfn
.M...... /usr/bin/chsh
S.5..... c /etc/rc.d/rc.local
S.5..... c /etc/sysctl.conf
S.5..... c /etc/ssh/sshd_config
S.5..... c /etc/updatedb.conf

The following is taken from the rpm man pages (Verify Options section):

c %config configuration file.
d %doc documentation file.
g %ghost file (i.e. the file contents are not
included in the package payload).
l %license license file.
r %readme readme file.

S file Size differs
M Mode differs (includes permissions and file type)
5 MD5 sum differs
D Device major/minor number mismatch
L readLink(2) path mismatch
U User ownership differs
G Group ownership differs
T mTime differs

Using this trick, I can quickly determine what configuration files have been modified as well as any metadata modifications (ownership, link etc.).

Cheers,
VVK

Friday, October 2, 2009

3rd Party RPM Packages

While auditing an RPM based system, I started wondering what would be the easiest way to determine all the packages installed from 3rd party repositories, such as rpmforge.  Such knowledge is often useful, since some of these repositories contain packages which replace standard packages (example: php packages from remi repository).

I did not find any RPM query flags to query based on Vendor or Packager.  Which meant, extracting the information using queryformat and then parsing the output.

rpm --querytags provides a list of tags you can request in your queries.  For my purposes, I could use VENDOR and PACKAGER.

To find all 3rd party packages you can run:

~]# rpm -qa --queryformat '%{NAME}: %{VENDOR}\n' | grep -v ": CentOS"
gpg-pubkey: (none)
rpmforge-release: Dag Apt Repository, http://dag.wieers.com/apt/
gpg-pubkey: (none)
ncftp: Fedora Project
segatex: Dag Apt Repository, http://dag.wieers.com/apt/
python-xattr: Dag Apt Repository, http://dag.wieers.com/apt/
gpg-pubkey: (none)
gpg-pubkey: (none)
ipython: Fernando Perez

As you can see from the output above, I have ignored packages vended by CentOS.

Similarly, to select packages packaged by Dag Wieers, you can run:

~]# rpm -qa --queryformat '%{NAME}: %{PACKAGER}\n' | grep Dag
rpmforge-release: Dag Wieers
python-xattr: Dag Wieers

Cheers,
VVK


Wednesday, July 22, 2009

Migrating OTR keys from Pidgin to Kopete

If you are trying to maintain your existing OTR (off-the-Record Messaging) keys and fingerprints between Pidgin and Kopete , you might run into a small road block. Simply copying your ~/.purple/otr.fingerprints and ~/.purple/otr.private_key to ~/.kde/share/apps/kopete_otr won't cut it. It seems there are some minor changes between the formats used by the two messaging clients. Here is how you can migrate your OTR settings:

In the following example, I am migrating from Pidgin to Kopete.
  1. Copy key and fingerprint files

    $ cp ~/.purple/otr.private_key ~/.kde/share/apps/kopete_otr/privkeys
    $ cp ~/otr.fingerprints ~/.kde/share/apps/kopete_otr/fingerprints

  2. Fingerprints and keys file will need some minor tweaking. You must edit both files and change all references to the following (vi example %s/prpl-aim/AIM/g):
    • (protocol prpl-aim) to (protocol AIM)

    • (protocol prpl-msn) to (protocol "WLM Messenger")

    • (protocol prpl-jabber) to (protocol Jabber). Additionally, remove the "/Home" or whatever resource name you used in Pidgin. So the Gtalk name parameter should look like (name "user@gmail.com"), and not (name "user@gmail.com/Home").

    • (protocol prpl-yahoo) to (protocol Yahoo)

    • (protocol prpl-icq) to (protocol ICQ)

  3. Save both files and restart Kopete.

  4. Go to Settings > Configure > Plugins > OTR and click on the tiny wrench icon. For each "Account" you should see your old "Fingerprint".

  5. If this trick does not work, then simply generate new keys and then compare the privkeys and otr.privkeys files and figure out the difference.
Note: Pidgin version 2.5.8 and Kopete version 0.70.4

Happy secure chatting!
VVK

Saturday, April 18, 2009

Apache + rotatelogs

While trying to enable rotatelogs feature on Apache (httpd-2.2.3-22.el5.centos) on CentOS 5.3, I ran into a very odd issue.

httpd.conf
LogLevel warn
ErrorLog "|/usr/sbin/rotatelogs -l logs/error_log 5M"
CustomLog "|/usr/sbin/rotatelogs -l logs/access_log 5M" combined
When attempting to restart apache, I kept receiving the following error message:
# /etc/init.d/httpd start
Starting httpd: 1 Previous file handle doesn't exists logs/error_log.1240101000 [ OK ]

[root@ofmon02 httpd]# /etc/init.d/httpd restart
Stopping httpd: [ OK ]
Starting httpd: 1 Previous file handle doesn't exists logs/error_log.1240098834 [ FAILED]


Apache instances would start, but no log file is created. apachectl configtest did not report any issues. Few times apache won't start at all

It turns out that when using rotatelogs, you have to specify absolute paths, which makes sense since the logs are getting piped to rotatelogs, and rotatelogs would not know what logs/error_log or logs/access_log refers too.

httpd.conf
LogLevel warn
ErrorLog "|/usr/sbin/rotatelogs -l /var/log/httpd/error_log 5M"
CustomLog "|/usr/sbin/rotatelogs -l /var/log/httpd/access_log 5M" combined

# /etc/init.d/httpd start
Starting httpd: [ OK ]



Cheers,
VVK

Friday, April 3, 2009

Zabbix & libcurl on x86_64 platform

While building Zabbix 1.6.3 from source on CentOS 5 on x86_64 architecture, I ran into a very strange issue. The following configure line kept giving me errors about libcurl not found.

./configure --prefix=/opt/zabbix --enable-server --with-mysql --with-net-snmp --with-jabber --with-libcurl --enable-agent --disable-ipv6 --with-ldap --with-openipmi
...
...
checking for the version of libcurl... 7.15.5
checking for libcurl >= version 7.13.1... yes
checking for main in -lcurl... no
configure: error: Not found libcurl library


I checked to make sure I had curl and curl-devel installed and sure enough I did. Not only did I have curl - 7.15.5-2.1.el5_3.4.x86_64 & curl-devel - 7.15.5-2.1.el5_3.4.x86_64, but I also had the i386 version.

It took me a long time to figure out that mixing i386 and x86_64 packages is not a good idea. To resolve the issue, I went back and uninstall every i386 package I had installed, many of which were pulled in as dependencies. The problem started when I started to install package without specifying the architecture. For example, instead of running "yum install package", try "yum install package.x86_64". Following are the list of i386 rpms I had to manually remove and install the x86_64 version of them.

[root@crash ~]# cat /var/log/yum.log | grep "Apr 02" | grep i386
Apr 02 20:33:27 Updated: e2fsprogs-libs - 1.39-20.el5.i386
Apr 02 20:33:48 Updated: libselinux - 1.33.4-5.1.el5.i386
Apr 02 20:33:55 Updated: krb5-libs - 1.6.1-31.el5.i386
Apr 02 20:34:05 Installed: mysql - 5.0.45-7.el5.i386
Apr 02 20:34:10 Installed: e2fsprogs-devel - 1.39-20.el5.i386
Apr 02 20:34:13 Installed: libsepol-devel - 1.15.2-1.el5.i386
Apr 02 20:34:15 Installed: libselinux-devel - 1.33.4-5.1.el5.i386
Apr 02 20:34:18 Installed: keyutils-libs-devel - 1.2-1.el5.i386
Apr 02 20:34:22 Installed: krb5-devel - 1.6.1-31.el5.i386
Apr 02 20:34:29 Installed: zlib-devel - 1.2.3-3.i386
Apr 02 20:34:43 Installed: openssl-devel - 0.9.8e-7.el5.i386
Apr 02 20:34:50 Installed: mysql-devel - 5.0.45-7.el5.i386
Apr 02 20:36:17 Updated: elfutils-libelf - 0.137-3.el5.i386
Apr 02 20:36:23 Updated: popt - 1.10.2.3-9.el5.i386
Apr 02 20:36:25 Updated: net-snmp-libs - 1:5.3.2.2-5.el5.i386
Apr 02 20:36:45 Updated: rpm-libs - 4.4.2.3-9.el5.i386
Apr 02 20:37:17 Installed: net-snmp-devel - 1:5.3.2.2-5.el5.i386
Apr 02 20:37:54 Installed: gnutls - 1.4.1-3.el5_2.1.i386
Apr 02 20:42:02 Updated: libgcc - 4.1.2-44.el5.i386
Apr 02 20:43:49 Installed: libidn - 0.6.5-1.1.i386
Apr 02 20:43:51 Installed: curl - 7.15.5-2.1.el5_3.4.i386
Apr 02 20:43:54 Installed: cyrus-sasl-lib - 2.1.22-4.i386
Apr 02 20:43:57 Installed: openldap - 2.3.43-3.el5.i386
Apr 02 20:44:01 Installed: cyrus-sasl-devel - 2.1.22-4.i386
Apr 02 20:44:06 Installed: glib2 - 2.12.3-4.el5_3.1.i386
Apr 02 20:44:08 Installed: gdbm - 1.8.0-26.2.1.i386
Apr 02 20:44:22 Installed: OpenIPMI-libs - 2.0.6-11.el5.i386
Apr 02 20:44:29 Installed: openldap-devel - 2.3.43-3.el5.i386
Apr 02 20:44:35 Installed: curl-devel - 7.15.5-2.1.el5_3.4.i386
Apr 02 20:44:42 Installed: OpenIPMI-devel - 2.0.6-11.el5.i386


After fixing the packages, I ran "make clean" and then ran the ./configure line again for a successful compile.

Bottom line, don't mix x86_64 and i386 packages, unless you absolutely have to.

Cheers,
VVK

Tuesday, March 17, 2009

jpackage-utils incompatibility

If you receive the following error on CentOS 5 or RHEL 5 machine, chance are you have installed jpackage-utils from jpackage17-generic repository.

---> Package java-1.4.2-gcj-compat.x86_64 0:1.4.2.0-40jpp.115 set to be updated
--> Processing Dependency: /usr/bin/rebuild-security-providers for package: java-1.4.2-gcj-compat
--> Processing Dependency: /usr/bin/rebuild-security-providers for package: java-1.4.2-gcj-compat
--> Finished Dependency Resolution
Error: Missing Dependency: /usr/bin/rebuild-security-providers is needed by package java-1.4.2-gcj-compat


You have to pick either the jpackage-utils provided by the distro or the jpackage repo version due to incompatibility ( see http://www.redhat.com/archives/rhl-devel-list/2006-March/msg00723.html).

Try (at your own risk):
# rpm -e --nodeps jpackage-utils
# yum --disablerepo=jpackage-generic install jpackage-utils

Keep in mind, any packages which depend on the jpackage repo version of jpackage-utils might not work after the above change.

Cheers,
VVK