wehuberconsultingllc.com

Yesterday I learned an important lesson about IIS logs. They do not show you all of the requests hitting your server! Evidently the log does not show canceled requests.

Over the last couple of days I was receiving complaints about slow responses from our web site. By the time I would look at the CPU utilization it would be within the normal range. I looked in the IIS log file for timeouts but could not find any. So I ran a two hour trace on Thursday afternoon. Friday morning I crunched the trace with PAL and discovered several unaccountable CPU peaks attributable to w3wp.exe. An Internet search for “w3wp.exe high cpu usage”  led me to this thread in which several people recommended using IISPeek to find the misbehaving request. So I installed a trial version of IISPeek and started watching the transactions coming in. Pretty soon I saw something I was not expecting. A shopping site was coming to our site and trying to retrieve a  product advertising feed. What was surprising was not that the shopping site was retrieving the feed but there was no log of it in the IIS log file.   This request had been consuming our CPU for several minutes and then disappearing without a trace. I knew this query had some serious performance issues but the IIS log indicated that it had been working on previous days. I did not know it was running so often. With IISPeek and Task Manager running together I could see the impact on the site. Evidently this particular shopping site would time out or cancel the query before our site either returned the data or timed out.  It was at this moment that I figured out that IIS must have a “no harm, no foul” policy about canceled queries. My reliance on the IIS log was a mistake in this case. Since this shopping site was not getting the data, the shopping site would try again at a later time. When I was watching it with IISPeek I was seeing this request about every fifteen minutes. Fortunately I had already developed some web page caching code I could implement quickly and get us over this hump. Within an hour the shopping site had its data and our web site was back to normal. I have solved a lot of web site problems by looking at the IIS log but in this case it was not the right tool for the job. On Monday I am buying a copy of IISPeek!

In December 2008 I was asked to clean up some problems with our newsletter at work.  We had over 100,000 people on the mailing list and over 90% of the people on the list have ordered from our website. After a little bit of analysis I determined that we were bouncing 30% of our newsletter emails because we had failed to follow the most basic rules of newsletter management and the automated newsletter cleanup procedures did not work. So here is my list of tasks I used to cleanup the newsletter and get the bounce rate down to 0.1%.   In our case we are sending the emails out from a dedicated server at our office.

1. Use a static IP to send out the newsletter.  One of the first SPAM checks email providers use is to see whether the  IP you are using to send out the newsletter is coming from your domain.  This means that you need to set up a sub-domain(e.g. mailserver.mycompany.com) and a PTR record for the sub-domain.
   1. Set up a sub-domain for the static IP. Since our web server is hosted we had to ask our host provider to set up the sub-domain.
   2. Set up a PTR record for reverse DNS lookup. I asked the folks who provided us our static IP to set up the PTR record to the sub-domain.
   3. If everything is set up correctly you should be able to pass the reverse DNS lookup test. This is the site, Forward Confirmed Reverse DNS Lookup Test, I used to confirm it was working properly.
2. Set up feedback loops if you can. Feedback loops are pretty dumb idea that email providers like. I dislike them since I have only feedback loop that is useful. I am grateful that AOL has made it reasonably easy for me to remove people who do not want to be on our mailing list. I was able to quickly modify our existing newsletter template to embed an unsubscribe link that would make it through the feedback loop processing. On the other hand I found I wasted a huge amount of time trying to set up a feedback loop with  Yahoo and Hotmail. Both Yahoo and Gmail  want you to sign your emails with DKIM or they will not talk to you. DKIM was supposed to reduce SPAM but I have not seen any reports showing it reducing SPAM. Implementing DKIM will require me to set up a new email server for the newsletter so it is pretty far down my priority list. So far I have been able to ignore this issue. Hotmail wanted me to get a letter from our local internet provider saying we were the only folks using the static IP. Our local internet provider, RoadRunner, told me several times that no customer had ever requested a letter like that and they were not going to provide it. I set up a feedback loop with Comcast but after a couple of months they increased the amount of information they redacted from the email and broke the unsubscribe loop in the feedback loop message. 
3. Embed an unsubscribe link in your newsletter template that will unsubscribe the user but does not require the email address.  Since most feedback loops redact the email address, this will allow you click on the link in the Feedback loop message to unsubscribe the user.
4. Go through the error log on your email server and look for the messages that indicate that email address is inactive or no longer used. Unfortunately there is a multitude of messages that are used to describe unknown users(5.1.1, 5.5.1, unknown user, alias not found).  This is one area that begs for a standard. This is where the feedback loop should have been.
5. Manually go through your newsletter inbox and look for:
   1. Earthlink, PeoplePC, Zonealert, and other verification replies
   2. Unknown user messages.
   3. Feedback loop messages
   4. Changed addresses and unsubscribe messages.
   5. Other replies.
      1. Mailbox full, Out of office replies
      2. Customer service requests. About once a week we get a reply to the newsletter that asks a question about a product.
      3. Miscellaneous SMTP problems(e.g. DNS and email forwarding problems)

I am not sure how I got this 2.2 GB folder on my “C”  partition but it was not necessary. With free space on my “C” partition down to 5% it was time to clean house. The Vista instructions will work as written if you run the Disk Cleanup utility as the Administrator. If you happen to run the utility as a “mere mortal” there is a button in the Windows 7 version to restart the utility as an Administrator if you want to “Clean up system files”.

How to remove the Windows.old folder that is generated when you perform a custom installation of Windows Vista

This week I implemented a grid style application using Classic ASP and Jeditable. The hardest part of implementing this application was trying to figure out what a save.asp version of save.php would look like. Here the template I created.

<%
dim sID, sValue,errorcode dim field1,field2,field3,field4
'The sID is a spreadsheet style ID
'As an example B3 would be the second editable field
'for DB ID field = 3
sID = request("id")
sValue = request("value")
sType = mid(sID,1,1)
sDataID = mid(sID,2)
'We have four editable fields
'The changed field will not be null
field1 = null
field2 = null
field3 = null
field4 = null
errorcode = 0
select case sType
case "A"
field1 = sValue
case "B"
field2 = sValue
case "C"
field3 = sValue
case "D"
field4 = sValue
case else
errorcode = 1
end select
if errorcode = 0 then
'Validate and update the data base
end if
if errorcode = 0 then
'Send back the value field
Response.Write sValue
else
Response.Write "<b>!Error " & errorcode & "</b>"
end if
%> 

This article was timely. I had just installed virtual version of Ubuntu on my ESXi server and set up VNC so I could access it. It was okay but FreeNX is a more elegant solution. The combination of FreeNX and Firehol to setup the firewall makes it a winner in my book.

How To Set Up A Terminal Server In Linux Using Ubuntu 9.10 And FreeNX

FreeNX is an open source implementation of NoMachine’s NX Server. It is a bit more akin to Microsoft’s RDP protocol that the usual VNC, so while keeping bandwidth to a minimum, it maintains good visual quality and responsiveness.

How To Set Up A Terminal Server In Linux Using Ubuntu 9.10 And FreeNX
(author unknown)
Mon, 25 Jan 2010 16:42:09 GMT

I think I can finally say that I have finished the upgrade, Free Cell is installed . This summer I installed Windows 7 RC and was pleased with the performance and the look and feel on my 3 year old laptop. It would have been nice if I could have just upgraded  the RC version but I was going the Professional version rather than to the Ultimate version. Since I had previous partitioned my disk and cleaned up the disk space I was in pretty good shape for a clean install. The hardest part of the install was install the device drivers for the old QMS printer and Epson scanner. Support for these devices was not included in Windows 7 so I had to install the old XP drivers using the XP compatibility mode.

A couple of weeks ago I installed the Network Monitoring Appliance using the tutorial on HowToForge.com. Prior to installing the Network Monitoring Appliance I was planning to give the latest community version of GroundWork Monitor, http://www.groundworkopensource.com/products/community-edition/index.html another trial. My network monitoring objectives were to have the Network Monitoring appliance notify me of problems on a remote web server and on my local network. Although these network monitoring objectives can be accomplished by a ping or a “HTTP ping”, I wanted to see a some network throughput graphs and I expected to eventually need a slightly more sophisticated data base monitoring in the near future. Nagios was at the core of the best solution for me since accomplished most of my needs and I was already familiar with Nagios from a previous trial of Groundwork Monitor. The primary attraction of the Network Monitoring Appliance over Groundwork was its much smaller resource requirements. In my environment it would be sharing a VMware ESXi server. I was also pleased to see that the Network Appliance used Jeos. For those unfamiliar with Jeos it is:

Ubuntu Server Edition JeOS (pronounced "Juice") is an efficient variant of our server operating system, configured specifically for virtual appliances.

Users deploying virtual appliances built on top of JeOS will benefit from:

• better performance on the same hardware compared to a full non-optimized OS
• smaller footprint of the virtual appliance on their valuable disk space
• fewer updates and therefore less maintenance than a full server installation

For my installation I decided to use VMware’s 32-bit Ubuntu template to create the virtual machine. The only modification to the template was to adjust the disk drive size down from 8 GB to 1 GB. As described in HowToForge tutorial I installed the following programs.

1. Ubuntu 8.04.3 JeOS as OS
2. Nagios 2.11 for monitoring and alarming
3. Smokeping 2.3 to observe latencies and packet loss
4. MRTG 2.14.7 to observe network traffic’s tendencies
5. RRDTool 1.2.19 as the Round-Robin Database for storing all measurement data
6. Lighttpd 1.4.19 as a fast, lightweight web server frontend
7. Weathermap4rrd for illustrating the network weather
8. sSMTP as extremely lightweight MTA for mail delivery

The installation was quick. Almost all of my challenges was in configuring the programs. Fortunately I had previous experience configuring the most difficult to configure programs, Nagios and MRTG. It helps if you have a basic knowledge of PERL since most of programs use it. Here are my installation notes.

1. One of the first things I needed to install to make this installation go smoother was an editor other than VIM so I could cut-and-paste from the tutorial to my SSH session. In my case I installed nano.
2. The first application I configured was smokeping. The configuraton file is pretty easy to figure out and can be found at /etc/smokeping/config.  If everything works you will see a nice graph of the the ping statistics at http://yourip/cgi-bin/smokeping.cgi.
3. Configuring Nagios is a bit more complicated. Since this is version 2 of Nagios, the configuration files are located at /etc/nagios2/conf.d. The main Nagios web page is at http://yourip/nagios2/. The Nagios QuickStart Document, http://nagios.sourceforge.net/docs/3_0/quickstart.html, is a good primer for the folks not familiar with Nagios.
4. The Debian logo did not appear in Nagios next to the localhost. It showed a missing image. After a little research I figured out that I needed to install nagios-images using apt-get install nagios-images.
5. For some reason I did not seem to have cron installed and running. This is easily solved by apt-get install cron.
6. MRTG is useful if you have a SNMP router to poll. I used my pfSense Firewall as the SNMP source. MRTG provides some nice graphs of network traffic and its page is located at http://yourip/cgi-bin/mrtg-rrd.cgi/
7. Configuring Weathermap4rrd is a little challenging since the documentation is sparse. Weathermap4rrd provides a clever network status graph once you figure how to configure it. It uses the same data as MRTG to create its graph. The network status page for weathermap4rrd is located at http://yourip/weathermap4rrd/weathermap.png
8. I installed apticron to nag me via email about installing security updates and Logwatch to find any problems posted in the log file by the installed programs.
9. If you plan on getting emails from Nagios when a host is down, you should test it. Duh! The easiest way to test it is to deliberately mistype the host name. If you do not get the email, you should check your Nagios configuration, sSMTP configuration, and the SMTP log file.
10. sSMTP is easy to configure and use. In the simplest configuration you point it at the SMTP server you are sending your emails to. If you are sending emails to more than one domain, you need to connect to a SMTP server that will relay emails for you.
11. I installed PHP version 5 to see how hard it would be to install under Lighttpd. I followed the instructions on the Lighttpd wiki and PHP appears to be running without problems. Most of these network monitoring programs have newer versions in PHP. Some day in the future I plan to migrate to the PHP versions of Nagios and weathermap but it is not necessary for this small network.
12. I created a simple navigational menu on the main page with links to the various network management status pages. It is much easier to use this menu then remembering the addresses of the different status pages.

Jason left a comment on a previous post about wanting to see the script I am using to email NTBackup log files. Recently I converted the script to powershell from vbs. Here is the old file.

#**************************************************
# Script Name: 	Ntbackup_E-Mail_Alert
#     Version: 	1.0
#      Author:	Bill Huber
#Last Updated:	19.Nov.2009
#
#     Purpose:	Concatenates two or more log files into the body of an email message. I schedule
#				this script to run at a time the backup job should be finished and to send me
#				the latest NTBackup log files as an email with a somewhat informative subject field.
#
#		Legal:	Public Domain.  Modify and redistribute freely.  No rights reserved.
#              	SCRIPT PROVIDED "AS IS" WITHOUT WARRANTIES OR GUARANTEES OF ANY KIND.
#              	USE AT YOUR OWN RISK. NO TECHNICAL SUPPORT PROVIDED.
#**************************************************

# Customize the following variables for your SMTP server, email from address,
# email address the message is going to, the minimum log size, and the log path.

$SmtpServer = "mySBServer"
$From = "mySBServer Administrator <administrator @myCompany.com>"
$To = "billhuber@myCompany.com"
$intLogSize = 1000 		#If the log file is less than this size, the backup probably failed
#The following variable point to the log file location
$logpath = "C:\Documents and Settings\Administrator\Local Settings\Application Data\Microsoft\Windows NT\NTBackup\data\*.log"

# End of Customization

$SmtpClient = new-object system.net.mail.smtpClient
$SmtpClient.host = $SmtpServer 

#Get the filenames and other stuff for the last two log files
#We are going to concatenate the last two log files into the body of the email message

$a = get-childitem $logpath | sort-object lastwritetime | select-object -last 2
#Get the last write time for the last report for the subject line
$b = $a | sort-object lastwritetime -descending | select-Object -first 1
$c = $b.LastWriteTime
$d = $b.length

$Title = "SUCCESS - Full Backup at $c"
if ($d -lt $intLogSize){
	$Title = "ERROR - Full Backup Failed at $c"
	}
$Body = ""
 foreach ($line in Get-Content $a)
    {
    $Body += "$line `n"
    }

$SmtpClient.Send($from,$to,$title,$Body)

I found when I was updating our web site I would run a VB script to open three Explorer windows. I have chosen to use Q-Dir to replace the script since it actually does a better job with screen real estate and allows me to open four windows. I use the the portable version to avoid installation headaches.

Sometimes when looking through a file manager, it would be nice to have more than a dual-pane view. Now you can manage your files with up to four viewing panes at once with Q-Dir.

Note: Q-Dir is available in regular install and portable versions.

Q-Dir – Multi-Pane File Manager :: the How-To Geek

When working on other people’s code it sometimes difficult to figure where the data is coming from and you really don’t have the time to spend figuring it out. We were having a problem with an ASP page that was blowing up when a certain XML field was empty so I wanted to simple command to dump the raw XML. Either the XML field had a different name or it wasn’t in the XML file. I knew the command must exist but it was surprising difficult to find. Here is what I used to dump an XML object called xmlData.

response.write xmlData.documentElement.xml

OBTW the element was not in the XML file.

Python(x,y) is a free scientific and engineering development software for numerical computations, data analysis and data visualization based on Python programming language, Qt graphical user interfaces (and development framework) and Eclipse integrated development environment.

Although I would say I am conversant in Python and can see why a lot of people like it, it is not necessary for any of my job functions. In fact I recently converted the only python program used at work over to PowerShell. It was a trivial program that has been written a million times in a multitude of scripting languages. In this case it had a bug so it was a fairly trivial exercise to convert it over to Microsoft’s favorite scripting language.

Scarcely could I imagine that I would be seriously playing with python just a couple of weeks later. The trigger for this event was a blog post on SQLServerCentral.com called Python for the SQL Server DBA. In the article I was intrigued when the author said he used Python(x,y). I had not heard of it so I checked out the web site, python(x,y) – Python for Scientists, and decided to convert an Excel spreadsheet graph over to python. The graph is a fairly standard multiple line plot of time data. This is the type of graph you can create in Excel in about five minutes.

It took a lot longer to create the graph in python but I am not disappointed. Much of my time was spent learning how to manipulate Matplotlib to achieve the desired graph. Matplotlib is a library for making 2D plots of arrays in Python and looks a lot like MATLAB™ . Since my knowledge of Matlab was nil, I had a lot of catching up. The flexibility of Matplotlib to customize a graph reminded me a lot of SAS/GRAPH. That is both the good and bad news. Although Excel has a lot of graphing options and I recommend it for most graphing requests, there is always some option it does not do quite right. Matplotlib overcomes those problems with lots of customization options and can be used to create some pretty exotic graphs. The bad news is there is a significant learning curve in understanding how to use those options.

Almost of all of my development for this simple graph program was done in IPython although more interactive environments like Eclipse and Spyder were available. In hindsight I would probably prefer Spyder to develop my next program. Most of my work is not very sophisticated and the lightweight integrated IDE of Spyder appealed to me more than Eclipse. Eclipse is still relatively slow at starting up. When I look at the whole python(x,y) download, the greatest contribution is the breadth of the products included in its download. You can start your work from the command line for simple programs like I did and progress all the way up to fairly comprehensive graphical user interface using QT and Eclipse for sophisticated programs. The python development has come a long way.

I have been seeing this message on and off with attachments for some time now in outlook:
Can’t Create file: VoiceMessage.wav. Right-Click the folder you want to create the file in, and then click properties on the shortcut menu to check your permissions for the folder

In my case the person having the problem was trying to bring up Word Document located in an Exchange shared public folder. It looked like a duplicate file problem but a file search did not show up any duplicates. I even told it to search through hidden files and folders. So I went to the Internet and found the solution you can read by following the link below. Lo and behold we had a 100 copies in the OutlookSecureTempFolder. You can find the folder location by looking in:

HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Outlook\Security

Can’t Create File – Outlook Error, and a the Fix

Sometimes I forget how I fixed problems in the past and this error is an example. In my case the fix was simple. I followed the instructions in http://support.microsoft.com/kb/867628 and increased the threshold from 104857600 to 704857600. I made an educated guess at this value.

Here’s a big thank you to Marc for this post on Fast_Forward Cursors. I happened to be debugging a performance problem with a SQL2000 stored procedure today. We have a stored procedure we use to release back orders and it was timing out. When I looked at the execution plan, SQL was using the date index rather than our order status/date index with the query.  Since SQL was using the date index, it had to read every order to determine the order status. When I added the FAST_FORWARD parameter to the CURSOR, it used the order status index with the query and the I/O cost dropped from 1119 down to .228.

Tips, Tricks, and Advice from the SQL Server Query Processing Team : Understanding SQL Server Fast_Forward Server Cursors

Consider the following scenario:

• You connect a Windows Vista-based computer to a network.
• A router or other device that is configured as a Dynamic Host Configuration Protocol (DHCP) server is configured on the network.
• The router or the other device does not support the DHCP BROADCAST flag.

In this scenario, Windows Vista cannot obtain an IP address.

Consider the following scenario:

• You just bought your son a new Toshiba tablet with Vista Business pre-installed on it. You want to make sure that when the tablet goes off to school it has all critical patches applied.
• You are using pfSense as a DHCP server.
• The XP, Linux, and Windows 7 computers are able to connect to the network and get an IP address.

You would have thought DHCP problems were solved twenty years ago but here is the solution.

Windows Vista cannot obtain an IP address from certain routers or from certain non-Microsoft DHCP servers

On the last Friday of June I came into the office and found myself confronted with one of the more unique disaster recovery scenarios I have ever encountered. The primary symptom was that no one could get to their email. The cable modem and the router attached to it looked like they were turned off. Turning the power on got the lights to blink for only a few seconds before they went off. Our Small Business Server server appeared to be working but the lights on the LAN adapters in the back were off. Even when we inserted a known good LAN connection the lights would not come on. I rebooted the server and it stopped seeing the disk drive array. It was about that time I noticed the aroma of burnt insulation. Ah! The smell of burnt insulation in the morning! The cable modem, router, and server were fried. During the night the RoadRunner cable serving us must have been hit by lightning.

It is not surprising that we were not prepared for this disaster recovery scenario. Here are my notes on how we recovered our original server and migrated it to a new server.

1. Probably one of the more interesting aspects of this disaster recovery story was that I took half of a mirrored drive pair and used it to create a Virtual Server of the SBS server. Since our development server(HP DL380) is similar to the server(HP DL360) that got fried, I was able to put the mirrored drive into the development server chassis. Everything on the drive looked good except for Exchange partition. The Exchange partition was missing. The lightning strike occurred during our backups, so our backups were not complete. We were looking at losing Thursday’s email. Since the development server had sufficient processing power and disk space, I decided to see if we could bring up the Small Business Server as a Virtual Machine. Using a virtualized server could allow our office to be fully operational  while we worked on getting a new server delivered. It looked like a fast way to recover the Active Directory and the office email. So I gave VMware vCenter Converter a try and I was amazed that the Small Business Server came up with only minor errors. The Exchange software complained that it could not find the Exchange partition and the HP diagnostic software complained about the hardware. Other than those problems the active directory, print queues, and the fax server were all operational.
2. The good news was that I had a virtual server running. The bad news was having problems recovering Exchange. The backup located on an external USB drive was restoring with errors. The first time I tried to restore Exchange I got a file corruption problem. This probably was due to USB problems with virtual servers. The next morning I decided to try something different. I downloaded some partition recovery software off of the Internet and to my surprise it found the partition on the mirrored drive. Using the EASEUS Data Recovery Wizard, I was able to recover the Exchange partition. The database had some integrity problems but it looked promising. So I followed this article, Using the Exchange tools ISINTEG and ESEUTIL to Ensure the Health of your Information Store, to repair the Exchange database. My final trick to getting Exchange to mount was to delete the Exchange log files. A little bit more than 24 hours after the lightning strike, our Small Business Server was operational and we had not lost any emails.
3. About a week later we had a “new” server delivered. Actually it was an old server we got off of eBay but it was identical to the server that had failed. Although it was tempting to leave the SBS server in virtualized form, we opted to install the server natively using the SBS Migration procedure. In this case both our SourceDC server and the MigrationDC server were virtualized servers.The first time I tried the SBSMigration procedure I failed. It took me awhile to figure out why but the SYSVOL share was not getting created on the MigrationDC. I traced the problem back to a communication configuration problem. The DNS parameter on the LAN adapter configuration for the SourceDC was pointing at the office router rather than itself. Although normal communications with the server appeared to be working fine, the active directory communications with the backup domain controller was not working. The domain controller could not find itself. ;(  After I changed the DNS parameter the domain communications and file replications worked correctly. As Jeff Middleton reminded me, a good indication that the backup Domain Controller is working properly occurs when the SYSVOL share is created on the MigrationDC sever.
4. My next mistake was installing the Exchange database to a new drive letter. Exchange is very finicky about this. I had to “repair” Exchange to get it to recognize the database at the new location. It was after I had started the repair operation that I figured out how long the repair was going to take. I ended up running the repair overnight. In hindsight we would have been up and running much earlier if I restored it to its original drive letter location and moved it to a new drive letter at a later time.
5. My final mistake was made when I upgraded the NewDC to Windows 2003 SP2 before completing the SBS installation. I had to uninstall SP2 and install SP1 before I complete the SBS installation.

In the first installment of Adventures with iRedMail I got it to send emails but I left the MS Exchange integration for another day. Since then I have updated my DNS zone with the DKIM information, set up local DNS information, decided on naming standards, and reconfigured Postfix several times before I got it right.

Updating the DNS with DKIM information

This task was relatively easy. I copied the DKIM information in the iRedMail.tips into a trouble ticket with my web provider. About 24 hours later it was ready to test. I sent an emails to my Yahoo account, sa-test@sendmail.net, and autorespond+dkim@dk.elandsys.com. Although the email from dk.elandsys.com was the first to respond, it said it did not work. When I checked my Yahoo account the headers said the email was signed correctly with DKIM. Ironically the return email from sendmail.net ended up in my Junk Mail folder. It said that everything worked correctly. For one more test I created a Gmail account and sent an email to it, too. It said the email was signed correctly.

Local DNS, naming standards, and more Postfix problems

The next challenge was to configure Postfix to accept both local email addresses and email addresses for the exchange server under the same domain. I used PostFixAdmin to create Aliases that pointed to the Exchange server emails(e. g. myemail@mybusiness.com points to myemail@mybusiness.local). PostFix complained about the DNS records for my Exchange server so I added mybusiness.local as a relay_domain and set up a psuedo DNS so that PostFix can find the IP address for my Exchange server. In my case I decided to let my pfSense firewall act as a local DNS server to serve up the local IP addresses. At this point I can email to everyone from a local iRedMail account but I cannot get replies until I set up iRedMail as the SMTP gateway and the Exchange server as a relay domain.

PostFix domain checks get me again!

It took me a long time to figure this out. When I changed the firewall to redirect SMTP traffic to the PostFix gateway I could not get any mail. I thought I had messed up the firewall settings so I kept trying different settings. I was pretty limited with my testing tools. If I could Telnet into port 25 I could see what is happening but I could not make the connection work as long as I was located on this side of the firewall. Fortunately I found a solution on the Internet. The dnsqueries.com site provides a page, http://www.dnsqueries.com/en/smtp_test_check.php, that allows me to check my local SMTP connection using their server.  Within minutes I figured out that my email server did not like my sender’s domain. In fact it did not like anyone’s domain. This was the same type of problem I had with the Postfix recipient domain check, so I removed the sender domain check and the emails starting flowing.

What have I achieved?

• I have a gateway that checks all incoming mail for spam and viruses. Postini offers a similar service for about $1 per user per month. We use MXLogic at work.
• I have an alternate email server that allows me to send email that passes the SPF and DKIM checks. One of the reasons I investigated iRedMail was to use it for sending out a newsletter at work. Like many Internet retailers we get a chunk of our business as a result of our biweekly newsletter. In our case DKIM is another piece of the puzzle to improve our sender reputation. Since both Yahoo and Gmail require DKIM signing in order to set up feedback loops, DKIM is probably essential if you have ambitions of having a pristine email list. For those folks looking at ways to cut the umbilical cord to Microsoft this is one of several low cost, low maintenance migration alternatives to a local Exchange server.

I read this article on HowtoForge and decided to give it a try. I was not as successful as the author.

iRedMail: Full-Featured Mail Server With LDAP, Postfix, RoundCube, Dovecot, ClamAV, DKIM, SPF On CentOS 5.x Debian (Lenny) 5.0.1

iRedMail is a shell script that lets you quickly deploy a full-featured mail solution in less than 2 minutes on CentOS 5.x and Debian (Lenny) 5.0.1 (it supports both i386 and x86_64).

iRedMail: Build A Full-Featured Mail Server With LDAP, Postfix, RoundCube, Dovecot, ClamAV,SpamAssassin, DKIM, SPF On CentOS 5.x | HowtoForge – Linux Howtos and Tutorials

My first try was to use the script to update a Centos 5.3 workstation installation. It went smoothly until I tried to update look at the keys used by DKIM. I ran into trouble with the LDAP option. OpenLDAP would not install do to a missing file. So I took the Mysql option. That was when I found a series or problems. Most of the problems were minor. My initial mail userid used Chinese. Since I was particularly interested in DKIM I was disappointed to find out that Amavisd was running at a version that did not support DKIM. I quickly realized that this was taking too much time and a better solution was to install a virtual machine using the iRedOS. This is a Centos 5 installation with all of the prerequisites already installed.

Creating a virtual machine mail server went pretty smoothly. The only problem I found with the installation was that I was unable to send mail. I quickly realized that I needed to install Webmin so I could perform normal system maintenance and troubleshoot. After I installed Webmin I found my problem. Postfix thought Yahoo was an unknown domain. Although I am not familiar with intricacies of Postfix I found that if I removed the configuration parameter “reject_unknown_recipient_domain” I could send emails successfully. This is a not a fix but it will work for me until I figure out the problem between the DNS and Postfix.

My next trick is to set up the mail server as a mail relay to my Exchange server. Technically this could be a first step in migrating off of Exchange to a non-Microsoft cloud computing environment. There are a lot of good things to be said about Exchange but there are even more good things to say about cloud-based email. Making the transition to a low cost, highly dependable, feature rich email environment with the least amount of pain is the challenge for both the Microsoft and open source communities.

It has been a long time since I actively worked with Microsoft’s ISA Firewall so it took me some time to fix this problem. Buy.com periodically offers a 3 computer version of McAfee at a very cheap price. Since I am somewhat ambivalent about the merits of one virus checking software over another, I bought a copy to replace a TrendMicro version up for renewal. The installation did not flag any errors or warnings so it took about a week before I noticed that the patterns had not updated. Yesterday I decided to fix the problem and write down for posterity how I accomplished it.

Unlike many firewalls Microsoft’s firewall typically restricts anonymous access. This typically is not a problem for most applications that run on Windows computers since the users are logged into the Active Domain. Occasionally there are applications that fail to connect to the internet despite the user being logged into the domain. Most of the time you need to open some non-standard ports to fix the problem. In this case McAfee is using standard HTTP and HTTPS ports and still failing to connect.

The solution is to create an anonymous access rule to the McAfee update site and to configure the client to not use the ISA Firewall client for these sites. One way to accomplish this  is to configure Internet explorer(Tools-Internet Options-Connections-Lan settings-Advanced) to not use the proxy. This is the way I got McAfee to update. Another way is to configure the properties for the internal network in ISA to use direct access for these sites. You can configure a GPO, too.

Last weekend I took the plunge and installed Windows 7 RC. The hardest part was freeing up some disk space and partitioning the hard drive. After a few defrag runs I was ready to partition. Dual booting is the way to go. There are no special tricks. Just let Windows 7 install in the unpartitioned space. The installation was pretty uneventful. My laptop is about three years old, it has 2 GB of ram, and it passed the Windows 7 compatibility check. The installation found drivers for everything although it had to get the Ricoh drivers off of the Internet.

The part I was most interested in was what would I install first. The first four programs were were the virus checking software, FeedDemon, Windows Live Writer, and Flash. I chose a trial version of AVG available at http://free.avg.com/download-avg-anti-virus-free-edition. I like AVG but it triggers a PC issue message in Windows 7. I guess AVG and Windows have a few things to work out. It did not take to long after running FeedDemon that I ran into a web page requiring Flash. When I decided to write a post about my Windows 7 experience I installed Windows Live Writer. To setup Windows Live Writer I needed KeePass since that is where I store my passwords.

To transfer files from Window XP into Windows 7 I am using two methods. With the first method I created a shared folder at the root of the Windows 7 drive and copied files into the folder using XP. I could not browse the XP version of the “My Documents” folder using Window 7  and was not into the “take ownership” thing. The second method uses Dropbox.

So far my experience has been very positive. The interface is nice and the computer seems as fast as it was under XP. I think Microsoft has a winner if they price it right. The interface of Linux and the Mac are nice, too!