I found myself troubleshooting a Verizon PC5470 wireless broadband issue today. This setup worked a couple of months ago when I last touched it. When my boss inserted the PC5470 card into his PC, the VZAccess Manager software could not talk to the card. We noticed that his PC would create two NEC PCI to USB Open Host Controller when the card was inserted and the device manager showed that a Curitel modem was unknown. At the time I did not know what the Curitel modem was used for. When the card was installed on another PC only one NEC PCI to USB Open Host Controller was created and VZAccess manager was able to create a wireless broadband connection. After exhausting my troubleshooting tricks I found this conversation on the Internet, NEC PCI to USB Open Host Controller – Everything USB Community. Although my solution was different than their recommendation, their ideas helped lead me in the right direction. My solution was to uninstall both NEC PCI to USB Open Host Controllers and force Windows to re-install all of the drivers. Disabling or uninstalling one driver did not work for me. When I inserted the PC card, Windows proceeded to re-install the drivers as expected. First it installed two NEC PCI to USB drivers. Next it tried to install the Curitel modem driver. During this process the PC announced that it was disabling a hardware device. When the driver installation was complete the Device Manager showed that the second PCI to USB driver was disabled and the Curitel modem was installed and functioning. When I started the VZ Access manager it found the PC5470 and configured itself to use the Curitel modem. After a little bit more automatic updating by VZ Access manager, we were able to establish a wireless broadband connection. Whew!
Interesting Computer Problem
When you are grateful you have a job you can’t be picky about the applications you support. This week I spent a lot of time on my one of my least favorite applications, credit card processing. To give you a little background I was not employed at the firm when it was originally installed in 2007 so almost everything I hear is secondhand. The application is pretty simple and it runs on its own computer. It downloads credit card orders from our orders data base, transfers the authorizations over to the bank, and then updates the orders data base. Despite its relative simplicity I hear that the application was a painful install with lots of support issues. The folks before me got it to work and it seemed to be working okay until the the middle of 2008. That was when intermittent response time problems started to crop up. Attempts at technical support lead us down that primrose path again without success. The folks in technical support recommended we re-install the software but they really did not have an explanation for our response time issue. The statement that sent us in a completely different direction was when they said that our transaction volume was too large for their application. Our bank took that statement back a week later but the damage was done. The bank and the credit card processing application were going to be replaced.
This fun and games started when the credit card processing computer rebooted after the “Patch Tuesday†updates. It came up okay but it was coming up with strange errors during credit card processing. At first we could not process any credit cards but we finally got the cards processed. Since it was “working†I did not look at the problem until Friday. That was when I found out that the problems had continued on every morning and it had reached the critical stage. After spending three hours of “quality time†with an upset user(my boss) getting the credit cards processed, I was convinced that there was more to this problem than “Patch Tuesday†problems and I was determined to fix the problem today.
It did not take too long before I found a disk corruption problem. It looked like DISKCHK might fix the problem so I needed credit card processing folks to complete the end of day tasks so I could take an immediate backup and reboot. So at 530 pm on a Friday evening we rebooted. The computer would not reboot. I tried safe mode and it would not reboot. I tried to boot from a CDROM and it would not reboot. I called the boss to let him know. He said he would meet at the office on Saturday morning. This had a really good chance of being a super catastrophe. Oh well! There goes my weekend!
On Saturday it took us about an hour to identify the problems. The boss actually found the problem. I heard a unusual spin up sound on either the disk drive or a fan. My little power supply tester showed we were missing –5v. I installed a new power supply and now we could boot off of the CDROM. Unfortunately now that we could boot up properly we could confirm that the disk drive was trashed. So I installed a new drive. About two hours later I had completed re-installing the operating system and the application from my backup. An hour of testing confirmed that we would be able to process credit cards on Monday and the business would continue for another week. This is probably the first time I have seen a double failure on a PC.
Christmas WordPress Theme For Free Download | Freebies | Smashing Magazine
A big thanks to the folks at Smashing Magazine and InstantShift for offering a free Christmas theme. The festive mood of the theme motivated me to embark on a web site conversion over the weekend.
We may be a little bit late to the Christmas party, but late is usually better than never. In this post we release a very compact, magazine-like theme designed especially for events such Christmas. We hope that the theme may be useful not only for this Christmas, but can also be used in general, for various purposes. This theme was designed by InstantShift as an exclusive Christmas gift for Smashing Magazine readers.
Christmas WordPress Theme For Free Download | Freebies | Smashing Magazine
A big thanks to the folks at Smashing Magazine and InstantShift for offering a free Christmas theme. The festive mood of the theme motivated me to embark on a web site conversion over the weekend. I had been meaning to switch our farm web site, www.legacyfarmltd.com, from phpWebsite to WordPress for several months but I always found a reason to not work on it. Installing WordPress 2.7 and the Christmas theme was almost effortless compared to the process of duplicating old posts and pages. It was nice that they included the PSD for the logo. I cranked up my portable copy of GIMP and after a few minutes of re-learning GIMP/PhotoShop I had a custom logo that fit the Christmas theme. Although I will have to change the theme in the next few weeks soon since Christmas is over, the theme layout and fonts look like a good base to work from.
Oops! Could not start error
I had not looked or touched my “dogfood†server in a long time. It appeared to be happily doing it’s thing. Appearances can be deceiving. I found that my scheduled jobs were not running according to the schedule. I had about a dozen jobs with the “Could not start error†message. What a mess! I am not sure how this happened but the fix is easy. Just enter the password again!
How to troubleshoot scheduled tasks in Windows XP and in Windows Server 2003
Disabling Weak Ciphers
This week I disabled weak ciphers on our production web server. This vulnerability was escalated again this last week. This vulnerability exists when your server allows communication using SSL version 2. Less than six months ago it was identified and classified as a low risk. SSLV2 is obsolete and is not available in some of newer browsers. Most new browsers use SSLV3 by default and it is my best guess that no customer is using SSLV2. A quick survey showed that most of the major ecommerce sites do not allow SSLV2. Despite the survey my boss was reluctant to turn off SSLV2. That was solved when the PCI folks mandated that SSLV2 should not be allowed. This may sound cruel but if a customer is using a really old browser that only supports SSLV2, they must update to a new browser if they want to buy stuff off of the Internet. That just the way it is.
Here is a good resource describing the problem and how to harden a variety of web servers, “WebApp Sec: RE: SSL Ciphersâ€. Since I was primarily interested in IIS I used “How to Restrict the Use of Certain Cryptographic Algorithms and Protocols in Schannel.dll†and created a registry file to apply the changes. Here is the registry file I used. It works with all of the browsers I test with. Both Foundstone SSL Digger and our PCI scan folks like the results.
REGEDIT4 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers] [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\DES 56/56] "Enabled"=dword:00000000 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\NULL] "Enabled"=dword:00000000 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 128/128] "Enabled"=dword:00000000 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 40/128] "Enabled"=dword:00000000 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 56/128] "Enabled"=dword:00000000 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 40/128] "Enabled"=dword:00000000 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 56/128] "Enabled"=dword:00000000 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0] [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0\Client] "Enabled"=dword:00000000 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0\Server] "Enabled"=dword:00000000 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0] [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Client] "Enabled"=dword:00000000 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Server] "Enabled"=dword:00000000
Quick Picks and Pans: Right Click Menu
The folks at MakeUseOf.com have a nice article on Right-Click Menu utlities, 3 More Tools to Bring Power to the Right Click Menu (Windows). One of the tasks I perform often is changing folders so any improvement in this area is always appreciated. I had been using QTAddressBar to quickly change between folders. It looks like Vista but the process of selecting the target folder is slower than the standard address bar. So I uninstalled it and replaced it with the first utility mentioned in the article, FolderGuide. FolderGuide looks intuitive and quick. It looks like a winner for me.
System.Management.Automation.dll missing?
Recently I have been playing around with Windows PowerShell. I had this desire to synchronize the date modified field between identical files in two directories. Awhile back I had created a repository with the copied code and during the copy the date modified had been set on all of the files to the current date. Since I am working on “other people’s code†the date the code was last modified is a helpful clue in troubleshooting. Now I wanted the repository to show the correct “old†date and this looked like a good way to write my first PowerShell script. The script objective is pretty simple. For every file in my source directory I wanted to update the date modified field in my target directory with the date modified if the date modified field in the target directory is less than my cut off date. The cut off date is the date I created the repository. So if a file has not been changed since the cut off date I wanted to set it to the original date modified. After a few fumbles I got it to work. Now I can change the date modified back to the original value for the unchanged files in the directory.
Then I started thinking about comparing directories and MD5 hash files. I knew my source and target directories were good matches since WinMerge told me but I was curious whether PowerShell natively supported MD5 hashes. After a little searching I found this Bart’s post about creating a file hasher cmdlet. This was interesting and looked like a very short task so I tried to create my first cmdlet. The instructions were simple but I fumbled over a lot of minor issues.
- You need to set the execution policy. I set my code signing to RemoteSigned.
- You need to compile the cmdlet.
- You need to create/modify your profile.ps1 so you can use it every time you get into PowerShell.
The hardest one of these tasks was the second task. Trying to find the System.Management.Automation.dll took me on a wild goose chase. I knew it was probably on my machine but I could not find it. When I gave up looking for it and tried to download the Windows SDK 2008, the download barfed on me. Finally I found Raj’s post about viewing the GAC. This confirmed that the System.Management.Automation.dll was in the GAC on my machine. To make things very simple I copied the file to my PowerShell default directory and compiled the cmdlet. Later I found this recommendation by Oisin Grehan in a Vista forum in which he says since it is in the GAC, the compiler will find it without any fancy path statements.
csc.exe cmdlet.cs … /r:System.Management.Automation.dll
Re: System.Management.Automation.dll missing? – Vista Forums
I tried it and it did not work. I was able to compile using a reference to the actual GAC location. So if we combine this all together we get something like this. You execute these statements inside PowerShell.
$ref = "$env:windir\assembly\GAC_MSIL\System.Management.Automation\1.0.0.0__31bf3856ad364e35\System.Management.Automation.dll" $compiler = "$env:windir/Microsoft.NET/Framework/v2.0.50727/csc" &$compiler /target:library /r:$ref hashcmdlet.cs
I had to install the plugin outside of PowerShell with:
%windir%\Microsoft.NET\Framework\v2.0.50727\installutil -i hashcmdlet.dll
Finally I had to create a profile.ps1 in the PowerShell directory to load the snap-in and extend type system every time you get into PowerShell. This post was a great help. This is so Unix it makes me chuckle! I have not figured out what I am going to do with this new found knowledge but I learned a lot about creating custom cmdlets and it was fun!
Frequently Forgotten Fundamental Facts about Software Engineering
Here are some jewels about Software Engineering that are worth reviewing. I recommend the rest of the article. It’s an oldie but goody!
Recently I have been doing a lot of work deciphering “other people’s codeâ€. I would say the largest part of the work has been in error detection and removal. Even a seemingly comprehensive test plan does not make me terribly confident. I have been humbled by logic problems and faulty design requirements too many times.
Reliability
RE1. Error detection and removal accounts for roughly 40 percent of development costs. Thus it is the most important phase of the development life cycle.
RE2. There are certain kinds of software errors that most programmers make frequently. These include off-by-one indexing, definition or reference inconsistency, and omitting deep design details. That is why, for example, N-version programming, which attempts to create multiple diverse solutions through multiple programmers, can never completely achieve its promise.
RE3. Software that a typical programmer believes to be thoroughly tested has often had only about 55 to 60 percent of its logic paths executed. Automated support, such as coverage analyzers, can raise that to roughly 85 to 90 percent. Testing at the 100-percent level is nearly impossible.
RE4. Even if 100-percent test coverage (see RE3) were possible, that criteria would be insufficient for testing. Roughly 35 percent of software defects emerge from missing logic paths, and another 40 percent are from the execution of a unique combination of logic paths. They will not be caught by 100-percent coverage (100-percent coverage can, therefore, potentially detect only about 25 percent of the errors!).
RE5. There is no single best approach to software error removal. A combination of several approaches, such as inspections and several kinds of testing and fault tolerance, is necessary.
RE6. (corollary to RE5) Software will always contain residual defects, after even the most rigorous error removal. The goal is to minimize the number and especially the severity of those defects.
Frequently Forgotten Fundamental Facts about Software Engineering
How do I run sp_spaceused for all tables in a database? – TechNet Forums
I was looking for a way to list the size of all the tables in database again and I found this nice bit of code. I made a small change on the select statement to create a calculated integer field so I could get a nice descending sort on size. It worked for me!
select 'Database Name: ', db_name() set nocount on if exists(select name from tempdb..sysobjects where name='##tmp') drop table ##tmp create table ##tmp(nam varchar(50), rows int, res varchar(15),data varchar(15),ind_sze varchar(15),unsed varchar(15)) go declare @tblname varchar(50) declare tblname CURSOR for select name from sysobjects where xtype='U' open tblname Fetch next from tblname into @tblname WHILE @@FETCH_STATUS = 0 BEGIN insert into ##tmp exec sp_spaceused @tblname FETCH NEXT FROM tblname INTO @tblname END CLOSE tblname deallocate tblname go select nam Table_Name ,rows Total_Rows ,res Total_Table_Size ,data Data_size ,ind_sze Index_Size ,unsed Unused_Space ,CAST(replace(res,'KB','') as int) as Total_Table_Size_KB from ##tmp ORDER BY Total_Table_Size_KB desc drop table ##tmp
–Vidhya Saga
How do I run sp_spaceused for all tables in a database? – TechNet Forums
Gordon’s Tech: Make Firefox 3 beta accept the Windows Live Writer Blog This extension
I installed this extension two ways. On a PC with an old version of Live Writer Blog This installed I followed the instructions to update the install.rdf. On a PC without a previously installed version of Live Writer Blog This I installed the extension and updated the install.rdf. Nothing happened! I installed the Firefox Nightly Tester Tools add-on and the Blog This extension appeared when it restarted.
Gordon’s Tech: Make Firefox 3 beta accept the Windows Live Writer Blog This extension
Pivot Table Analysis of the Event Log
Recently I had to investigate a problem with our SMTP server. One of things I wanted to know was when the SMTP problems started. Like most computer problems multiple event IDs were were being triggered each time a problem occurred. The Pivot Table Wizard is a great tool for quickly summarizing the event log data. Here is how I did it.
- Open the Event Viewer, filter your view to the event source you are interested in, export the list, and transfer the exported list back to your work station.
- Open a new blank worksheet in Excel and import the data using the Import External Data Wizard.
- Open the Pivot Table Wizard. Drop the “Date†into the row area. Drop the “Event†field into the column area. Drop any other field into the data area. I used the “Source†field. You should now have a pivot table that has columns for each event ID and a count of the number of events per day per event ID.
- I prefer the data to sorted in descending order so I went into the Advanced Field properties for the “Date†and set it to descending.
In my case with the pivot table analysis I could see that one event ID, 4000, was the primary event. The rest of the event IDs were secondary events.
Experimental Mail Server Analyzer Online
Dan Kaminsky wrote a post about an Experimental Mail Server Analyzer Online. This might have helped me with a problem I was dealing with earlier this month. I am not sure whether my problem is related to what he is trying to accomplish but here is my problem description and resolution.
At the place I am working our customer service representatives said they were getting a lot of customer complaints about order confirmation emails never arriving. I did a little pivot table analysis on the event log and found a lot of SMTP 4000 and 4006 warnings. The frequency of the errors started escalating on July 29th. The good news is that I had not applied any patches in that time frame. The 4006 warning message I was familiar with since I had fixed a naked line feed issue in June. The 4006 warning message also appears when a customer enters their email address incorrectly so I expect to see it in the event log but at low levels. The 4000 warning message was new and said “Unable to bind to the destination server in DNSâ€. The problem appeared to be intermittent since not all emails were failing. However when I ran NSLOOKUP on the web site for the mail server that appeared in the 4000 message, it failed. When I used NSLOOKUP on my workstation to look up that same server, it worked. The DNS used by the web server was failing for a group of mail servers. The solution was very simple. Although both the ISP help desk and I thought the web site had local problems and needed a mid-day reboot, the help desk changed our domain name server. Based on their initial testing they did not think they had fixed the problem. I suspect they used ping to try and verify the mail server. Since a lot of people have turned off ping responses on their mail servers, I looked at the event log. I could see that the problem was probably fixed. The 4000 messages had disappeared completely and the mail queue was emptying. The 4006 messages went down to previous levels, too. In fact they are at a level slightly lower than I had noticed before the problem.
Supporting Technology » Blog Archive » Remove old versions of Java & Install the latest with no auto update!
I had several versions of Java on my PC and I lamented the fact that I needed to use “Add/Remove Programs†to remove them. This script came in handy although my copy and paste of the code needed several fixups before it would work. The single and double quotes from the copy were funky according to DOS. So I have posted my cleaned up copy. The SyntaxHighlighter Plus plugin has solved this copy problem for me in the past. Good Luck!
Rem to run this file and log the output use: "uninstall java.bat">>java_remove.log Rem @echo off & cls Rem List all Installation subkeys from uninstall key. echo Searching Registry for Java Installs for /f %%I in ('reg query HKLM\SOFTWARE\microsoft\windows\currentversion\uninstall') do echo %%I | find "{" > nul && call :All-Installations %%I echo Search Complete.. goto :NoneFound :All-Installations Rem Filter out all but the Sun Installations for /f "tokens=2*" %%T in ('reg query %1 /v Publisher 2^> nul') do echo %%U | find "Sun" > nul && call :Sun-Installations %1 goto :EOF :Sun-Installations Rem Filter out all but the Sun-Java Installations. Note the tilda + n, which drops all the subkeys from the path echo %1 for /f "tokens=2*" %%T in ('reg query %1 /v DisplayName 2^> nul') do echo . Uninstalling - %%U: | find "Java" && call :Sun-Java-Installs %~n1 if errorlevel 1 ( echo Doing further conditional checking on variables. for /f "tokens=2*" %%T in ('reg query %1 /v DisplayName 2^> nul') do echo . Uninstalling - %%U: | find "J2SE" && call :Sun-Java-Installs %~n1 ) goto :EOF :Sun-Java-Installs Rem Run Uninstaller for the installation MsiExec.exe /x%1 /qb /quiet /passive /promptrestart echo . Uninstall Complete, Resuming Search.. goto :EOF :NoneFound Rem No Java found to remove from this system echo No Java found or all versions already removed from this system goto :EOF
Using remote drives in a scheduled job
Recently I was debugging why a backup of a QuickBooks file was not working. A scheduled job running on the server was supposed to create a compressed encrypted version on a removable USB drive. The job was written by someone else and I guess they never checked to see if it worked!? I found that itt worked when I ran it interactively but it failed when I ran it as a schedule job. It took me a while to figure this problem out but the QuickBooks folder was actually a shared folder on another computer and the drive mapping was not available in the batch environment. To fix the problem I added the following statements to create a drive mapped as ‘r:’.
Dim objNetwork Set objNetwork = WScript.CreateObject("WScript.Network") strLocalDrive = "r:" strRemoteShare = "\\myserver\Backups_Daily" objNetwork.MapNetworkDrive strLocalDrive, strRemoteShare, False
Testing DNS Randomness
I found an interesting problem today. If I run the DNS Randomness test at work using Doxpara’s widget, our ISP fails this test. If I run the test at DNS-OARC, I get mixed results. Sometimes the source port randomness is good and sometimes it is bad. Hmm… I am guessing our ISP has not patched their DNS yet!
SourceForge.net: notepad-plus » FindReplaceNewlineHowTo
Simple find/replace compared to Advanced search/replace.
Simple find/replace is found in the Edit menu. Its shortcut is control+H. Advanced find/replace is in the Plugins menu, in the TextFX Quick menu. Its shortcut is control+R.
SourceForge.net: notepad-plus » FindReplaceNewlineHowTo
I finally researched an issue I was having with finding tab, new line, carriage return characters at the beginning of the line. I have an automated SQL procedure to create product data feeds in a tab delimited format. One of the symptoms of a soft SQL failure is that the line will wrap. This situation can be detected by looking for a first character that is either a tab, new line, or carriage return using the regex expression ^[\t\n\r]. . The simple find/replace does not find the problem lines while the Advanced version works as expected.
Quick Picks and Pans on my del.icio.us favorites for last week
Build a Generic Histogram Generator for SQL Server
Histograms help people analyze large amounts of data, whether you display them as tables or as charts. This article shows you how to do both.
… 1 day ago
JkDefragGUI – Freeware made by Emiel Wieldraaijer
… saved by 488 other people … 1 day ago
VirusTotal – Free Online Virus and Malware Scan
… saved by 1677 other people … 1 day ago
Meet ratproxy, our passive web security assessment tool
… saved by 1 other person … 4 days ago
… saved by 50 other people … 5 days ago
PCI Requirement 6.6 has merchants gearing up
… 5 days ago
- The Histogram Generator for SQL Server looks interesting but I think I will work on making pivot tables easier to use first.
- I tried JkDefrag last week. I like the look and feel of the Auslogics defrag tool. JkDefragGUI makes JkDefrag a little easier to use and adds a few features. Both tools are nice upgrades to the standard defrag tool.
- VirusTotal is a nice way to generate hashes for downloadable files.
- I need to check Ratproxy out. Ultimately I would like to compare this to WebScarab. I need to read this Google document again and play with to figure out why Google re-invented the wheel.
- Microsoft’s contribution to fight against SQL Injection would have been more appreciated if it worked for me. It did not generate a report for me. The program was more than willing to tell me that I did not have it configured properly. When I finally had it configured properly, the result is no report?!
- PCI 6.6 is a mess. I am suspicious that the firewall option will ever be a cost effective solution for retail web servers. My first run at automated code analysis was inconclusive. I believe that low cost automated and manual code analysis are probably the best cost to benefit options.
It’s So Important To Keep Talent In Ohio « Cincinnati IT – an andy erickson blog
Andy wrote:
My jaw dropped this morning as I read Chester Finn’s Wall Street Journal opinion piece titled The Self-Inflicted Economic Death of Ohio. First, because the piece quantified some of my gut feelings about Ohio’s talent drain, and second, because of the magnitude of the problem.
It’s So Important To Keep Talent In Ohio « Cincinnati IT – an andy erickson blog
Unlike Andy I am still searching for a word to describe how I felt when I read this article. I have a seventeen year old son and we talked frequently about college and jobs over the last year. Our talks were stimulated by what they did not discuss at high school. Let’s just say he is learning a lot more about the job market from me than from his school and he goes to a “good†school. About a year ago his school started the college selection process. This is a lot earlier than when I was going to school so I was duly impressed. I don’t think I even talked about colleges and majors until I took my SAT test. I was pretty clueless about colleges but I was well prepared for a major in a science or engineering field. What shocked me was that my son’s college counselor was encouraging him to pursue a liberal arts major. I was curious about their reasoning so I started asking questions. The deeper I probed the less I liked about the philosophy of our present education system. It was not hard to figure out that the college counselor’s recommendations were a natural result of the educational system that was in place. My son had plenty of opportunities to try out science and technology courses but was generally steered away from them. When we met with the counselor and I mentioned that the biggest job needs for the next decade are still in the science and technology fields, she said that he should have pursued a course selection with more science and technology courses in it. Okay, let’s drop “Conversational Frenchâ€. Her answer was, “He can’t do that!†Besides she said, “He is taking Honors English and Historyâ€. Huh! Another point she made and I have heard it repeated several times, is that you what you study in college does not determine your career choice. This is definitely not true for those pursuing a job in a technology field. Wow! When I worked for a large chemical company we rarely interviewed college graduates who did not have an engineering degree. That was just a fact of life. Another fact of life is that engineers are paid well. If you happened to have and engineering degree and work experience from a co-op program you were gold and would get top dollar. It really looks like the high schools are just going through the motions of college/career counseling.
About a month later he got his PSAT results. He got a 700 on the math portion and the PSAT report recommended that he consider a college with science or technology major. Go figure!
Wordle is like a mirror of your soul
As seen from del.icio.us I must be seen as…
Re-engineering an application
Recently I found myself trying to debug an active server page application. It appears to be a simple application. When you go to the page, the server generates a text file which I call a data feed. It is used by search engines to build links to your products. The final step in the process is to download the data feed file and then upload it on the the search engine site. This is such a simple application you could have programmed this in a variety of languages without much effort or concern. The original developer chose to develop the application as an active server page. ASP would not have been my first choice primarily because programming it in SQL is a much simpler solution. In SQL the solution is so simple and straight forward it approaches the holy grail of computer programming, self documenting.
I got involved with re-engineering the ASP application because it was not working anymore. The page was not displaying and their were no error messages. By definition applications are no longer simple if they fail and do not produce an easy to understand error message. I suspected that the error might be related to a “response buffer limit exceeded†issue so I increased the buffer limit. This worked on the development system but it had no effect on the production system. That is not good! Now I was going down a path I did not want to go, fiddling with IIS parameters on a production system trying to fix a problem. Since I am definitely “old school†and evidently SQL centric, I decided to turn this into a batch operation and skip out on the human download/upload process altogether. My plan was to schedule a SQL job to download the data feeds into files using SQL and then use FTP to upload the feeds to their respective search engine sites.
I originally thought I would have this finished this task in a day or two. Boy was I wrong! The combination of ASP, XML, XSL, and SQL stored procedure put the processing in various places and difficult to follow. Of course there wasn’t any program documentation and the original programmer was unavailable. My plan was to combine everything into a SQL view that either BCP or OSQL would use to create a tab delimited file. Using BCP I can use the ultra-simple “Select *†query on the view.
The first big problem was to create the category field. I needed to recursively lookup the category parent from a table of categories. This was process was originally performed in ASP. After some effort I created a SQL table to mimic the process.
The next problems came in rapid succession. The description field needed the HTML tags removed and some HTML entities needed to be escaped. Then I found that some products were being listed in multiple categories and the category being used by my view was a defunct category.
One of the nice benefits of using the “SQL View†approach was that it was easy to test and verify. I also had a backup plan if the batch process failed for some reason. Although I briefly tried OSQL I found that BCP had a more direct way of creating tab delimited files. Since it only takes a minute and half to create the four feeds, processing requirements are not an issue. Once I had copied the headers to the front of the file I was good to go. I matched the data using WinMerge on the development system since the ASP screen still worked on it.
The data matched and now I am ready to submit the files. This minor re-engineering took a lot more time than I planned but I think the process if very to explain.
The next problems were more annoying. There were permission problems with running BCP. Yahoo created FTP problems for me. They allow you to update files using FTP but your FTP client better support PASV. I was able to upload the file using FileZilla but not Microsoft FTP. I am searching for a command line FTP client I can use. I think MOVEit Freely from Ipswitch might be the answer. Ipswitch is probably best known for WS_FTP. A few years back WS_FTP was the standard bearer for FTP clients and servers.
Finally I am not sure what happened to MSN’s product upload page, http://productupload.live.com. Suffice to say it has had major problems every time I tried to use it. At this time I am not sure MSN wants me to update the data feeds using FTP. It is too bad they are so difficult to use. Most of our traffic comes from Google and Yahoo. Not surprisingly they get the bulk of our advertising expenses. MSN has always been a distant third place.