I finally called Tech Support and we found out that there is a hotfix out related to RPC Issues in ISA 2004, also there is an “SBS Protected Networks Access Rule” . Rt click it and “configure RPC protocol and uncheck the “Enforce strict RPC compliance”. This will allow DCOM to pass.
Source: Outlook by the sound : RPC server is unavailable since SP1
Okay this should not be that difficult but I found a way to screw it up. I started to suffer these problems when I installed SP1 for SBS Premium in 2005(?). The most prominent symptom of this problem is that you suffer Autoenrollment errors on the client and 537 login audit failures on the server. The 537 errors are kerberos errors but they are particularly ambiguous. This was an annoying problem in my case but surprisingly everything still works. From a different source than the one listed above, I unchecked the “Enforce strict RPC compliance” box. The problem is that there are two boxes, one in the System Policy and another the box on the “SBS Protected Networks Access Rule”. I unchecked the box in the System Policy and it did not fix my problem. So I spent a lot of hours after installing SP1 trying to figure out why I was still getting errors. Over the last two days I have been rebuilding my desktop computer so I made another attempt to clear up this problem. Lo and behold, I found this in one of my searches. Unchecking the box on “SBS Protected Networks Access Rule” appears to have fixed the Autoenrollment errors and 10009 DCOM errors on the client. It also fixes the 537 audit failures on the server.