While working on the configuration of my TMG Server I observed inconsistent behavior and slow system performance. After a quick investigation I noticed a number of log entries after a system reboot in the Event Log.
The entries came from MSSQL$MSFW with Event IDs 17204 and 17207. Additionally I noticed that the Default Log Queue location of TMG (C:\Program Files\Microsoft Threat Management Gateway\Logs) was filled up with more then 650.000 files consuming 14 GBytes of system disk space.
The problem is that for some reason one or more databases had become corrupt and prevented SQL Server from correctly starting up and processing log entries from TMG. Therefore Forefront TMG falls back to its Log Queue.
Since the issue was already going on for some time in my case, I could not establish the root cause that caused the Database files to corrupt in the first place. But I do recall a BSOD from some time ago on the host OS due to a driver issue. That probably caused the issue.
Remove Corrupt Databases
I found a solution on the Microsoft Forums that describes how to remove the corrupt databases from Microsoft SQL Server in order to get SQL Server back in business.
- Start an command prompt and connect to the SQL Server database with the SQL Server Command Line Tool: OSQL -S %TMGComputerName%\msfw -E
- Delete each corrupt database with the following command: DROP DATABASE %DATABSENAME%
- Execute the commands by entering the command GO.
- Restart the SQL Server service for the SQL Server Instance msfw.
- Verify in the Event Log that no new corrupt databases are logged, if so repeat steps 1 to 4 until no events 117204 and 11207 are logged.
Once the SQL Server becomes available, the Log Queue should start processing again and write all queued up entries to the SQL Database. This can be monitored from the Management Console.
In my case it took a couple of hours due to the size.
TMG Log Queue Status
I like time to be on time and manually configure the NTP server for my domain. This way I can choose a time source which I have good connectivity to. But Kerberos needs the time to be synced within the domain or authentication will fail.
To configure a manual time source take the following steps:
- Choose a NTP Server as Source
- Find the PDC emulator
- Stop the Time Service
- Configure the Time Provider
- Open any firewall ports
- Start the Time Service
- Verify result
Choose NTP Server as Source
For me the Time Servers from xs4all are the best. But one good place to start is pool.ntp.org.
I’ll just stick to ntp.xs4all.nl and ntp2.xs4all.nl
Find the PDC Emulator
To find the PDC there are multiple options, from old to new:
Stop the Time Service
Well, here again multiple possibilities. But let’s just stick to PowerShell:
Configure the Time Provider
w32tm /config /syncfromflags:manual /manualpeerlist:”0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org”
Since the time is configured on the PDC there is no need to set it as a reliable source.
Open Firewall Ports
The NTP protocol uses UDP on port 123. See RCF 1305
Start the Time Service
Verify the Result
With the following command the configuration can be verified:
w32tm /query /configuration
To perform a sync:
This should respond with a message saying: The command completed successfully.
To check on the status:
w32tm /query /status
For the first time in my life I saw S.M.A.R.T. in action. Both the BIOS and Windows reported that an issue was detected with a hard disk and that failure could occur.
The Windows Event log didn’t show any hard disk read errors yet, so cloning the disk would have a high chance of success.
I’ve seen disks fail all the time, so nothing new there. But I’ve never seen the S.M.A.R.T. system warn up-front so a replacement is possible without losing any data.
So I added a new disk, did a clone of the drive. Updated the MBR on the new disk, removed the old disk, update BIOS and good to go with a new drive. No unexpected down time, or even worse… data loss.
I didn’t take the time to look what exact S.M.A.R.T. parameter triggered the error.
To quickly open a command window and have it at that specific location use Shift + Right Click in folder windows and use the Open command window here option. Make sure nothing is selected, else it will not show the correct menu option.
Neat is that when doing this on an UNC path, it will automatically map a network drive and have the Command Prompt at that network drive location. Since the Command Prompt can’t handle UNC paths as current directory.
The network drive will be automatically disconnected when the Command Window is closed. Having multiple Command Prompts will created multiple network drives, starting backwards from the letter Z.
For the installation of Forefront TMG I slipstreamed the updates which saves a lot of time. For Forefront UAG it is even more interesting to slipstream the updates. Since both Forefront TMG and Forefront UAG have a set of updates that need to be applied.
When searching for a tutorial I only found a post on the Microsoft Forum regarding the slipstreaming the updates for Forefront UAG. However, no one is reporting any results good nor bad.
Forefront TMG and Forefront UAG still need Windows Server 2008 installations. My WSUS servers keeps reporting update KB972493 for all Windows Server 2008 installations on a consistent basis. Which is incorrect, since these machines are not running WSUS or anything at all. There are clean retail installations, so the mistake must be in the update detection logic.
I found that Jeremy Jameson has the same issue also and created an computer group for the servers and then declined the update for that computer group.
After successful slipstreaming all updates for the Forefront TMG installation media I tried to duplicate the success for the Forefront TMG Client installer, which lacks update KB2616324.
I failed to slipstream this update, although it looks like it is going fine, the installer is not updated. After running the “updated” installer it still installs as 7.0.7734.100 instead of 7.0.7734.186.
I checked the patch file with Orca, and it looks all OK but doesn’t seem to work out. Applying the update to a system with the Forefront TMG Client installed, works as expected. Will need to deploy this update after Forefront TMG Client deployment.
From Microsoft only the Forefront TMG Retail (7.0.7734.100) version is available, so if you do a fresh install you need to apply at least the following updates:
- Service Pack 1 (7.0.8108.200)
- Service Pack 1 Software Update 1 (7.0.9027.400)
- Service Pack 2 (7.0.9193.500)
- Service Pack 2 Update Rollup 4 (7.0.9193.601)
These updates all take serious time to install when applied to a system. Not a bug issue, since you don’t have to attend the whole process. But when a new installation is part of the recovery strategy, time is precious.
To slipstream the files:
- Copy the Forefront installation media to a writable location
- Download the updates.
- Apply them using the Windows Installer (msiexec.exe) with the /a /p parameters.
To request certificates from Forefront TMG ports need to be opened to allow access from Forefront TMG to the Certificate Authority. This is a known situation and there is a blog post at ISA Server on how to accomplish this.
But if the CA is a Windows Server Core installation, it is a little more tricky to configure the CA to use a static port. It is not possible to remote manage the DCOM part of the CA using the MMC Component Services Add-in.
You can do this directly through the registry using regedit on the Server Core installation.
- Find the Application ID GUID of the Certificate Server Request component.
- Update the key to use a fixed port.
- Restart the Certificate Service.
While settings up Forefront TMG proxy I wanted to verify that I could access websites on a non default HTTP port. Turned out that finding one was kinda hard. But I found portquiz.net, a nice website that listens on all ports.