Monday, March 30, 2015

Group Policy Fails to Apply on Domain Controllers With IPv6 disabled

Forward

Let me start by saying, don't disable IPv6 on domain controllers. There's no reason for it, and it will cause you more headaches in the long run. From what I've read, it used to be best practices in the early days of IPv6, but Windows is smart enough now that it shouldn't have any problems with it.

So I inherited a domain setup where the domain controllers (Running Server 2008 (not R2)) had IPv6 disabled for some reason. It's not well documented why this was done, and it's on my list of things to fix, but for the moment I'm stuck with it.

As to why I'm still running DCs on 2008, shut up it's on my to-do list.

Solution

Disabled 6to4 adapter. 6to4 adapter will register itself in DNS and cause lookup problems. I couldn't find a good way to tell the adapter not to register itself, so I settled for disabling it. From an admin command prompt:
netsh int ipv6 6to4 set state disabled
been running this way for a week or so, and haven't had any more problems. I am able to run a gpupdate on the domain controllers and have it apply successfully.

Problem & Full Story

 See "Forward" as to why I am operating DCs with IPv6 disabled.

Started running into this problem a while back, but had never had the time to troubleshoot it fully. Domain Controllers would randomly stop working correctly requiring reboots and extensive testing. These problems would take the form of

  • Slow logins -- stuck at applying user settings
  • Slow boot -- stuck applying computer settings
  • "no trust relationship" errors
  • DCs failing to apply group policy updates via "gpupdate /force"
    • Updates would apply correctly on reboot
Eventually, going through logs, I was able to narrow it down to a DNS issue. Mostly there were "RPC Server Unavailable" Errors which indicated a lookup failure. The DCs also function as DNS servers, so the fact they had lookup problems when nothing else seemed to was double strange.

Parsing through the "Forward Lookup Zones" I found that the DCs were still registering IPv6 addresses. If I cleared those addresses out manually then everything seemed to work. The DNS entries I cleared out looked like this:

Name                    Type                    Data                    Timestamp
(same as parent folder) IPv6 Host(AAAA)         2222::etc::2220         2/2/22
(same as parent folder) IPv6 Host(AAAA)         2222::etc::2221         2/2/22
...
DC1                     IPv6 Host(AAAA)         2222::etc::2222         2/2/22
DC2                     IPv6 Host(AAAA)         2222::etc::2223         2/2/22

After clearing these out everything would work for awhile. But the entries would eventual re-create themselves and the problems would come back. The (same as parent folder) -- which is a reference to the resolution of mydomain.com -- would recreate every hour or so, whereas the DC1,DC2 entries seemed to only come back on a reboot. It was baffling because the IPv6 adapters were totally disabled, so I couldn't figure out why/how they were continuing to register themselves.

After looking through ways to more forcibly disable IPv6 (through registry hacks, etc.), and deciding that was a bad idea, I thought to look more closely at the addresses that were being registered.  Issuing a ipconfig /all, I realized that those addresses were associated with the 6to4 adapters, not the actual ipv6 interface.

A quick google later to find out how to disable the 6to4 adapter and everything was in order. As mentioned above it's been a week and I haven't seen any side effects to disabling the 6to4 adapter.

Long-term solution is to reenable IPv6 on the domain controllers, but it's been disabled for so long, and since I have no idea why it was disabled in the first place, that will require more careful testing.

Monday, March 16, 2015

Print Server CPU spikes on reboot -- Spoolsv.exe

Solution

Problem in my case turned out to be a bad driver. Luckily it was a driver from an old printer that was no longer on the network.

Cleaning out print drivers is pretty easy: Open up two instances of print manager, go to the Drivers menu on one, and the printers on the other. Arrange the columns on the printers page so that you can see the printer name and the driver name. Sort by driver name, then compare the list to the ones in the Drivers menu. This makes it easy to see which drivers are no longer in use.

If you've cleaned out old drivers, and the problem is still occurring, you may need to start removing printers until the problem stops. Remove a printer, reboot the server, and see if the problem still exists. Of course, make sure you copy the printer information so you can re-add it later. Once you've determined which printer/driver combo is causing the issue, see if you can find an alternative driver for that printer. Most printer manufacturers recommend using their "Universal Print Driver" in a print server environment, rather than the 'named' or 'model specific' driver.

 Problem

In brief, our print server would hang with 100% CPU usage whenever it was rebooted. This necessitated spoolsv.exe (the 100% process) to be manually terminated, and the spooler and all dependant services to be restarted. After manual termination/restart, the CPU levels would remain normal. Printer shares would be unavailable during the CPU spike.

Machine Specs:

  • Server 2008 R2 -- fully updated
  • VM on top of XenServer 6.2 - Intel based system
  • 2 core
  • 3 GB of RAM
  • Additional Printer Related Software
    • Print Manager (print tracking software)
    • Citrix Universal Print Server
Things I tried that didn't help

  • Windows updates
  • Updating paravirtualization driver.
  • SFC / Checkdisk/ other system file scans
  • Re-installing drivers/print queues  
Problem ended up being an issue with some unused print driver. Apparently the spooler service loads up drivers even if they are not being used by any print queue. So the only way to fix the problem was to remove all drivers from printers that had been retired -- see details in Solution above.