Monday, December 11, 2017

ZFS on CentOS 7 : Preface and HW info

Preface

These are the notes on my adventure into getting a ZFS server running on centos 7; with the objective of creating a software-defined-storage solution without the hardware compatibility limitations of vendor offerings.

I am using a mixture of enterprise and consumer grade hardware to start, basically the spare stuff I have around the office.

Hardware info

  • Server
    • Dell Poweredge R610
    • 24GB Ram
    • 4x Gbs ethernet
    • HDD
      •  x2 10k SAS 300GB HDD
      • Raid-1 handled by Poweredge controller
      • These are for the OS,  ZFS will not touch these
    • SDD
      • x2 Sandisk SATA 32GB
      • Passed through Poweredge controller to be handled by ZFS
      • These will be used for various caches as I test different tuning options
    • SAS HBA connected to MD3200
  •  JBOD-ish Device
    • DELL MD3200
    • 10x 7.2K SAS 1.2TB HDD
    • Technically this is not, and cannot be, a JBOD. But I've set each drive in it's own disk group, so it presents itself more-or-less like a JBOD.
  • Other Hardware
    • 4 Additional R610 servers, running XenServer to use as clients to test virtualization performance.
    • Dell Force10 10GB switch. This is in use for other things, but should provide sufficient bandwidth since my hosts are limited to 1GB by their hardware 

>> CentOS install and ZFS install

Thursday, December 7, 2017

package: zfs-dkms requires dkms >= 2.2.0.3

Solution

First install EPEL release repo

sudo yum install epel-release

This should fix the initial install issue. However once the install is complete zfs may not work properly, you may see:

# zpool status
The ZFS modules are not loaded
Try running '/sbin/modprobe zfs' as root to load them

However, running modprobe results in the following
 
# modprobe zfs
modprobe: FATAL: Module zfs not found

Looking at DKMS you will see the modules are added, but never installed and attempting to install them fails. This seems to be a bug if you don't install epel-release before installing kernel-devel.
 
# dkms status
spl, 0.7.3: added
zfs, 0.7.3: added

# dkms install -m spl -v 0.7.3
Error ! echo
Your kernel headers for kernel 3.10.0.693.el7.x86_64 cannot be found at
/lib/modules/3.10.0-693.el7.x86_64/build or /lib/modules/3.10.0.693.el7.x86_64/source

A reboot seems to solve the spl module issue, but the zfs module still refuses to install 

#reboot
[truncated reboot output]
 ...
[login]
 ...
#dkms status
spl, 0.7.3 3.10.0-693.11.1.el7.x86_64, x86_64: installed
zfs, 0.7.3: added

# dkms install -m zfs -v 0.7.3
[truncating error message, but there's a lot of exit code status 2, failed to clean build area, unable to remove file messages]

However, then run the zfs install again and it works... for some reason? 

# dkms install -m zfs -v 0.7.3
 [truncating a really long install process output]
DKMS: install completed

# modprobe zfs
# zpool status
no pools available

This doesn't make much sense, but best guess is that the install fails initially (due to the epel-release not being installed) in a not-very-elegant way. This leaves behind junk that causes the install to fail in the future. Rebooting and manually running the install clears that junk. This is all conjecture, but I hope it helps someone!
 

Additional troubleshooting

I've run through this a few times now, and the error doesn't seem to resolve the same way everytime. Most recent time I did it, the restart/manual install didn't work for the zfs module (spl installed after a reboot as before). I had to remove the ZFS module and re-add it.


# dkms remove -m zfs -v 0.7.3 --all
[output from removal]

# dkms add -m zfs -v 0.7.3
[output from add]

# dkms install -m zfs -v 0.7.3
[and it should work now]

Tuesday, May 17, 2016

Winmgmt ResetRepository File Already Exists

Solution

This error comes up because the WMI repository has been reset too many times, and windows runs out of file names.

Under "C:\windows\system32\wbem\" you'll see a bunch of "repository.###" folders. These are the backups windows makes of the repository whenever you reset it. However this seems arbitrarily limited to three digits, so once it hits repository.999 it can no longer back the wmi repo up before the reset, so the reset fails.

The quick and dirty solution is to delete the various repository.### folders then try the reset again. Better solution is to find out why the repository is having to be reset so often and try to fix the underlying cause of the WMI corruption.

Full Story

I have a number of problems with WMI repositories becoming corrupt and requiring to be reset. I've never been able to track down the precise cause of these corruptions, so I've just been doing my best to keep things up and running.

Today I ran into a fun problem. I had a VM that wouldn't allow logins (or logins would take forever). Closer inspection saw that it had the telltale signs of WMI problems (1 CPU core maxed out, high memory usage). I rebooted to safe mode to try the reset, but the reset (winmgmt /resetrepository) failed with the error.

WMI repository reset failed. Windows cannot create a file if that file already exists

Or something along those lines. I forgot to write down the exact error before rebooting the machine, sorry. 

Further investigation found the hundreds of "repository.###" folders in the wbem directory. Deleting these folders allowed the reset to run correctly. VM rebooted to normal mode without issue.

Friday, July 24, 2015

Win2D First Impressions

Introduction

I like making little 2D games from time to time. There are a myriad of tools to do this, but as a Systems Admin I like working as close to the computer as I can without getting bogged down in pointers. So many tools like Unity, RPGmaker, etc. are too high level (pun) for me -- I like knowing what's going on in the code -- while using the Directx/Direct2D in c++ gets too frustrating with it's micromanaging minutia, lack of generic object lists, pointer resolution, and memory management.

So it's a struggle, and it's always bugged me that C# (My favorite mid-level language) doesn't have any official API for DirectX. Someone at Mircosoft seems to have been thinking the same thing as they're working on Win2D -- a c# implementation of Direct2D, which is supposed to bring at least the 2D portions of DirectX to native Windows applications. Previously I've used third party offerings (SFML, namely) to accomplish this.

As I write this introduction I am just getting started setting it up. So what follows will be my first-impressions look at the (Alpha, Beta?) package as of version --  0.0.22 at time of writing.

I am following the  quickstart and getting set-up guides on the github site. Using my work machine (windows 8.1, vs 2015 community RC) and my home machine (win 10 RC, vs2015 community RC).

Raw Notes

  •  There is no "blank app" listed under my Visual C# templates, hopefully the "Blank Solution" under Visual Studio Solutions works. Couldn't find it in the online templates either. 
  •  Error "no projects in this solution supported by nuget" so that's fun
  • Trying to add additional packages to VS install, see if that helps
  • Yes that worked. From the Visual studio installer I added everything under "Windows and Web Development" and now I have the blank app as an option. It's listed as "Blank App (Universal Windows 8.1) which isn't quite the same as the guide says, but it's closer
  • As always, visual studio is weird about network locations.
  • Have to get a developer license for this type of app. It's apparently free, so whatever.
  • Nuget works now
  • Getting "Cannot Create an instance of "Canvas Control"" when following step 4 in quickstart
    • Specified module cannot be found 0x8007007E
    • appears to happen for all canvas modules
    • Switched Processor target under Build > Configuration Manager from 'any CPU' to 'x64' this seems to have cleared up the issue (did have to compile before the warning went away)
    • The cpu target thing was mentioned in the instructions, I just missed it.
  •  Have drawn some text and some shapes, neat
  • getting into animation
    • CanvasAnimatedControl is kind of neat, appears to handle the looping for you for drawing. Just have to say draw these things, then it redraws as those things change.
  •  trying to figure out the precise commands by adding a button to the mix
    • Hurray I have a generic object list I can add/remove things from that automatically draws to the screen.
  • Method of loading images is a bit wonky, but ultimately works
  • Wonder if I can get this working in a non win8 style app. Just like, a wpf app or something.

Conclusion

So there's that. Not really what I'm looking for since it only supports Windows apps. Still it seems nice, really straight forward and what have you. If I were looking to make a windows app it would certainly appeal.

Maybe one day Microsoft will make an official supported C# wrapper for DirectX. For now I think I'm going to try SharpDX and see how that goes.

Friday, July 17, 2015

Code Dump: Get-UserProfile / Remove-UserProfile

function Get-UserProfile{
        <#
        .SYNOPSIS
            Use WMI to query a computer about local profiles on the machine
        .Description
            This Fucntion is used to get information about local profiles on a computer. This will return information about any local profile, including the local cache of the domain accounts. By default returns the SID, Local path, and Last Use Time of the account; the -verbose flag can be used to return additional information.
        .PARAMETER UserID
            UserID to search for. If left blank will default to all users. By default UserID must match exactly, but you can use the wildcard '%' to perform more general seraches
        .PARAMETER Computer
            Computer to query for user accounts. Leaving Blank will default to 'localhost'. 
        .PARAMETER ExcludeSystemAccounts
            Filters out System accounts (e.g. System, Network Service). This is done by looking at the 'special' property, which does not filter out users non-windows programs may create.
        .PARAMETER OnlyLoaded
            Setting this parameter shows only profiles that are currently in Use -- Combine with -ExcludeSystemAccounts and you can get a pretty good idea of who is currently logged into a machine.
        .PARAMETER ExcludeLoaded
            Returns only user profiles that are not currently in use. This is useful if you need to clear out profiles.
        .PARAMETER Verbose
            Returns Full user porfile data, rather than the default SID,LocalPath,LastUseTime
        .PARAMETER OlderThan
            Filter Results based on datetime. This requires a datetime object
        .Example
            Get-UserProfile -UserID MyUser
            Basic usage to see if the user "MyUser" exists on the local machine.
        .Example
            Get-UserProfile -Computer RDSServ1.mydomain.com
            Lists all user Profiles from remote computer "RDSServ1.mydomain.com" 
        .Example
            Get-UserProfile -Computer RDSServ1.mydomain.com -ExcludeSystemAccounts -OnlyLoaded
            Lists non-system user profiles from remote computer currently marked as loaded. This gives a pretty good idea of who is currently logged into a remote machine.
        .Example
            Get-UserProfile -OlderThan $((get-date).adddays(-14))
            Lists user profiles that have not been used on the localhost in 14 days.
        .Notes
            Author: Keith Ballou
            Date: Oct 15, 2014

            This Script Relies on Convert-UTCtoDateTime -- A function for converting UTC strings to DateTime Objects


    #>
    [CmdletBinding()] 
      param( 
     [Parameter(Mandatory=$False)][string]$UserID="%",
     [Parameter(Mandatory=$False)][string]$Computer="LocalHost",
     [Parameter(Mandatory=$False)][switch]$ExcludeSystemAccounts,
     [Parameter(Mandatory=$False)][switch]$OnlyLoaded,
     [Parameter(Mandatory=$False)][switch]$ExcludeLoaded,
     [Parameter(Mandatory=$False)][datetime]$OlderThan   
     
    )
if(!(Get-Command Convert-UTCtoDateTime -ErrorAction SilentlyContinue)){
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "################################################################################"
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "#                                                                               "
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "This Program Requires cmdlet ""Convert-UTCtoDateTime""                          "
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "Find it here:                                                                   "
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "http://pastebin.com/SSKJ4bwt                                                    "
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "#                                                                               "
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "################################################################################"
    break;
}
if($Computer.ToLower() -eq "localhost"){
    
    
    $Return = Get-WmiObject -Query "Select * from win32_userprofile where LocalPath like '%\\$UserID'" 
    

}
else{
    $Return = get-wmiobject -ComputerName $Computer -Query "Select * from win32_userprofile where LocalPath like '%\\$UserID'" 
}

#Filter System Accounts
if($ExcludeSystemAccounts){
    $Return = $Return | Where-Object -Property Special -eq $False
}
#Filter out Loaded Accounts
if($ExcludeLoaded){
    $Return = $Return | Where-Object -Property Loaded -eq $False
}
#Filter otherthan loaded accounts
if($OnlyLoaded){
    $Return = $Return | Where-Object -Property Loaded -eq $True
}

#Filter on lastusetime
if([bool]$OlderThan){
$Return | Where-Object -property LastUseTime -eq $Null | % {Write-Host -BackgroundColor "Black" -ForegroundColor "Yellow" $_.LocalPath " Has no 'LastUseTime', omitting" }
$Return = $Return | Where-Object -property LastUseTime -ne $Null
$Return = $Return | Where-Object {$(Convert-UTCtoDateTime $_.LastUseTime -ToLocal) -lt $OlderThan }
}

if($PSBoundParameters['Verbose'])
{
Write-Output $Return
}
else{
 Write-Output $Return | Select SID,LocalPath,@{Label="Last Use Time";Expression={Convert-UTCtoDateTime $_.LastUseTime -ToLocal}}    
}

}


Function Remove-UserProfile{
            <#
        .SYNOPSIS
            Removes User Profiles from a machine via WMI
        .Description
            This Cmdlet is used to remove user profiles from a machine when it is inconveinient to do so from the System menu. The script relies on Cmdlet Get-UserProfile. It is especially userful on Terminal Server/RDS machines where opening the system menu to delete user profiles can take hours (because reasons, I suppose). 
        .PARAMETER UserID
            UserID to search for. Unlike Get-UserProfile, this cannot be left blank. If you intentionally want to remove all user profiles, or need to select multiple, you can use WMIs limited regex. Check this Link for more information: http://blogs.technet.com/b/heyscriptingguy/archive/2012/07/13/use-the-like-operator-to-simplify-your-wql-queries.aspx
        .PARAMETER Computer
            Computer to delete user accounts from. Leaving Blank will default to 'localhost'. 
        .PARAMETER Batch
            This Flag will suppress confirmation dialogs, as well change console output to the more redirect friendly Write-Output, rather than the Write-Host it otherwise uses.
        .PARAMETER OlderThan
            Filter user accounts based on last use time. Remember that this Cmdlet does not assume 'all users', so you must specify -UserID %. 
        .Example
            Remove-UserProfile myuser remotemachine.mydomain.com
            Remove Users using positional parameters
        .Example
            Remove-UserProfile -UserID myuser -Computer remotemachine.mydomain.com
            remove users specifying with flags
        .Example
            Get-Content machinelist.txt | % {Remove-UserProfile -User myuser -Computer $_ -Batch} >> C:\Temp\UserRemoval.log
            Remove a user from a list of machines using the batch mode, seding output to a log file
        .Example
            Remove-UserProfile -OlderThan $((get-date).adddays(-14)) -UserID %
            Remove all user accounts that haven't been used in 14 days
            The '%' here is the WMI regex for anything (analogous to '*' (or '.*') in most regex) -- it is necessary because this Cmdlet never assumes all users

        .Notes
            Author: Keith Ballou
            Date: Oct 16, 2014
         #>

    [CmdletBinding()]
    param(
        [Parameter(Mandatory=$True)][string]$UserID,
        [Parameter(Mandatory=$False)][string]$Computer="LocalHost",
        [Parameter(Mandatory=$False)][datetime]$OlderThan=(get-date).adddays(1),
        [Parameter(Mandatory=$False)][switch]$Batch

    )

    #Make Sure Necessary Cmdlets Exist
    if(!(Get-Command Get-UserProfile -ErrorAction SilentlyContinue)){
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "################################################################################"
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "#                                                                               "
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "This Program Requires cmdlet ""Get-UserProfile""                                "
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "Find it here:                                                                   "
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "http://pastebin.com/wvUDki7p                                                    "
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "#                                                                               "
    write-host -BackgroundColor "Black" -ForegroundColor "Red" "################################################################################"
    break;
    }

    #to simplify query, if OlderThan was not specified, assume anything earlier than right now (plus a day to account for timezones, rounding, etc)
    #if(![bool]$OlderThan){
     #   $OlderThan = (get-date).adddays(1)
    #}

    #This Part relies on another of my Cmdlets "get-UserPorfile" to simplify the code a bit
    #This Could be substituted with: $ProfileList = Get-WmiObject -Computer $Computer -Query "Select * From Win32_UserProfile where LocalPath like \\$UserID"
    #..............................: $ProfileList = $ProfileList | Where-Object -Property Special -eq $False
    #Additional logic may have to be added if WMI doesn't like 'localhost' as a ComputerName. Seems to work ok for me, but I wouldn't count on it.
    #Get_UserProfile includes this logic
    $ProfileList = Get-UserProfile -Verbose -UserID $UserID -Computer $Computer -ExcludeSystemAccounts -OlderThan $OlderThan


    #If no Users were found, exit
    if(!$ProfileList){
        Write-Warning "NO USER PROFILES WERE FOUND"
        RETURN;
    }

    #Confirmation Dialog if -Batch is not set
    if(!$Batch){
        Write-Warning "ABOUT TO REMOVE THE FOLLOWING USER ACCOUNTS"
        Foreach($User in $ProfileList){
            $User | Select SID,LocalPath,@{Label="Last Use Time";Expression={Convert-UTCtoDateTime $_.LastUseTime -ToLocal}}
        }
        $Title = "PROCEED?"
        $Message = "ARE YOU SURE YOU WANT TO REMOVE THE LISTED USER ACCOUNTS?"
        $Yes = New-Object System.Management.Automation.Host.ChoiceDescription "&Yes","Removes User Accounts"
        $No = New-Object System.Management.Automation.Host.ChoiceDescription "&No","Exits Script, No Changes Will be Made"
        $options = [System.Management.Automation.Host.ChoiceDescription[]]($yes, $no)
        $result = $host.ui.PromptForChoice($title, $message, $options, 1) 
        switch ($result)
        {
            0 {}
            1 {return;}
        }

    }

    #Remove Users provided they are not currently Loaded
    Foreach($User in $ProfileList){
        if($User.Loaded){
            if(!$Batch){
            Write-Host -BackgroundColor "Black" -ForegroundColor "Red" "User Account " $User.LocalPath "is Currently in user on" $Computer ":`tSkipping"
            }
            else{
            Write-Output "User $($User.LocalPath) on $($Computer) was in use and could not be removed"
            }
            continue;
        }
        if(!$Batch){
        Write-Host -BackgroundColor "Blue" -ForegroundColor "Green" "Removing User $($UserID.LocalPath) from $($Computer)"
        }
        else{
        Echo "Deleting $($User.LocalPath) from $($Computer)"
        }
        $User.delete()


    }

}

function Get-SNMPWalk{

    [CmdletBinding()]
    param(
        [Parameter(Mandatory=$True)][string]$IP,
        [Parameter(Mandatory=$False)][int]$SNMPVersion=2,
        [Parameter(Mandatory=$False)][int]$Port=161,
        [Parameter(Mandatory=$False)][int]$TimeOut=3000,
        [Parameter(Mandatory=$False)][string]$Community="public",
        [Parameter(Mandatory=$False)][string]$OIDStart=".1.3.6.1.2.1.1.1"
    )

[reflection.assembly]::LoadFrom( (Resolve-Path ".\SharpSnmpLib.dll")) | Out-Null

if($SNMPVersion -gt 2 -or $SNMPVersion -lt 1)
{
    Write-Error "Invalid SNMP Version Number, This script supports Versions 1 and 2"
    return;
}

$OIDObject = New-Object Lextm.SharpSnmpLib.ObjectIdentifier ($OIDStart)
#if($SNMPVersion < 3)
#{
#    $results = New-GenericObject System.Collections.Generic.List Lextm.SharpSnmpLib.Variable
#else
#{
    $results = New-Object 'System.Collections.Generic.List[Lextm.SharpSnmpLib.Variable]'
#}

$IPObject=[System.Net.IPAddress]::Parse($IP)
$Server=New-Object System.Net.IpEndPoint($IPObject,$Port)

switch($SNMPVersion)
{
    1
    {
        $SNMPVersionObject = [Lextm.SharpSnmpLib.VersionCode]::V1
    }
    2
    {
        $SNMPVersionObject = [Lextm.SharpSnmpLib.VersionCode]::V2
    }
    #3
    #{
    #    $SNMPVersionObject = [Lextm.SharpSnmpLib.VersionCode]::V3
    #}
}

$WalkMode = [Lextm.SharpSnmpLib.Messaging.WalkMode]::WithinSubtree

#try
#{
    [Lextm.SharpSnmpLib.Messaging.Messenger]::Walk($SNMPVersionObject,$Server,$Community,$OIDObject,$results,$TimeOut,$WalkMode) | Out-Null
#}
#catch
#{
    #Write-Host "SNMP Walk error: $_"
#     return;    
#}
    $res = @()
    foreach ($var in $results) {
        $line = "" | Select IP,OID, Data
        $line.IP = $IP
        $line.OID = $var.Id.ToString()
        $line.Data = $var.Data.ToString()
        $res += $line
    }
 
    $res
}


function Convert-UTCtoDateTime{


    #Parameter Binding
    [CmdletBinding()]
    param(
        [Parameter(Mandatory=$True,Position=1)][string]$UTC,
        [Parameter(Mandatory=$false)][switch]$ToLocal
        )

    #Breakout the various portions of the time with substring
    #This is very inelegant, and UTC 
    $yyyy = $UTC.substring(0,4)
    $M = $UTC.substring(4,2)
    $dd = $UTC.substring(6,2)
    $hh = $UTC.substring(8,2)
    $mm = $UTC.substring(10,2)
    $ss = $UTC.substring(12,2)
    $fff = $UTC.substring(15,3)
    $zzz = $UTC.substring(22,3)

    #If local, add the UTC offset returned by get-date
    if($ToLocal){
    (get-date -Year $yyyy -Month $M -Day $dd -Hour $hh -Minute $mm -Second $ss -Millisecond $fff) + (get-date -format "zzz")
    }
    #else just return the UTC time
    else{
    get-date -Year $yyyy -Month $M -Day $dd -Hour $hh -Minute $mm -Second $ss -Millisecond $fff
    }
}
    

Monday, March 30, 2015

Group Policy Fails to Apply on Domain Controllers With IPv6 disabled

Forward

Let me start by saying, don't disable IPv6 on domain controllers. There's no reason for it, and it will cause you more headaches in the long run. From what I've read, it used to be best practices in the early days of IPv6, but Windows is smart enough now that it shouldn't have any problems with it.

So I inherited a domain setup where the domain controllers (Running Server 2008 (not R2)) had IPv6 disabled for some reason. It's not well documented why this was done, and it's on my list of things to fix, but for the moment I'm stuck with it.

As to why I'm still running DCs on 2008, shut up it's on my to-do list.

Solution

Disabled 6to4 adapter. 6to4 adapter will register itself in DNS and cause lookup problems. I couldn't find a good way to tell the adapter not to register itself, so I settled for disabling it. From an admin command prompt:
netsh int ipv6 6to4 set state disabled
been running this way for a week or so, and haven't had any more problems. I am able to run a gpupdate on the domain controllers and have it apply successfully.

Problem & Full Story

 See "Forward" as to why I am operating DCs with IPv6 disabled.

Started running into this problem a while back, but had never had the time to troubleshoot it fully. Domain Controllers would randomly stop working correctly requiring reboots and extensive testing. These problems would take the form of

  • Slow logins -- stuck at applying user settings
  • Slow boot -- stuck applying computer settings
  • "no trust relationship" errors
  • DCs failing to apply group policy updates via "gpupdate /force"
    • Updates would apply correctly on reboot
Eventually, going through logs, I was able to narrow it down to a DNS issue. Mostly there were "RPC Server Unavailable" Errors which indicated a lookup failure. The DCs also function as DNS servers, so the fact they had lookup problems when nothing else seemed to was double strange.

Parsing through the "Forward Lookup Zones" I found that the DCs were still registering IPv6 addresses. If I cleared those addresses out manually then everything seemed to work. The DNS entries I cleared out looked like this:

Name                    Type                    Data                    Timestamp
(same as parent folder) IPv6 Host(AAAA)         2222::etc::2220         2/2/22
(same as parent folder) IPv6 Host(AAAA)         2222::etc::2221         2/2/22
...
DC1                     IPv6 Host(AAAA)         2222::etc::2222         2/2/22
DC2                     IPv6 Host(AAAA)         2222::etc::2223         2/2/22

After clearing these out everything would work for awhile. But the entries would eventual re-create themselves and the problems would come back. The (same as parent folder) -- which is a reference to the resolution of mydomain.com -- would recreate every hour or so, whereas the DC1,DC2 entries seemed to only come back on a reboot. It was baffling because the IPv6 adapters were totally disabled, so I couldn't figure out why/how they were continuing to register themselves.

After looking through ways to more forcibly disable IPv6 (through registry hacks, etc.), and deciding that was a bad idea, I thought to look more closely at the addresses that were being registered.  Issuing a ipconfig /all, I realized that those addresses were associated with the 6to4 adapters, not the actual ipv6 interface.

A quick google later to find out how to disable the 6to4 adapter and everything was in order. As mentioned above it's been a week and I haven't seen any side effects to disabling the 6to4 adapter.

Long-term solution is to reenable IPv6 on the domain controllers, but it's been disabled for so long, and since I have no idea why it was disabled in the first place, that will require more careful testing.

Monday, March 16, 2015

Print Server CPU spikes on reboot -- Spoolsv.exe

Solution

Problem in my case turned out to be a bad driver. Luckily it was a driver from an old printer that was no longer on the network.

Cleaning out print drivers is pretty easy: Open up two instances of print manager, go to the Drivers menu on one, and the printers on the other. Arrange the columns on the printers page so that you can see the printer name and the driver name. Sort by driver name, then compare the list to the ones in the Drivers menu. This makes it easy to see which drivers are no longer in use.

If you've cleaned out old drivers, and the problem is still occurring, you may need to start removing printers until the problem stops. Remove a printer, reboot the server, and see if the problem still exists. Of course, make sure you copy the printer information so you can re-add it later. Once you've determined which printer/driver combo is causing the issue, see if you can find an alternative driver for that printer. Most printer manufacturers recommend using their "Universal Print Driver" in a print server environment, rather than the 'named' or 'model specific' driver.

 Problem

In brief, our print server would hang with 100% CPU usage whenever it was rebooted. This necessitated spoolsv.exe (the 100% process) to be manually terminated, and the spooler and all dependant services to be restarted. After manual termination/restart, the CPU levels would remain normal. Printer shares would be unavailable during the CPU spike.

Machine Specs:

  • Server 2008 R2 -- fully updated
  • VM on top of XenServer 6.2 - Intel based system
  • 2 core
  • 3 GB of RAM
  • Additional Printer Related Software
    • Print Manager (print tracking software)
    • Citrix Universal Print Server
Things I tried that didn't help

  • Windows updates
  • Updating paravirtualization driver.
  • SFC / Checkdisk/ other system file scans
  • Re-installing drivers/print queues  
Problem ended up being an issue with some unused print driver. Apparently the spooler service loads up drivers even if they are not being used by any print queue. So the only way to fix the problem was to remove all drivers from printers that had been retired -- see details in Solution above.