Tuesday, September 30, 2014

More Printing Problems - Spooler and Citrix Print Manager Crash


Today the solution appears to be use HP UPD (I'm using 5.8, for the record) and recreate print queues (fancy name for shared printers) and have a nice script to handle failures elegantly.

Use of the HP Universal Print Driver is pretty much mandated for use in a terminal server environment (check out the HP compatibility list here (pdf)). Most HP printers new/old do not support the use of their device specific driver in a Citrix Xen* environment. 

Recreating the print queues apparently helps for not-quite-adequately-explained reasons. Apparently switching which driver a print queue is using can cause some sort of corruption that can crash the spooler. So you have to delete and re-add the shared printer (you can use the same port, etc.) but starting with the UPD driver, rather than using the device-specific one and changing later. I've done this for a couple of printers and it has already drastically reduced the number of crashes, will be recreating other printers that seem to be causing problems and will update this will progress.

Having a script that handles failures nicely is key to reducing user impact. Printers are always problematic, especially in a Xen/RDS environment, so expecting 0 crashes is probably over optimistic. I've written a script that runs when a service is detected to have failed (I do this through the 'recovery' tab in the service properties). The script restarts both the spooler and the Citrix Print Manager Service. For some reason, if these two are not restarted together, they don't seem to talk to each other. So whenever one fails, both need to be restarted. I'll put the script at the bottom of the article.


We've had more problems with printing since fixing the issue with the Citrix Print Client. The issue now is that the print spooler on our terminal server keeps crashing and it causes people not to be able to print, printers not to map, etc.

In brief, there are two services "Citrix Print Management" (CpSVC) and "Print Spooler" (spooler). Even though we are no longer using the Citrix Print Client, we still the the CpSVC service because it handles the mapping of printers through Citrix Group Policy. The Citrix Group policy gives us some additional functionality when mapping printers that would be difficult to replicate in normal AD group policy. Anyway, when either of those services crash it breaks everything; Simply setting the service to restart on crash doesn't work either. The processes must be restarted together, otherwise they don't seem to talk to each other.

I've written a small script that reboots both services whenever one fails, which minimizes the impact of a failure, but I'm still working on solving the underlying problem. So it's time for another long rambling post trying to figure out what's happening, the last one went pretty well, so let's give it a shot.


XenDesktop Controller: Server 2008r2SP1,  XenDesktop 7.0, Physical Server with way more resources than necessary
XenDesktop Hosted Desktop: Server 2008r2SP1, runs RDS server/ XenDesktop 7.1, clients connect from Wyse Xenith2 thin clients, about 100 possible clients, but generally have only 30-50 at any given time.
Print Server: Server 2008R2SP1, Microsoft Print Server

Errors in Event Viewer

Here's a brief rundown of the various errors I've gotten and what I've been able to find out about each.

splwow64.exe crash

Type: Error
Source: Application Error
EventID: 1000
Task Category: (100)
Faulting application name: splwow64.exe, version: 6.1.7601.17777, time stamp: 0x4f35fbfe
Faulting module name: ntdll.dll, version: 6.1.7601.18247, time stamp: 0x521eaf24
Exception code: 0xc0000374
Fault offset: 0x00000000000c4102
Faulting process id: 0xeb30
Faulting application start time: 0x01cfd83a3b2e4f74
Faulting application path: C:\Windows\splwow64.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll

 splwow64.exe is a process that translates the x64 print drivers for use by 32-bit applications. E.G. All of our print drivers must be 64-bit, because it's running 2008R2SP1, but the server runs 32-bit Office (for various plug-in compatibility). When office wants to print it has to go through splwow64.exe because it wouldn't know what to do with a 64-bit driver.

As for why this crashes, I have no idea. You see the "faulting module" is one "ntdll.dll", and the error code is "0xc0000374". ntdll.dll is explained here in Wikipedia, I'd tried to summarize but since my understanding is vague at best, probably best to read it yourself. "0xc0000374" is an error code that indicates "Heap Corruption", which is a fancy way of saying the memory was modified in a way that wasn't expected. Neither of these bits of information are particularly insightful, but they come up over and over in these errors.

spoolsv.exe crash

Type: Error
Source: Application Error
EventID: 1000
Task Category: (100)
Faulting application name: spoolsv.exe, version: 6.1.7601.17777, time stamp: 0x4f35fc1d
Faulting module name: ntdll.dll, version: 6.1.7601.18247, time stamp: 0x521eaf24
Exception code: 0xc0000374 OR 0xc0000005
Fault offset: 0x00000000000c4102
Faulting process id: 0x9044
Faulting application start time: 0x01cfd8232b50fcd7
Faulting application path: C:\Windows\System32\spoolsv.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
This error is very similar to the splwow64.exe one above, with the exception of the "0xc0000005" variant. This, as far as I can tell, is a memory access violation.

Couldn't Load Print Processor

Type: Error
Source: PrintService
EventID: 365
Task Category: Initializing a print Processor
Windows could not load print processor hpcpp160 because EnumDatatypes failed. Error code 126. Module: 18\hpcpp160.dll. Please obtain and install a new version of the driver from the manufacturer (if available), or choose an alternate driver that works with this print device.

This error is a bit more informative. There are variants for other print processors (e.g. hpzppwn7). hpcpp160 happens to be the HP universal print driver (version 5.8). Anyway it's our first indication of something wrong with the print service. Problem is, most of the time this print processor has no problems. Most of our printers use the HP UPD, and they work most of the time.

I've also tried reinstalling the UPD (on the client, on the server would require more extend downtime). This hasn't had an effect.

Citrix -- "Environment is incorrect" "no printers were found" "printer auto-creation failed"

Type: Error
EventID: 1114 / 1116

I'm not going to list out the full text of these because they are erroneous (at least as far as my problem goes) -- see this forum for more info. Basically these errors can be logged even if printer creation is succeeding. It's very obnoxious. It's possible it's logged because printers are not being deleted at logoff, but I haven't found anything to suggest that is still an issue (that forum post is pretty old).

Print Spooler Can't copy file

note: you must enable the PrintService operation log to see this error. In event viewer find it under Applications and Services > Microsoft > Windows > PrintService. Right-click the operational log, select "enable log"

Type: Error
Source: PrintService
EventID: 811
Task Category: Executing a file operation
 The print spooler failed to move the file C:\Windows\system32\spool\PRTPROCS\x64\hpcpp160.dll to C:\Windows\system32\spool\PRTPROCS\x64\202_hpcpp160.dll, error code 0xb7. See the event user data for context information.

This message again may vary with the print processor (replace hpcpp160.dll with whatever.dll). This is an odd message. The folder it references is full of duplicate print processor dlls (1_hpcpp160.dll to 499_hpcpp160.dll). I have no idea why, this is the current lead I'm working on.

Things I've Tried

  1. Create Script that restarts processes
    1. First did this restarting "spooler" and "cpsvc" every 5 minutes
      1. this technically worked, but caused some strange behavior and is over inelegant
    2. Set "spooler" and "cpnsvc" to run the script when either crashes
      1. this can be done in the services MMC snap in.
      2. Still doesn't solve the underlying problem, but is a nice band-aid fix until I can figure the bigger issue out
      3. it's also way more elegant that the "restart every 5  minutes" solution.
      4. Note: had to change CpSvc to log on as a local service with permission to interact with desktop (was just local service), otherwise the script wouldn't run correctly when it failed.
  2. Moving printers to the HP UPD
    1. Thought here is that one of the device-specific print drivers wasn't terminal compatible
    2. This hasn't exactly panned out. Moved device-specific printers to the UPD but errors continue to show up. I've just finished this migration recently, so maybe it'll pay off over time.
  3. Clearing out system32\spool\prtproc\x64
    1. This folder is full of duplicate .dll files (see "print spooler can't copy file" above) 
    2. Found out not to delete everything from that folder. The WinPrint.dll file will not recreate itself.
    3. Print spooler still crashed (0xc0000005). at like 5am when no one would have been using it. So that's fun.
      1. Actually, someone did log it at 5am, just seconds before the crash. So that's something to go on maybe.
    4. Printing hasn't gotten any worse, so at least I haven't broken anything
    5. This has at least seems to have stopped the 811 errors. Watching to see if the prtproc folder starts to build up again.
  4. Recreating Print Queue
    1. Some things I read suggested that print queues my become corrupt when switching drivers/print processors.
      1. Print queue is the technical term for a shared printer
    2. So I deleted and recreated the print queues that seemed to be causing the most issues
      1. "most issues" was determined by cross referencing the crash time with our user tracking to determine which stations (and thus which printers) were most recently logged into before the crash.
    3. This actually appears to have had some effect. I've only had one crash (and it was the splwow64.exe crash, not the spooler or CpSvc) today.
      1. Today's Friday, so load is low, but will continue to monitor.

Days Pass....
Spooler/CpSvc/splwow64 continue  to crash, but much more infrequently. Average maybe once or twice a day, much lower than the every couple of hours it used to be. I am going to continue to create print queues and see if I can eliminate the crashing all together and will update this page as I learn more.


write-host "Shutting down Citrix Print Manager"
stop-service -force cpsvc
write-host "Waiting for CpSvc to shut down Gracefully" -nonewline

while($(Get-service cpsvc).Status -ne "Stopped" )

if($Count -gt 5)

    write-host ""
    write-host "CpSvc has not shutdown gracefully, shutting down manually"
    stop-process -force -Name cpsvc
write-host "." -nonewline
Start-Sleep 1
write-host ""

write-host "Shutting Down Print Spooler"
stop-service -force spooler
write-host "Waiting for Spooler to shut down Gracefully"

while($(Get-service spooler).Status -ne "Stopped" )

if($Count -gt 5)

    write-host ""
    write-host "spooler has not shutdown gracefully, shutting down manually"
    stop-process -force -Name spoolsv
write-host "." -nonewline
Start-Sleep 1
write-host ""

write-host "Bringing Spooler Back up"

start-service spooler

write-host "Bringing Citrix Print Manager back up"

start-service cpsvc

date >> c:\temp\restartprinters.txt


Wednesday, September 10, 2014

Printing issues on Win 7 Virtual machines with XenDesktop -- Citrix Universal Print Client


The  fix for this ends up being to remove the "Citrix Universal Print Client" from the XenDesktop clients. According to certain sources, this happens when the UPC is attempting to contact the Universal Print Server, even when a Universal Print Driver isn't being used. The server obviously doesn't respond, and so there's a considerable timeout before it falls back to windows printing.

I'm skeptical that this is actually what's happening, at least in my environment, for a couple of reasons.
  1. The timeout occurs even when using the UPS/UPD/UPC.
  2. The timeout occurs for some printers/drivers and not for others

The only way I've found to remove the UPC is to reinstall the XenDesktop Client without the UPC. From command line it looks something like:

#note: reboot before running the first command, and between each command. If you don't specify the "/noreboot" flag, it will reboot automatically after each command.
#<XenDesktopDir> = unzipped iso location
#<XDInstaller> = <XenDesktopDir>\x86\XenDesktop Setup\XenDesktopVdaSetup.exe
#   or for 64-bit = <XenDesktopDir>\x64\XenDesktop Setup\XenDesktopVdaSetup.exe

#Remove Current install
<XDInstaller> /quiet /removeall

#Reinstall Without UPC
#You can check here for the flags you need for any other customizations you make
<XDInstaller> /quiet /Components vda /EXCLUDE "Citrix Universal Print Client" /logpath "c:\temp\xdinstalllogs\"

#Configure Controller hostname/port
#If you set this through group policy, you shouldn't need this step
<XDInstaller> /quiet /reconfigure /controllers "mycontroller.mydomain.com" /portnumber 9999



Been having some issues with printing from our thin clients. The most common symptom is that whatever program is trying to print (word, notepad, browser, appears to effect all about the same) locks up for 30-60 seconds. During this time the a window saying "connecting to printer" may be present (though not always), and the main program window appears unresponsive (not responding in task manger).

 I'm writing this as I troubleshoot so, apologies if it's a bit schizophrenic.


Client: Windows 7 x86 - fully updated -- mostly, some are not 100% updated, but all have at least SP1, and it doesn't appear to make much of a difference. Clients have XenDesktop Client installed (7.0/7.1 -- I end up trying 7.5 as well)

HyperVisor: XenServer 6.2.0 (fully updated, SP1 + a couple more updates). Clients all have XenTools installed.

Print Server: Server 2008R2 (VM) x64. Has "Print Manager Plus" software, also has Citrix Universal Print Server.

New Printer Server: Server 2012R2 (VM) x64. Does not have Print Manager Plus.

Drivers: To be discussed

A More Detailed Description of the Problem

My general view of the problem is that the printer is taking a long time to respond to the program trying to print to it. The main way this appears, is when you click "print" in a program, it tries to contact the printer to get availability, status, capabilities, etc. This takes a long time to finish when the problem occurs, which means when you click "print" the program stops working for 30-60 seconds. The window will show "not responding". Sometimes a box saying "connecting to printer" will appear, but not always.

This problem will happen several times when trying to print, because the computer appears to talk out to the printer several times. First when you go to the print menu to select a printer, then again when you click on the printer itself (to select it) then again when you click print to actually send the job. This means there can be a delay of several minutes for a user trying to print a basic document. This is understandably very annoying.

What I've tried so far

My first thought was that it had something to do with the Citrix Universal Print Server. That's why I built up the 2012R2 server. However the problem is present when mapping printers from that server. This only happens with some printers, other printers work just fine.

Mapping the printers directly (via IP on the client) appears to work fine as well.

Most of our printers are HP, and thus most use the HP Universal Print Driver. This is currently my main suspect, but it's unclear as to why that would cause such a problem with only VMs. Physical Clients (your standard desktop machines) do not have this problem.

I also thought it might be the universal print manager (Client side of the Citrix Universal Print Server), but I have disabled that service and deleted the Citrix Universal Print Driver from a machine and the problem persists. It's possible that the Citrix software still causes some problems, currently installing a vanilla windows 7 machine to test this theory.

Problem occurs whether connected over XenDesktop, Remote Desktop, or Directly through XenCenter Console.

Being an Administrator or  not does not appear to have an effect.

Using the FQDN or IP of the server rather than the hostname doesn't appear to have an effect.

I've tried Type 3 and Type 4 drivers (What does that mean?). Both word fine from the print server (that is the print server never has problems printing test pages). Type-4 drivers are not technically supported on Windows 7, so when a windows 7 machine trys to connect to a type-4 printer, they are given a "enhanced Point and Print Compatibility driver". These work fine, however this is not an apples-to-apples comparison, because there is no type-4 HP universal print driver. So the Type-3 not working where the Type-4 does is as much comparing device-specific to Universal as it is type-3 to type-4 (trying to find type-3 device specific drivers). But for what it's worth the HP UPD (type-3) has the problem, the device specific (type-4) do not.

Older version of the HP UPD appear to have the same problem. Nor does PCL 5 vs PCL 6 vs PS.

For some reason HP doesn't always have device-specific drivers on their website -- they'll just like the UPD. So it's really hard to find a printer that has the UPD, Type-3 Device Specific, and Type-4 Device Specific to do some real testing on. Testing the difference between type-3 DS and type-3 UPD at least..... Type-3 DS driver has the problem as well, at least on my "HP LaserJet 400 M401".

Problem also occurs on one of my terminal servers (2008R2),  also running XenDesktop client, but is a physical server.

Problem does not appear to occur on another terminal server (2008R2) which is a virtual machine, but is not running the XenDesktop Client. Doing further testing to verify. For all intents, problem does not exist on this machine. You can see the "connecting to printer..." dialog come up for a split second, but it's almost as fast as a physical machine, not enough to make a difference user experience wise. So it looks like we may be looking at the XenDesktop software as the culprit. It's strange that it only has problems with certain (mostly HP) drivers.

As I mentioned, I've also done tried using the Citrix UPD, but it shows the same problem. I'd have to do more testing to verify whether the CUPD locks up only when using it to print to printers that otherwise exhibit those symptoms, or whether the CUPD is just broken in general.

For now, I'm getting a clean windows 7 machine built up and will install software one-by-one to determine when the problem starts happening.


OK, fully updated windows 7 machine with nothing else on it is ready. My method here is
  1. Map the printer
  2. Wait a minute
  3. Open notepad
  4. Try to print
  5. Wait another minute
  6. Open Printer properties
  7. Unmap Printer
  8. Reboot after each full test (between changing variables, not between each printer)
If either 4 or 6 take more than about 10 seconds, I'll consider the problem to exist. I try this with three different printers, all of which have exhibited the problem in the past. Here are the variables and results of the tests.

  1. Clean machine
    1. Problem does not exist
  2. Install XenTools
    1. Problem does not exist
  3. Domain Join
    1. Problem does not exist
  4. Move to Correct OU
    1. Problem does not exist
  5. Install XenDesktop Client (VDA)
    1. Problem defiantly exists.
So I guess that settles it. It's something in the XenDesktop VDA that breaks printing. Keep in mind I did not run any of the XenDesktop Optimizations, so it's not one of those; the problem is within the client itself, or some change it makes without the option not to. So let's see if I can narrow down specifically what's causing it here.

  • This Forum suggests stopping "Net Driver HPZ12" service (some sort of HP monitor thing)
    • Did one better and disabled "Pml Driver HPZ12" as well (another HP monitoring thing)
      • This did not help
    • Lets try disabling the service and rebooting.
      • No Dice
  • This Post on experts exchange says to turn off bidirectional support in the printer properties. Lets try that.
    • It's under the "ports" tab in printer properties. I changed it on the server, then deleted/re-added it to the client.
    • This doesn't appear to have any effect
  • Downloading XenDesktop 7.5 -- just the VDA I'm not upgrading my whole installation yet. Since This "clean" machine isn't even joined to the XenDesktop Controller, I don't see how that would have any effect anyway.
    • Installed 7.5 VDA, doesn't appear to have had any effect.
    • Just to be sure I removed the device and uninstalled all drivers and tried again
      • Still locks up
  • Disabled Citrix Print Manager Service
    • I've tried this before, but thought I'd try again under 7.5
      • Still locks up
  • Disabled "Citrix Personal vDisk" Service
    • At this point I'm just disabling Citrix Services one-by-one to see if there's any change
    • Still locks up
  • Disabled "Citrix Profile Management" Service
    • Still Locks up
  • To reduce redundancy, disabled each "Citrix Service" one-by-one
    • Still Locks up
  • Tried disabling "allow direct connection to printers" in Citrix Policy.
    • This made it slightly less terrible. It still hangs for a bit while "connecting to printer" but the program doesn't stop responding (or at least, windows doesn't think it has). Not a perfect solution but it's progress at least.
  • Found this forum post, trying the solution listed at the bottom - installing the VDA without the universal printing component
    • Completely uninstalled current VDA first
      • Verified problem had gone away
    • Reinstalled using "XenDesktopVdaSetup.exe /components vda /EXCLUDE "Citrix Universal Print Client" /logpath "c:\ctxinstall.log" /quiet /noreboot"
      •  Sweet baby Jesus I think that actually worked
      • Yep, that appears to solve the problem
I've applied this fix to my main two XenDesktop Terminal Servers (aka ServerOS Hosted Desktops) and it appears to have solved the problem. You no longer get the "connecting to printer..." dialog, the program doesn't go to a not responding state, users will hopefully got breaking things by clicking a bunch of buttons while it appears to be frozen.

If the forum post I linked above is to be believed, the issue is that the XenDesktop client attempts to talk to the universal print server, even when the UPD is not being used. I'm a little skeptical that this is the entire problem, because this error would happen even when using the UPD/UPS. But at any rate it's fixed. Obviously this precludes using the Citrix Universal Print Server in the future, but it's honestly been such a pain to manage/get working that I have to call that a 100% positive effect.

    Friday, August 29, 2014

    IMacros for Firefox Failure Corde 0x80500001 (Error code: -1001)


    The root issue is the encoding of the files. I've had this problem before with iMarcos, but it's never been quite this specific. Usually saving the datasource (the CSV file) as UTF-8 works. But some update to Firefox or iMacros has made it really inflexible. Both the datasource (.csv) AND the macro file (.iim) file must be saved as "UTF-8 with BOM". I Used Sublime Text to do this, but any full-featured word editor should work.

    For the record, various versions of things I use.
    Firefox : 29.0.1
    iMacros : 8.8.2
    Windows : 8.1 x64

    Problem/Full Story

    I often have to fill out web forms over and over again to perform certain tasks. A lot of these web forms are poorly designed at best, and don't support batch-type inputs. So having a program like iMacros is essential for me not wanting to kill myself while filling out a DHCP registration form 200 times.

    I've used iMacros for a number of years and never had too many problems; In Firefox at least, in Chrome the sandboxing makes it an exercise in keyboard snapping frustration to read/write to files -- but that's another story. However I needed to do a bunch of the previously mentioned DHCP registrations today (this is a system managed by another department, and the web interface is the only way to do it besides submitting a work request, which can take days), and found that the Macro/CSV I had previously used to do this were not working. I received the following error message:

    Error: Component returned failure code: 0x80500001 [nsIConverterInputStream.init], line 4 (Error code: -1001)
    I'd actually run into this error before, or at least one similar to it. iMacros (or possibly firefox), can be rather picky about the encoding it uses. Previously saving the .csv I use for inputs as UTF-8 had solved the problem. Today that didn't work though.

    After fiddling around with it for a bit, I found something strange. I Created a new macro (.iim) file to see if the other one was corrupt or something, but writing/saving it through Sublime Text (not the built in iMacros editor) as UTF-8, then opening it in the iMacros editor just showed a blank file. Strange, after trying a handful of different encodings for the macro file I found one that it would recognize "UTF-8 with BOM". After saving the file with this encoding through sublime, it would show up correctly in iMacros. However, I was still getting the same error when I tried to run it. Tried saving the csv file with the same "UTF-8 with BOM" encoding, and then it ran.

    Thursday, August 28, 2014

    Citrix Receiver for Mac "Cannot start the desktop ... OSStatus -1712"


    In my case there were non-responsive processes on the mac client that were causing the problem. To resolve, I closed out of receiver and closed any active desktop connection. I then brought up the activity monitor (command+space to bring up the search, enter "activity monitor"). There were several Citrix processes, one non-responsive process with the name of the personal desktop that wouldn't load and a few helper processes. I force-quit all Citrix processes, then restarted the receiver client. Connected to desktop successfully at that point.

    It may not have been necessary to force-quit all Citrix processes, but it doesn't seem to have had any consequences, they started back up when I reloaded receiver.

    Problem / Full Story

    Had a user this morning that couldn't connect to their windows desktop over XenDesktop (7.1). User is one of our few Mac (running Mavericks) users, and uses XenDesktop to get to windows applications he needs. When he tried to log in this morning, he got the following error when he tried to connect to his windows 7 machine.

    Cannot start the desktop "Personal Desktop"
     Contact your help desk with this information: The application "Personal Desktop" could not be launched because a miscellaneous error occurred. (OSStatus -1712).
    Odd thing was, he was able to connect to his Windows 8 desktop just fine. So the connection to the server was working, as was the connection to at least one VM. The Win7 machine was showing up as registered and ready in Citrix Studio on the XenDesktop Controller. Win7 appeared to be responsive when interacting with it through XenCenter. I tried restarting the Windows 7 machine but the error persisted. A brief look through the longs on the Win7 machine and the XDC didn't show any errors, so it seemed like the problem wasn't server-side. Had the user logout/close receiver on his machine and reopen it, but the error continued to occur.

    Up in the user's office I brought up the activity monitor and saw the unresponsive process -- see "Solution" above. After killing and restarting all citrix processes the user was back up and running. Rebooting the Mac probably would've had a similar effect.

    Tuesday, August 12, 2014

    a security package specific error occurred - Security-Kerberos EventID 4


    Root problem was that there were static DNS entries set for some computers whose IP addresses had changed. Deleting static entries and waiting for changes to propagate out solved the problem.

    Full Story

    Had an issue this morning where some new computers on our network were not getting printers mapped. This is not an uncommon occurrence, because printers, but the cause of the problem was a new one for me. These computers had just been upgraded (new hardware, same hostnames) and seemed to be functioning fine on the domain. The print driver was working fine on other machines, and the usual fix, restarting the print spooler, had no effect.

    Trying to access the Event Viewer on the lab machines I got the error "A Security Package Specific Error Occurred". This error (or a variation) came up trying to access the computer via any WMI / RPC / DCOM method.

    On the print server I had the following error, listed as Level:Error, Source:Security-Kerberos, Event-ID: 4

    The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server MYLAB-04$. The target name used was cifs/MYLAB-02.My.Domain.Com. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please ensure that the target SPN is registered on, and only registered on, the account used by the server. This error can also happen when the target service is using a different password for the target service account than what the Kerberos Key Distribution Center (KDC) has for the target service account. Please ensure that the service on the server and the KDC are both updated to use the current password. If the server name is not fully qualified, and the target domain (MY.Domain.Com) is different from the client domain (My.Domain.Com), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

     One thing jumped out here right away, the error is from lab computer 04 (SPN: MYLAB-04$), but the FQDN is listed as computer 02 (cifs/MYLAB-02.My.Domain.Com). So that set off some alarm bells, but I still did some additional research before jumping in.

    Supposedly this error can be caused by a number of things (a Google of "A Security Package Specific Error Occurred" returns about 6 difference causes on the first page of results). In my case, as mentioned above, was a DNS issue. While upgrading these lab machines, the IP addresses we assigned through DHCP changed slightly. Normally, we just let the machines register themselves with the DNS server after they pick up their IP via DHCP, we don't have many static DNS entries. For some reason, these machines had static entries, though, so our DNS server was resolving their hostname differently than AD was, which is what caused the authentication errors. Deleting the static entries and waiting (DNS changes can take a while to replicate) solved the problem.

    Thursday, July 24, 2014

    Samsung Galaxy Light - WiFi calling "called ended", WiFi messaging fails


    Update 9/24/14

    Been using for a bit now, haven't had many problems. Still have an occasional call that goes straight to voice-mail or a text that won't send until you reboot, but the problem is much less pronounced than before, and doesn't have the same repeatability that it had before.

    Writing this now because I just got a new firmware upgrade today. We'll see if it has any effect on the situation. That page says that it only enables "in flight texting", but it wouldn't surprise me if there were additional bugfixes included.

    --------------------------------------- Original Text -----------------------------------------------------

    In contrast to most of my posts, the solution to this is actually available pretty readily on the internet . I am posting this to confirm that it worked on my two phones, because people are pretty bad at updating forum posts once they've found a solution.

    The fix in this case is to run a firmware update via the "Samsung Kies" software. This firmware update is not available OTA. Testing over a ~24 hour period I have been unable to reproduce the problem, which was quite easily reproduced before the update.

    While researching this problem, I found many similar reports of other Galaxy phones (s3,s4,etc) having the same problem. None of those reports seemed to have a resolution, only the Galaxy Light thread had anyone confirm that they had fixed it. So, if you have another Galaxy phone with the same problem, I would check to see if there is a firmware update available via the Kies software.

    Installing and running Kies software

    1. Download the Kies software for your platform from Samsung's website
      1. There are two versions "Kies" and "Kies 3". For the Galaxy Light, you want "Kies", newer phones may need the "Kies 3"
      2. There is also a version for Mac, only one version though, not sure if that works for all phones or what.
    2. Install the Kies software.
      1. There isn't much to do here. Just click next a whole bunch really. Though I did opt to install the "universal drier tool", not sure if that's necessary. 
    3. Open the Kies software, you should see a "connect your device" type prompt. Connect your device.
    4. If this is the first time you've plugged in your phone to the computer it may take a few minutes to install drivers.
      1. Note: I did have one of my phones lock up while connecting it to the computer (screen and hardware buttons became unresponsive). A Force reboot fixed it.
      2. The device needs to be in MTP mode, not PTP mode. Kies will warn you if it's not. 
    5. When I hooked up my phones, it immediately prompted me to do the firmware upgrade.  If this doesn't happen for you, the "Basic Information" tab should show the current firmware and whether or not it's up to date.
    6. Follow Kies instructions to upgrade the firmware. 
      1. The first time I ran this, I let the phone go to sleep while Kies downloaded the firmware update. Since the download took a while (I don't have super fast internet at home) the phone disconnected from Kies and I had to start the process all over again. Had to sit there swiping the screen back and forth while the download happened to keep it from sleeping/disconnecting.
    7. Phone will reboot and install, don't turn it off or do anything to it while this happens.
      1. The upgrade performed just fine for me on both phones, with no loss of data. Still, if you have critical stuff on your phone (why do you have critical stuff on your phone?, keep that stuff somewhere less steal-able) maybe back up the phone (can be done with Kies (Backup/Restore tab), or about 1000 other things) before doing the upgrade, just in case.
    That's about it. Once the firmware update is complete you shouldn't have any more problems with WiFi Calls not dialing, not receiving calls on WiFi, or not being able to send/receive text messages on WiFi.

    In interest of full documentation, One call I made right after the upgrade had really poor call quality (sounded like I was underwater). This was one call and have not had the problem reoccur since. 

    The Problem / Full Story

    We recently switched to T-mobile because they're reasonably cheap (for what you get) and their business model is slightly less troublesome than most big carriers. <Rant>took about 8 hours on the phone over 2 days to get the plan set up properly because the original guy who sold us the plan didn't know what he was doing </Rant> . Anyway we brought our devices, because you can pick up the Galaxy Light on Amazon for dirt cheap.  It's not a high-end phone, but it's a reasonable spec with a fairly recent version of android (4.2.2). 

    But enough advertising, you wouldn't be here if you weren't having a problem. The phones worked fine for a few days, but we started having problems with the WiFi calling within about a week. WiFi calling was a big deal for us because all carriers (besides Verizon, but fuck them) have not-so-great coverage in my town, but we have WiFi pretty much everywhere we go. So problems with the WiFi calling was problems with our service in general.

    Problems were as follows. If the phone was allowed to sleep for awhile (seemed to be 30-45 minutes on average) WiFi calling would stop working. This means, with out a cell signal (which I don't get at work because my building is made out of concrete and florescent lighting) no calls could be made or received and no texts could be sent or received. Not that the phone was aware of this, the WiFi calling icon in the notification bar was still blue and it said it was making calls over WiFi. When you actually tried to place a call, however, it would immediately end the call (at 0:00 seconds) and call status would be "Call Ended". Looking at call history you would see "Canceled". Trying to send Texts would result in a "Failed to Send" message. People trying to call us would get send straight to voicemail, or occasionally one ring then voicemail. Then only way to restore service was to turn WiFi calling off-and-on (usually) or reboot the phone. This fix would only last until the phone went into some low sleep state (seemed to be ~30 min of screen off).

    After much internet searching and playing around with settings on the phone I came across this thread (also linked at top) about the WiFi calling on the Galaxy Light. A non-OTA firmware upgrade was available and fixed the problem (see above for steps).

    Here's some things I tried that did not work
    1. Clearing data (via application manager) from "WfcService" and "Wi-Fi Calling Settings"
    2. Changing Wi-Fi calling preferences (prefer cell, etc.)
    3. Different Wi-Fi networks
    4. Standing close to router (router is on my desk at home, so while sitting at desk phone is <5 feet from router)
    5. Turning off voice control
    6. Turning off other wireless radios (bluetooth, gps, NFC, etc.) 
    Here's some other things I've seen reports of people trying that have not worked
    1. New phone - This appears to be a problem with at least all Galaxy phones. Seen reports of people getting phones replaced 4 or more times without resolution. 
    2. Factory Reset
    3. Changing network mode (LTE/WCDMA/GSM, WCDMA/GSM, WCDMA only, GSM only)
    4. Changing Sim cards/upgrading sim cards
    5. opening ports on router

    Monday, July 21, 2014

    Condusiv V-Locity - setup and first thoughts


    I'm going to be pretty brief here because I feel I'm not going to have much to actually say about this piece of technology (note from the future: I wasn't able to get this to work in my environment, read on if you're interested in the problems I ran into, but otherwise this article probably isn't worth your time). We'll leave it at : managing Storage IO in a virtualized environment is a pain, so I've taken to investigating some technologies that look at improving storage performance without simply buying more storage devices. This post is written in steam of consciousness style as I go through the setup process. I try to document anything I notice and or am thinking during the install. I do some minimal editing afterwords, but for the most part it'll be a rambling mess.

    V-locity is a program from Condusiv, a name that was obviously dreamed up by someone with no respect for spoken language, ease of typing, or autocorrect. Here on I'll probably just refer to it as "the program." The idea behind the program is that windows file system driver is poorly optimized for an age of virtualization and non-local storage. Breaking file read/writes into multiple chunks isn't noticeable on local storage, but can add serious overhead when it has to go over Iscsi. So through a new driver an a bunch of caching, the program hopes to optimize storage to give you better density without buying more hardware (or increasing Cap ex, as they say) </marketing>. I won't go into much of the details of how it works here (I'm still a little fuzzy after a webinar and like 6 sales calls, and let me tell you it's not for lack of paying attention) if you're interested you can read all about it here.

    First let me say, Condusiv certainty isn't trying to save you any money over buying more hardware. We were quoted a price of around sixty thousand dollars (+ about seven thousand in yearly licensing) to run on our three servers (32 cores each, which is how it's sold). That's insane, that's roughly three times as much as the server's themselves cost. This more or less makes it only an option if you're out of rack space, or for whatever reason can't move your data to faster storage devices.


    I've got a test environment setup. 50 VMs and a server. VMs are running on some Dell R610 servers connecting to their storage over a 6GBs Direct attach SaS link. Server are running XenServer 6.2.0 (sp1 + all other patches). VMs are 64 bit windows 7, all updates installed, basic office applications for testing. Tests will use XenDesktop to measure login performance (connecting via thin clients) and a more manual approach to measure some application launches (visual studio 2012 is one in particular we've had take a really long time to load on VMs due to excessive file system access during first-run)

    Setup is broken into three parts, the VMC (controller) and master node (velocity server), and the clients. Since this is a test setup, my VMC and master node are living on the same server. VMC setup is simple, just click next and it installs inself and a webserver to interface from. One thing, it doesn't tell you to access it via the web page. The installer just finishes and you have to figure it out. The setup instructions don't really say this directly either, you just have to kind of guess at it (I figured it out because the install directly had a bunch of .js, .html, etc. files).

    After that, the setup runs a discovery on your domain to find machines to install on. I didn't set any sort of filters on this, but it is currently stuck (about 20 minutes) on 740/742, we'll see if it ever finishes.

    ... 30 minute mark now, still spinning on 740/742.
    ... well over an hour now. Neither the "close" or "next" buttons do anything.
    ... two hours and no sign of movement. I'm about to go home, so I'll let it run overnight and reboot the dumb thing if it hasn't sorted itself out by morning.
    ...Still at the same spot, think it's safe to say it's stuck, going to try restarting the service. Now it says discovery complete, 1 record processed. Sounds legit. Looking through the machine list, it seems to have detected a fair amount of my machines, but none of the VMs I created specifically for testing.

    After another restart of the VMC service and some time it picked up all my servers, but I've run into a bigger issue. The master node component won't install on my virtual server. The server meets all of the requirements listed in the various install guides and readme files, but it doesn't show up on the list of machines available for deployment. Trying to run the installer manually gives the error "OS not supported".

    Looking further, it is only presenting the option to install the master node component to physical machines. I can't find this listed as a requirement anywhere, and the sales rep/tech people say that it isn't a requirement, but that's the only option it's giving me. 

    Worked briefly with the sales rep/tech support team that's been helping me, they gave some new licenses to  try, but for whatever reason the program still only gives me the option to install to physical servers. I don't have spare physical servers lying around, so we're a bit dead in the water.

    On a hunch I looked up Vlocity + XenServer (my hypervisor of choice), and have found some conflicting reports of support for the XenServer platform (PDF). At best it has partial support, and that possibly only for the guest/client. So maybe that's the issue. Looking back through my emails I defiantly mentioned that's what I was running on (and I'm pretty sure we covered that more in-depth during one of the 7-8 phone calls they made me sit through), but maybe I wasn't clear enough.

    So, unfortunately this is where it my review of  vlocity must end. I'd spend more time with their tech support troubleshooting it, but I have other projects that need my attention. So, take my experiences for what they're worth (probably not much) but if you're looking to evaluate and are using XenServer, maybe be sure you're clear with your reps about the setup.

    Edit: Sales rep assures me that Vlocity works "with Citrix" (I haven't gotten him to say "with XenServer") and in interest of objectivity I was able to get it to start seeing virtual machines. Still doesn't see the test server I built up as a valid install location, so I'm still stuck, but there you go.