c750i deleted due to error

[rbillyg]

So, Recently we have been getting errors on our c750i that jobs are being deleted due to errors.
When I look at the logs a successful print shows the scenarionID with the print job, the code which matches the user box id, and the external server name for authentication.

a failure shows the scenarioID as 0 , no user code, and no authentication parameters. This has been difficult to troubleshoot as the same user will randomly work, then hours alter it breaks, then starts working again. Have changed the domain controllers that authentication happens at and we still have the issue. windows updates were not applied when the issue started so I am ruling that out.

hoping someone has some info on this..


sample
success:


Message : 609 - [meta sequenceId="3462"] JobInf : LogTyp="Job", No= 17012 , EntTim=2022-11-22T11:31:52, FinTim=2022-11-22T11:31:53, Nam="user", Typ="Print", RelJobNo=4294967295, SubScenarioID= 17012 , Code="User3", Nam="user", NetTyp="IntraNetwork", ExtSrvName="DomainDC", TrcCode=Non, IF="PrinterReception", Res="OK", EventTyp="Info"



failure:
609 - [meta sequenceId="3419"] JobInf : LogTyp="Job", No= 17006 , EntTim=2022-11-22T11:20:02, FinTim=2022-11-22T11:20:02, Nam="test", Typ="Print", RelJobNo=4294967295, SubScenarioID=0, Code="UserNon", Nam="test", NetTyp="IntraNetwork", TrcCode=Non, IF="PrinterReception", Res="LogInError", EventTyp="Error"

[Synthohol]

was there a page limit on the account and is it only 1 account affected?

[rbillyg]

was there a page limit on the account and is it only 1 account affected?


No page limits and its every domain user randomly having the issue. AD accounts are not being locked and we are using SSO.

[emujo2]

This has always been a tough environment to setup, and is better left to 3rd party solutions to handle logins and follow print/secure print ect, but saying that...

1. Single Sign on requires External server authentication and SSO checked
2. print driver must reflect SSO option enabled when getting a new config and should not even be allowed unless the machine picks up the authentication settings.
3. SSO check box in authentication settings checked and correct username reflecting in the print box.

OK,,So all this seems to be setup correctly and works, just intermittent failures..

1. Does the machine show any indication that it has failed to connect to AD
2. Does your time/date match +/- a few seconds, or you have a NTP server in use
3. Do you allow bring your device to work and allow mobile connections to the LAN? Does AD show any user that has deleted print jobs getting a account lock out at or around the same time? I ask because mobile devices with an out of date password will lock AD accounts as they try to log on to the wireless..
4. Have you tested the NIC? Any timeouts when pinging, is the NIC configured to best match the switch it's connected to? E

[rbillyg]

This has always been a tough environment to setup, and is better left to 3rd party solutions to handle logins and follow print/secure print ect, but saying that...

1. Single Sign on requires External server authentication and SSO checked
2. print driver must reflect SSO option enabled when getting a new config and should not even be allowed unless the machine picks up the authentication settings.
3. SSO check box in authentication settings checked and correct username reflecting in the print box.

OK,,So all this seems to be setup correctly and works, just intermittent failures.. E

1. Does the machine show any indication that it has failed to connect to AD - None
2. Does your time/date match +/- a few seconds, or you have a NTP server in use - We have a ntp server configured. I have checked this setting multiple times as well thinking it could be a time issue. especially as we just had daylight savings early November when this started happening.
3. Do you allow bring your device to work and allow mobile connections to the LAN? Does AD show any user that has deleted print jobs getting a account lock out at or around the same time? I ask because mobile devices with an out of date password will lock AD accounts as they try to log on to the wireless.. I have checked their credentials for lockouts with to be sure that is not the case as I thought similar.
4. Have you tested the NIC? Any timeouts when pinging, is the NIC configured to best match the switch it's connected to? We use PRTG for monitoring and shows a 2% packet loss for 5 minutes over the past 48 hours..

Thanks for the suggestions.

[zaicnupagadi]

Hi,

We got exactly the same issue, (happened after the November Microsoft CU patches), on all of our printers Bizhub c250i.

It all works without SSO - so when users put their credentials in the printer preferences.

But as soon as people check the "Enable SSO" in printer preferences and try to print it ends with errors.

We have opened a case at MS but no luck so far. Does anyone got any clue?

Cheers,
Pawel

[rbillyg]

Hi,

We got exactly the same issue, (happened after the November Microsoft CU patches), on all of our printers Bizhub c250i.

It all works without SSO - so when users put their credentials in the printer preferences.

But as soon as people check the "Enable SSO" in printer preferences and try to print it ends with errors.

We have opened a case at MS but no luck so far. Does anyone got any clue?

Cheers,
Pawel


You know, I thought it might be the updates provisioned as well. I removed it from our DC as well as applied the OOB patch and still having the issue.

[zaicnupagadi]

You know, I thought it might be the updates provisioned as well. I removed it from our DC as well as applied the OOB patch and still having the issue.

Hi!

Really thank you for reply! I've finally fround someone with the same issue that we have - Internet is completely silent about it.

We did the same - tried to uninstall it, later installed once again and installed the OOB as well. Applied the registry entries (fixes) manually - still no luck.

We managed to establish that this is the server fault as we had a W10 machine without November patches - still the same issue. Printing with Login/Password works, but not with SSO checkbox marked.

I have opened a case at MS, they havesent following plan:

##################

Thank you for your response, based on it we will need you to take the following traces when you attempt to reproduce the issue:




Download from the following link the file containing the tracing scripts https://aka.ms/authscripts


NOTE: The authentication scripts should be executed from a Windows PowerShell (Admin) session.



To start tracing execute the start-auth.ps1 script




Attempt to reproduce the issue by attempting to print, when it fails take not of the exact time.




To stop tracing execute the stop-auth.ps1 script


The data will be collected in a subdirectory, from where the scripts are executed, called “Authlogs”

When executing the Start-auth.ps1 command, if it is the first time the user has executed the script on this device, the user will be asked to accept a EULA. If the user accepts the EULA, the scripts will execute on the device and the following registry key will be created.
HKEY_CURRENT_USER\SOFTWARE\Microsoft\CESDiagnostic Tools

Subsequent usage for this user on the same device, will not require the EULA to be displayed or accepted.

#####################

I've reproduced the issue on our print server, send them the output folder and waiting for the info them their side.


Cheers,

Pawel Jarosz

[zaicnupagadi]

Ehh.. got reply that logs need to be collected from the domain controller:



Good day Pawel,

Thank you for your response, my apologies but the traces had to be taken from the Domain Controller for the Domain precisely to verify the authentication process that may be failing when attempting to print, we will need you to take the trace from the DC. Besides this we will also need you to provide us with the Security log from that same DC. Meanwhile we will begin checking the traces provided from the printer server.

[rbillyg]

Ehh.. got reply that logs need to be collected from the domain controller:

Thanks for the info. Still stabbing away at it ourselves.

[zaicnupagadi]

Thanks for the info. Still stabbing away at it ourselves.

Microsoft, after 1 month (as the first email about it was on 21.11.2022), sent me agreement for the scope for the case, claiming there that the user "admnistrator" is only affected. EEhh..

Anyway, we checked the wireshark one more time, our finding are that when sending a print job with SSO there are some retransmission and reset package with flags accordingly:

Flags: 0x011 (FIN, ACK)
and later
Flags: 0x014 (RST, ACK)

Those packages are not present when trying to print leveraging just login/pasword, but appear when checking the SSO checkbox and trying to print then.

Also, when checking the kerberos tokens with the "klist" command, we noticed that the "KerbTicket Encryption Type:" only for this particular ticket (printer) is in type "RSADSI RC4-HMAC(NT)"


#1> Client: user @ DOMAIN.COM
Server: PRINTER/printer-prn5-36 @ DOMAIN.COM
KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)
Ticket Flags 0xa10000 -> renewable pre_authent name_canonicalize
Start Time: 12/21/2022 14:29:51 (local)
End Time: 12/22/2022 0:29:51 (local)
Renew Time: 12/28/2022 14:29:51 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0
Kdc Called: Our_DC_Server.domain.com

Called my other colleague who also uses Konica Minoltas printers with SSO at his workplace (but they haven't installed the November patches on DCs) and in his klist "RSADSI RC4-HMAC(NT)" is not present.

We think that it might be the case, and that it is somehow connected to the new ticket type, which printers cannot handle.

@Billy - could you please check how it looks at your end?

Cheers,
Pawel

[rbillyg]

Microsoft, after 1 month (as the first email about it was on 21.11.2022), sent me agreement for the scope for the case, claiming there that the user "admnistrator" is only affected. EEhh..

Anyway, we checked the wireshark one more time, our finding are that when sending a print job with SSO there are some retransmission and reset package with flags accordingly:

Flags: 0x011 (FIN, ACK)
and later
Flags: 0x014 (RST, ACK)

Those packages are not present when trying to print leveraging just login/pasword, but appear when checking the SSO checkbox and trying to print then.

Also, when checking the kerberos tokens with the "klist" command, we noticed that the "KerbTicket Encryption Type:" only for this particular ticket (printer) is in type "RSADSI RC4-HMAC(NT)"



Called my other colleague who also uses Konica Minoltas printers with SSO at his workplace (but they haven't installed the November patches on DCs) and in his klist "RSADSI RC4-HMAC(NT)" is not present.

We think that it might be the case, and that it is somehow connected to the new ticket type, which printers cannot handle.

@Billy - could you please check how it looks at your end?

Cheers,
Pawel

I'll get a look at that. thanks for the recommendation.

[rbillyg]

I'll get a look at that. thanks for the recommendation.



same results as you buddy: KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)

[rbillyg]

SUCCESS!!!

with member : zaicnupagadi

So when the November update came out it changed defaulting to AES session keys for encryption. so the RC4 encryption type was getting converted to AES which the printer was not supporting. Thanks to zaicnupagadi and all his information I we were able to find some other articles in regards to netapp where users were having similar issues. I found that we could change the printer attribute msDS-SupportedEncryptionTypesin AD to 0x4 which forced it to stay as an RC4 session key. This worked for us, and then I messaged zaicnupagadi who made the change and also confirmed it worked.
Teamwork across the globe!

[zaicnupagadi]

SUCCESS!!!

with member : zaicnupagadi

So when the November update came out it changed defaulting to AES session keys for encryption. so the RC4 encryption type was getting converted to AES which the printer was not supporting. Thanks to zaicnupagadi and all his information I we were able to find some other articles in regards to netapp where users were having similar issues. I found that we could change the printer attribute msDS-SupportedEncryptionTypes in AD to 0x4 which forced it to stay as an RC4 session key. This worked for us, and then I messaged zaicnupagadi who made the change and also confirmed it worked.
Teamwork across the globe!

Haha!! Actually my colleague Konrad (from the team) called me yesterday and he told me he has some hunch the problem is in the encryption, we were wiresharking around, found some retransmission packages etc, called my other colleague who has NOT applied the november patches and he did some "klist" for me, telling that he has no RC4 enryption for tickets on his side. So I told that everything to Billy, after reading yesterday's messages, Billy told me he also had some hunch it was about encryption aaaaaand did the whole magic :)

So summing up - intercontinental cooperation between America and Poland went well! We were able to solved the case :D for which Konica didn't have a clue what is going on, and Microsoft was working for a month without any progress.

So now our TGS ticket is:


#1> Client: user @ DOMAIN.COM
Server: PRINTER/printer-prn5-36 @ DOMAIN.COM
KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)
Ticket Flags 0xa10000 -> renewable pre_authent name_canonicalize
Start Time: 12/22/2022 8:28:41 (local)
End Time: 12/22/2022 18:28:41 (local)
Renew Time: 12/29/2022 8:28:41 (local)
Session Key Type: RSADSI RC4-HMAC(NT)
Cache Flags: 0
Kdc Called: DC_Server.domain.com

So the "Session Key Type" is now "RSADSI RC4-HMAC(NT)"

Today me and my colleague are the celebrities at the office :D

Thank you Billy!!!

Hopefully our cooperative investigation might help someone else as well,
Pawel