Intune Hybrid - NDES Cert Issue
As many of you who might be running a SCCM/Intune hybrid scenario for MDM will have learned. There are a few to many certificates in use between NDES, Intune, Wifi Profiles, VPN profiles, and well anything else that may have a cert buried in it.
We had an instance were the NDES wasn't handing out the certs for WiFi and VPN profiles. However it was handing out the base communication certificate so phones could sync correctly. This was one of those head scratchers on our park, but if you know where to look it makes sense. Problem is (at least in my opinion) there is a lack of useful documentation and the logging leaves a little to be desired.
Therefore, I'm going to give a bit about our journey and hopefully help some of you folks out if you run into something similar.
When we were notified of this issue, we started to look at the usual logs, you know CRP.log NDESPlugin.log those types of things. First thing you will notice is that the CRP.log will show the same message over and over. This message "Perf Test Hook. Challenge is read from registry." is pretty useless, however it does tell you that no requests have been submitted. You can also confirm this by trying to find the IP address of your NDES server in the CRP IIS logs (Assuming your NDES and CRP are running on different servers).
The slightly more useful information is all on the NDES box and services. The first clue is that you don't see any "Verify challenge returns true" entries in the NDESPlugin.log file. In the IIS logs, when a request comes in you see an entry with an HTTP error code of 403.17 which translates (to those of us who don't have this memorized) as "Client Certificate has expired or is not yet valid". The other thing you may notice is that in the Application Event Log you see multiple entries of "The password in the certificate request cannot be verified. It may have been used already. Obtain a new password to submit with this request."
All this definitely tells you that you have a certificate problem. Now if your lucky (we weren't) when you go look at the certificate store on your server (Certificate (Local Computer)\Personal\Certificates) you'll see a certificate that is expired and hence make you go "D'oh". However if you aren't lucky (believe me we wish it was as easy as checking the certificate store cause that would have saved us a ton of time) you have to dig deeper cause your certificate store says "I'm A-OKAY", okay so it doesn't really say that but you get the idea.
Checking the certificate tied to the client auth for the NDES plugin. Navigate your trusty hands to the registry editor and find - HKLM\Software\Microsoft\Cryptography\MSCEP\Modules\NDESPolicy - then there will be a value in there called "NDESCertThumbnail" this is the thumbnail of the certificate NDES is using for communication. We checked all our certs in the certificate store and didn't fine thumbnail match. So we pulled out a commandline utility that would export the certs on the box to a file.
From an elevated command prompt run the following "Certutil -v -store MY >> %computername%_MY.txt" this will put All Certificates into a txt file. We where able to find the certificate with the thumbnail in question in the file and when we went to see what certificate that was we got a nice little surprise the file showed and entry of "Archived!".
Now the lightbulb came on in our heads and went "Well Crap, There's your problem". Now you are probably wondering you can just change the thumbnail in the registry where you found it. Guess What! That .. wait for it.... won't work. Yeah, we were pretty disappointed as well. So to correct this you have to run the NDESPlugin setup. Which means you have to remove the plugin and then install the plugin using the new client auth cert and the root cert (which if you have misplaced your root cert, you can find it on your SCCM server in the cert manager inbox).
Now that NDES has been setup and has the correct cert, everything should be fat and happy. It was for us anyways. While you are at it, once you have fixed it, set yourself up a calendar reminder for around the time the certificate you are using is set to expire with these notes. That way if it happens again you will instantly know were to look first.
Hope you guys find this post enlightening and helpful.
We had an instance were the NDES wasn't handing out the certs for WiFi and VPN profiles. However it was handing out the base communication certificate so phones could sync correctly. This was one of those head scratchers on our park, but if you know where to look it makes sense. Problem is (at least in my opinion) there is a lack of useful documentation and the logging leaves a little to be desired.
Therefore, I'm going to give a bit about our journey and hopefully help some of you folks out if you run into something similar.
When we were notified of this issue, we started to look at the usual logs, you know CRP.log NDESPlugin.log those types of things. First thing you will notice is that the CRP.log will show the same message over and over. This message "Perf Test Hook. Challenge is read from registry." is pretty useless, however it does tell you that no requests have been submitted. You can also confirm this by trying to find the IP address of your NDES server in the CRP IIS logs (Assuming your NDES and CRP are running on different servers).
The slightly more useful information is all on the NDES box and services. The first clue is that you don't see any "Verify challenge returns true" entries in the NDESPlugin.log file. In the IIS logs, when a request comes in you see an entry with an HTTP error code of 403.17 which translates (to those of us who don't have this memorized) as "Client Certificate has expired or is not yet valid". The other thing you may notice is that in the Application Event Log you see multiple entries of "The password in the certificate request cannot be verified. It may have been used already. Obtain a new password to submit with this request."
All this definitely tells you that you have a certificate problem. Now if your lucky (we weren't) when you go look at the certificate store on your server (Certificate (Local Computer)\Personal\Certificates) you'll see a certificate that is expired and hence make you go "D'oh". However if you aren't lucky (believe me we wish it was as easy as checking the certificate store cause that would have saved us a ton of time) you have to dig deeper cause your certificate store says "I'm A-OKAY", okay so it doesn't really say that but you get the idea.
Checking the certificate tied to the client auth for the NDES plugin. Navigate your trusty hands to the registry editor and find - HKLM\Software\Microsoft\Cryptography\MSCEP\Modules\NDESPolicy - then there will be a value in there called "NDESCertThumbnail" this is the thumbnail of the certificate NDES is using for communication. We checked all our certs in the certificate store and didn't fine thumbnail match. So we pulled out a commandline utility that would export the certs on the box to a file.
From an elevated command prompt run the following "Certutil -v -store MY >> %computername%_MY.txt" this will put All Certificates into a txt file. We where able to find the certificate with the thumbnail in question in the file and when we went to see what certificate that was we got a nice little surprise the file showed and entry of "Archived!".
Now the lightbulb came on in our heads and went "Well Crap, There's your problem". Now you are probably wondering you can just change the thumbnail in the registry where you found it. Guess What! That .. wait for it.... won't work. Yeah, we were pretty disappointed as well. So to correct this you have to run the NDESPlugin setup. Which means you have to remove the plugin and then install the plugin using the new client auth cert and the root cert (which if you have misplaced your root cert, you can find it on your SCCM server in the cert manager inbox).
Now that NDES has been setup and has the correct cert, everything should be fat and happy. It was for us anyways. While you are at it, once you have fixed it, set yourself up a calendar reminder for around the time the certificate you are using is set to expire with these notes. That way if it happens again you will instantly know were to look first.
Hope you guys find this post enlightening and helpful.
Yes, yes,yes.... This (and the fact that my RAS server Cert had expired) was my problem as well.
ReplyDeleteThank you!
You're very welcome. I'm glad it helped you resolve your issue.
DeleteI ran into this exact issue with our Intune AAD join NDES configuration. I am happy to say that copying the thumbprint out of the txt file and into the registry key DID work for me. It took about 2-3 minutes, but the test machine that was stuck running Autopilot, suddenly continued after it received it's certificate.
ReplyDeleteThank you for this post, I owe you a beer!
You're very welcome. Always good to see that my posts (even as sporadic as they are) help those that have similar headaches. :-)
Delete