The Case of the Malformed Chain

Investigating an mTLS, Safari and Certificate Chain Issue on iOS and macOS

Sometimes, a bug is obvious. Sometimes, it takes years to figure out the cause. In the last week, I finally discovered the root cause of an issue that has frustrated us and our customers for years. The discovery is the interesting part of the story, as I was not even looking into the issue when I found the thread to pull on that led me to the solution.

What Happened

After folks purchased our Smart Card Utility for iPhone and iPad software and readers, we occasionally got reports of smart cards working with some websites, but not others. The same smart card and certificates would work on Windows to the same websites but didn’t work when attempting to access it using iOS or macOS. Sometimes, we could get it to work and, randomly, sometimes we could not. We never had a smart card or identity in-house that could replicate the problem consistently so we did a lot of remote troubleshooting with limited progress. It was all very random and not very repeatable.

A Brief Overview of mTLS

When authenticating using a smart card with a web server, the user is prompted to select a certificate, the authentication happens, and the user has access to the website (yes, this is greatly simplified, but stay with me because the bug doesn’t really have much to do with authentication).

When a browser starts a connection with a website, the website provides a web service certificate. The client needs to validate that this certificate is valid and trusted, so the server may provide a group of certificates, called a certificate chain, to make it easier for the client to validate the server certificate. 

During mTLS, when the client sends the client certificate to the web server, the server verifies the client certificate. The client can send a certificate chain to help the server validate the client certificate. The client certificate chain is not required. It does not have to be related to the server certificate chain to trust the server certificate. Trusting the client certificate chain on the client is not required and should not affect the authentication (spoiler: it did).

Think about it this way. You have just started a job at a local medical clinic owned by the local hospital, which is owned by a large hospital chain. You need scrubs, so you head off to the approved uniform store to get your new green scrubs. You are only allowed to buy from approved uniform stores, and the clinic said that any uniform stores owned by ABC Uniform are acceptable. On entering the uniform store, you look at their license on the wall. It says they are owned by a local business chain but you don’t know if the local chain is owned by the approved uniform store parent company (ABC Uniform). So you call the local uniform chain and ask who owns them. They confirm that they are owned by the ABC Uniform. You have verified that this is an approved store so you go in and find your green scrubs. You have just verified the server certificate chain.

When checking out, the clerk says that only employees of the large hospital chain can buy scrubs so you show your badge. Your badge says you are employed by the local clinic owned by the local hospital, so the clerk calls the local hospital and asks who owns them. They verify that the local hospital is owned by the large hospital chain and you are now allowed to buy the scrubs. They have now verified the client certificate chain.

You needed to verify it was an approved store (the server chain), and the clerk needed to verify that you worked at an approved clinic (client certificate chain). You didn’t need to validate your own documents, you just needed to be accepted by the clerk. How the clerk validates them is up to the clerk, and while you can provide supporting documents, the clerk must verify them regardless of where they came from.

A Breakthrough

None of it made any sense. It was like the clerk went into a back room and sometimes came back right away and denied, sometimes came back after a while and approved, and sometimes came back after a while and denied.

At the same time, I was working to implement ECA smart cards on iPhone, iPad, and Mac. The card was working great until I tried to use it in Safari against our test web server. The test web server log showed the error “tls: failed to parse client certificate: x509: trailing data”. 

I got very excited. It was like I finally saw what the clerk was concerned about. I now had a window into the back room. This was the same error that our customers had been seeing sporadically, but I now had a card that appeared to replicate the issue. 

It was like I was in front of the clerk, and I could present different documentation to the clerk and see what caused the clerk’s concern.

The Solution

I could now correlate the logs, the app in a debugger, and a network trace. The network trace showed that the client was sending the client certificate chain back, but one of the certificates was 3 times the size the other certificates:

Certificates (17837 bytes)
	Certificate Length: 2061
	Certificate Length: 1174
	Certificate Length: 1421
	Certificate Length: 1799
	Certificate Length: 9774
	Certificate Length: 1590
	

Wireshark showed that the large certificate was a correct certificate but with a bunch of extra data after it but before the next certificate. I also found that if I imported the correct certificate chain into macOS and trusted the root certificate, the problem disappeared. This didn’t make much sense since Safari doesn’t need to trust the client certificate. It was like I was looking at my own documentation allowing me to buy scrubs before presenting it to the clerk and deciding I didn’t think it was sufficient.

So I looked further.

Issuing Authority

Every certificate is signed by another certificate, including the top level, called a root certificate and is signed with its own certificate. To figure out which certificate signed a certificate, each certificate can have an attribute with a URL to the certificate that signed it. So starting with the certificate on a smart card, you can follow the URL to the issuing certificate, look at that certificate for the issuing authority and follow that URL to the issuer. This can be done up the chain until the top level, or root certificate is discovered.

So I did exactly that. I found that the data for the certificate with a large amount of extra data discovered in Wireshark was the same as the data found at one of the issuer URLs. Looking at the format, it was not an X.509 certificate but rather a PKCS#7 or P7C, which is a single certificate or a collection of certificates.

When a client certificate is sent to the server, Safari collects the certificate chain and includes each certificate in a list to the server. The server is expecting a list of certificates but instead gets a list of certificates and an unexpected p7c in the list. The server sees malformed data and drops the connection. At least my server did. Other web servers may accept it, build the chain itself, or error out differently.

When the clerk checked you out, they had to call the local hospital. Instead of doing that, you could have given them a letter from the local hospital showing that they were owned by the hospital chain. This would save a call and make your checkout faster. However, if you gave them the letter but it had some extra information at the bottom of the letter that the clerk didn’t understand, the clerk could either:

1. Accept the letter since it looked fine and ignore the extra info
2. Reject the letter and call the local hospital
3. Reject the transaction and send you on your way.

So the solution was either to get Safari to not send the client certificate chain, or get the certificate chain from somewhere else. On macOS, if the certificate chain is in the Keychain, Safari will not follow the issuer URL and will use the certificates from the Keychain to build the client certificate chain. This fixes the problem on macOS, and provides some level of irony as well: trusting the client certificate chain is not needed for the client but resolves the issue because bad data is not sent to the server. 

Unfortunately, on iOS, this fix doesn’t seem to work. To test whether this was the same issue we were seeing with other customers, I sent them a project called Smart Card Utility Browser. Smart Card Utility Browser is a simple browser based on WebKit that logs TLS and mTLS connections. It also allows you to turn off sending the client certificate chain. I uploaded the app to TestFlight for them to test, and it allowed the connection to our test server without issue.

The issue was that Safari was giving extra information then webserver didn’t understand, and some webservers chose option 3, reject the transaction.

Apple Feedback

The core of this issue is probably in URLSession or Safari, so I needed to report it to Apple. I filed Feedback FB16208348 and escalated the issue with WWDR to see if there was a way to stop Safari from sending the client certificate chain. This doesn’t just affect our software and our readers but any smart card reader on iOS or macOS. It only happens for some certificate chains, and only if the chain is not in the Keychain.

Going Wrong in Interesting Ways

I love the fact that the solution didn’t come from investigating the problem, but recognizing when something goes wrong. This process reminded me a bit of the story of how penicillin was discovered. I remember a professor telling me that discoveries come not when things go right but go wrong in interesting ways. 

Client certificate chains are not antibiotics, but I was glad to discover the cause of the TLS failure. I knew why the clerk was approving to sell scrubs to some customers and denying others, and I could work on getting it fixed.