Apple Developer Support examined this issue and came to the conclusion that a particular .p12 file has DER encoding not with iOS 11 security requirements.
Below is the Apple DTS Quinn email:
To understand what is going on, you need to set a breakpoint on the SecKeyCreateRSAPrivateKey function. This is a private function within the security framework, so you will need to use the symbolic (in the Breakpoints navigator, click the + button and select Symbolic Breakpoint from the menu. Once you do this, click the "Test 1" button and you will stop at the breakpoint.
This function takes 4 arguments, but the two relevant ones are the second and third arguments, which are a pointer to the RSA private key
for decoding and the length of this data. Once you hit the breakpoint, you can reset the data as follows:
(lldb)
The result is too long to include here, so I turned off the hexadecimal encoding and attached the result as a file, RSAPrivateKey.asn1 .
If you reset this with dumpasn1 , you will see a problem:
$ dumpasn1 -p -a RSAPrivateKey.asn1 SEQUENCE { INTEGER 0 INTEGER … INTEGER 65537 INTEGER 00 0B 67 67 29 0A 38 38 7A 7E 17 39 E7 84 FB 7D 02 D1 AB AD 21 27 4D CD 2C 09 C1 1F CB 73 5C 18 37 D4 CE E2 98 61 19 3B 70 6C 1A 1B 33 E8 69 8C 65 5B 77 B8 24 9C 90 8A 79 A7 4A 77 26 38 7C 3E 70 3C 80 24 BD 73 DA 97 5A F3 90 0F 79 2D 45 F8 2A 5A 37 03 4D 0C 80 DB 8F 99 67 55 D3 F3 70 CA 2D F0 B6 48 6A D8 9A 95 CE AA C8 F1 96 24 04 61 38 A1 7C CD 56 70 5E F9 13 D4 1B E2 F2 5D 28 67 9A 30 E0 ED 71 84 4C EC 86 44 18 5F 64 70 46 D8 6B 3A 52 2C D7 DE AF E5 E2 35 41 0D 1E FF E7 DE AC 43 83 5C A6 F2 2E C8 79 1A 30 87 7A 78 9B 42 3E D9 2D AA C0 9D A6 7C E9 5C E3 6C 1E 8D 87 DF EB 05 DF CA F5 B9 2D BA B6 01 71 18 22 4D 25 4E D5 77 CB B8 9B 95 F9 C6 39 1C 0D D2 46 E4 4A 45 D8 26 6F B4 25 03 E7 BE 91 02 43 7D DC B0 1E C8 67 E8 2E 5F EA 3A 8D 1C 69 43 80 F9 60 69 BF BA 01 Error: Integer has non-DER encoding. INTEGER … INTEGER … INTEGER … INTEGER … INTEGER … }
Note that the "Integer has incorrect DER encoding" error, indicated by the icon in the fourth INTEGER element. This is because INTEGER data is not canonical. In ASN.1 DER, INTEGERs are signed in which ambiguity. A value of type 80 00 can be interpreted as -32768 or 32768. To resolve this, ASN.1 DER requires that the positive form be encoded as 00 80 00 (this is a similar case, requiring a leading one further). In addition, DER specifically requires that this leading byte be added only when necessary. In the case shown above, the second byte, 0b, does not have the upper bit set and therefore should not have this leading 00.
Earlier versions of iOS did not apply this requirement. iOS 11 includes a number of changes to simplify security and reject the incorrectly encoded ASN.1 DER INTEGER - this is one of these changes.
I saw that other developers bit this security problem, but these cases were easier to debug, because the keys werent built in under PKCS No. 12. For example, consider the following DevForums thread. https://forums.developer.apple.com/message/262593#262593 In addition, this problem was difficult to debug, because when you unpack .p12 using openssl , it ignores this error on input and emits on output fixed version of the private key!
Quin recommended I solve this problem by setting which code generates PKCS # 12 data. If this is not possible, you will need to write (or purchase) your own code to analyze PKCS # 12. iOS 11 correctly rejects the erroneously formatted private key, and this is unlikely whether to change.