How to use ssl client certificate (p12) using Scrapy?

Question

I need to use the client certificate file in p12 format (PKCS12) to talk to the web server using scrapy, is there any way to do this?

Answer 1

I cannot offer you a proven and complete solution here, but I know several places where some settings can give you what you need.

The starting point is the scrapy object ContextFactory, which defines the SSL / TLS configuration. The standard implementation ScrapyClientContextFactorydoes not use client certificates, and also does not perform server certificate verification, it simply accepts any certificate. ( More )

If you look in the source code , you will see that the alternative BrowserLikeContextFactorycreates an optionsForClientTLS object.

This object can also accept a parameter clientCertificatefor server authentication. ( Details )

So, theoretically, you need a subclass BrowserLikeContextFactory, write your own method there creatorForNetlocand create one optionsForClientTLSthat also hasclientCertificate

In fact:

@implementer(IPolicyForHTTPS)
class ClientCertContextFactory(BrowserLikeContextFactory):

    def creatorForNetloc(self, hostname, port):
        with open('yourcert.pem') as keyAndCert:
            myClientCert = twisted.internet.ssl.PrivateCertificate.load(keyAndCert.read())
        return optionsForClientTLS(hostname.decode("ascii"),
                                   trustRoot=platformTrust(),
                                   clientCertificate=myClientCert,
                                   extraCertificateOptions={
                                        'method': self._ssl_method,
                                   })

Factory context activation in settings.py:

DOWNLOADER_CLIENTCONTEXTFACTORY = 'your.package.ClientCertContextFactory'

twisted.internet.ssl.PrivateCertificate pem asn.1, si pem:

openssl pkcs12 -in client_ssl.pfx -out client_ssl.pem -clcerts

( pfx pem openssl)

. PKCS12 p12:

openssl pkcs12 -in client_cert.p12 -out client_cert.pem -clcerts