How can I get LWP to verify SSL server certificates?

Question

How can I get LWP to make sure that the certificate of the server I'm connecting to is signed by a trusted authority and issued to the correct host? As far as I can tell, it does not even verify that the certificate claims that I am connecting for the host name. This seems like a serious security issue (especially with recent DNS vulnerabilities).

Update: It turns out that what I really wanted was HTTPS_CA_DIR because I do not have ca-bundle.crt. But HTTPS_CA_DIR=/usr/share/ca-certificates/ did the trick. I still note the answer because it was close enough.

Update 2: It turns out that HTTPS_CA_DIR and HTTPS_CA_FILE only apply if you use Net :: SSL as the base SSL library. But LWP also works with IO :: Socket :: SSL, which ignores these environment variables and happily talks to any server, no matter what certificate it presents. Is there a more general solution?

Update 3: Unfortunately, the solution is still incomplete. Neither Net :: SSL nor IO :: Socket :: SSL checks the hostname for a certificate. This means that someone can obtain a legal certificate for a domain, and then impersonate any other domain without LWP complaints.

Update 4: LWP 6.00 finally fixes the problem. See my answer for more details.

Answer 1

This long-term security hole has finally been fixed in version 6.00 of libwww-perl . Starting with this version, by default, LWP :: UserAgent verifies that HTTPS servers present a valid certificate that matches the expected host name (unless $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} is false or, for backward compatibility, if this variable is not set at all , set either $ENV{HTTPS_CA_FILE} or $ENV{HTTPS_CA_DIR} ).

This can be controlled with the new ssl_opts option for LWP :: UserAgent. See this link for details on how certification authority certificates are located. But be careful how LWP :: UserAgent works, if you provided the ssl_opts hash for the constructor, then verify_hostname defaults to 0 instead of 1. ( This error was fixed in LWP 6.03.) To be safe, always specify verify_hostname => 1 in ssl_opts .

So use LWP::UserAgent 6; should be sufficient to verify server certificates.

Answer 2

There are two ways to do this, depending on which SSL module you have installed. LWP docs recommends installing Crypt :: SSLeay . If this is what you did by setting the environment variable HTTPS_CA_FILE to point to your ca-bundle.crt, you need to do the trick. ( Crypt :: SSLeay docs mentions this, but highlights the details a bit). Also, depending on your installation, you may need to set the HTTPS_CA_DIR environment variable HTTPS_CA_DIR .

Example for Crypt :: SSLeay:

 use LWP::Simple qw(get); $ENV{HTTPS_CA_FILE} = "/path/to/your/ca/file/ca-bundle"; $ENV{HTTPS_DEBUG} = 1; print get("https://some-server-with-bad-certificate.com"); __END__ SSL_connect:before/connect initialization SSL_connect:SSLv2/v3 write client hello A SSL_connect:SSLv3 read server hello A SSL3 alert write:fatal:unknown CA SSL_connect:error in SSLv3 read server certificate B SSL_connect:error in SSLv3 read server certificate B SSL_connect:before/connect initialization SSL_connect:SSLv3 write client hello A SSL_connect:SSLv3 read server hello A SSL3 alert write:fatal:bad certificate SSL_connect:error in SSLv3 read server certificate B SSL_connect:before/connect initialization SSL_connect:SSLv2 write client hello A SSL_connect:error in SSLv2 read server hello B
use LWP::Simple qw(get); $ENV{HTTPS_CA_FILE} = "/path/to/your/ca/file/ca-bundle"; $ENV{HTTPS_DEBUG} = 1; print get("https://some-server-with-bad-certificate.com"); __END__ SSL_connect:before/connect initialization SSL_connect:SSLv2/v3 write client hello A SSL_connect:SSLv3 read server hello A SSL3 alert write:fatal:unknown CA SSL_connect:error in SSLv3 read server certificate B SSL_connect:error in SSLv3 read server certificate B SSL_connect:before/connect initialization SSL_connect:SSLv3 write client hello A SSL_connect:SSLv3 read server hello A SSL3 alert write:fatal:bad certificate SSL_connect:error in SSLv3 read server certificate B SSL_connect:before/connect initialization SSL_connect:SSLv2 write client hello A SSL_connect:error in SSLv2 read server hello B 

Note that get does not matter die , but returns undef .

Alternatively, you can use the IO::Socket::SSL module (also available from CPAN). In order for this to confirm the server certificate, you need to change the SSL context settings:

 use IO::Socket::SSL qw(debug3); use Net::SSLeay; BEGIN { IO::Socket::SSL::set_ctx_defaults( verify_mode => Net::SSLeay->VERIFY_PEER(), ca_file => "/path/to/ca-bundle.crt", # ca_path => "/alternate/path/to/cert/authority/directory" ); } use LWP::Simple qw(get); warn get("https:://some-server-with-bad-certificate.com");
use IO::Socket::SSL qw(debug3); use Net::SSLeay; BEGIN { IO::Socket::SSL::set_ctx_defaults( verify_mode => Net::SSLeay->VERIFY_PEER(), ca_file => "/path/to/ca-bundle.crt", # ca_path => "/alternate/path/to/cert/authority/directory" ); } use LWP::Simple qw(get); warn get("https:://some-server-with-bad-certificate.com"); 

This version also causes get() return undef, but it displays a warning on STDERR when it is executed (as well as a bunch of debugging if you import debugging symbols * from IO :: Socket :: SSL):

 % perl ssl_test.pl DEBUG: .../IO/Socket/SSL.pm:1387: new ctx 139403496 DEBUG: .../IO/Socket/SSL.pm:269: socket not yet connected DEBUG: .../IO/Socket/SSL.pm:271: socket connected DEBUG: .../IO/Socket/SSL.pm:284: ssl handshake not started DEBUG: .../IO/Socket/SSL.pm:327: Net::SSLeay::connect -> -1 DEBUG: .../IO/Socket/SSL.pm:1135: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed DEBUG: .../IO/Socket/SSL.pm:333: fatal SSL error: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed DEBUG: .../IO/Socket/SSL.pm:1422: free ctx 139403496 open=139403496 DEBUG: .../IO/Socket/SSL.pm:1425: OK free ctx 139403496 DEBUG: .../IO/Socket/SSL.pm:1135: IO::Socket::INET configuration failederror:00000000:lib(0):func(0):reason(0) 500 Can't connect to some-server-with-bad-certificate.com:443 (SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed)
% perl ssl_test.pl DEBUG: .../IO/Socket/SSL.pm:1387: new ctx 139403496 DEBUG: .../IO/Socket/SSL.pm:269: socket not yet connected DEBUG: .../IO/Socket/SSL.pm:271: socket connected DEBUG: .../IO/Socket/SSL.pm:284: ssl handshake not started DEBUG: .../IO/Socket/SSL.pm:327: Net::SSLeay::connect -> -1 DEBUG: .../IO/Socket/SSL.pm:1135: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed DEBUG: .../IO/Socket/SSL.pm:333: fatal SSL error: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed DEBUG: .../IO/Socket/SSL.pm:1422: free ctx 139403496 open=139403496 DEBUG: .../IO/Socket/SSL.pm:1425: OK free ctx 139403496 DEBUG: .../IO/Socket/SSL.pm:1135: IO::Socket::INET configuration failederror:00000000:lib(0):func(0):reason(0) 500 Can't connect to some-server-with-bad-certificate.com:443 (SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed) 

Answer 3

I landed on this page, looking for a way around the SSL check, but all the answers were still very helpful. Here are my findings. For those who want to bypass the SSL check (not recommended, but there may be times when you definitely need it), I am on lwp 6.05, and this worked for me:

 use strict; use warnings; use LWP::UserAgent; use HTTP::Request::Common qw(GET); use Net::SSL; my $ua = LWP::UserAgent->new( ssl_opts => { verify_hostname => 0 }, ); my $req = GET 'https://github.com'; my $res = $ua->request($req); if ($res->is_success) { print $res->content; } else { print $res->status_line . "\n"; } 

I also tested on the page with POST, and it also worked. The key is to use Net :: SSL along with verify_hostname = 0.

Answer 4

If you use LWP :: UserAgent directly (not through LWP :: Simple), you can verify the host name in the certificate by adding the "If-SSL-Cert-Subject" header to your HTTP :: Request object. The header value is considered a regular expression that should be applied to the subject of the certificate, and if it does not match, the request fails. For example:

 #!/usr/bin/perl use LWP::UserAgent; my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new(GET => 'https://yourdomain.tld/whatever'); $req->header('If-SSL-Cert-Subject' => '/CN=make-it-fail.tld'); my $res = $ua->request( $req ); print "Status: " . $res->status_line . "\n" 

will print

 Status: 500 Bad SSL certificate subject: '/C=CA/ST=Ontario/L=Ottawa/O=Your Org/CN=yourdomain.tld' !~ //CN=make-it-fail.tld/ 

Answer 5

All of the solutions presented here contain a serious security flaw, as they verify the validity of the certificate trust chain, but do not compare the common name of the certificate with the host name with which you are connecting. This way, the person in the middle can provide you with an arbitrary certificate, and the LWP will gladly accept it until it is signed by a CA that you trust. Confirming a common dummy certificate does not matter, since it has never been verified by LWP.

If you use IO::Socket::SSL as the LWP server, you can enable Common Name verification by setting the verifycn_scheme parameter as follows:

 use IO::Socket::SSL; use Net::SSLeay; BEGIN { IO::Socket::SSL::set_ctx_defaults( verify_mode => Net::SSLeay->VERIFY_PEER(), verifycn_scheme => 'http', ca_path => "/etc/ssl/certs" ); } 

Answer 6

You can also consider Net :: SSLGlue ( http://search.cpan.org/dist/Net-SSLGlue/lib/Net/SSLGlue.pm ) But take care this depends on the recent IO :: Socket :: SSL and Net :: SSLeay versions.

Answer 7

You are right to worry about this. Unfortunately, I don’t think it can be done 100% securely under any of the low-level SSL / TLS bindings that I looked at Perl.

Essentially, you need to pass the hostname of the server that you want to connect to the SSL library before the connection process begins. Alternatively, you can arrange a callback at the right time and interrupt the handshake inside the callback if it is not checked. People writing Perl bindings to OpenSSL seem to be facing problems making the callback interface consistent.

The method for verifying the host name against the server certificate also depends on the protocol. Thus, this should be a parameter of any perfect function.

You might want to see if there are any bindings to the Netscape / Mozilla NSS library. It looked pretty good when I looked at it.

Answer 8

Just run the following command in terminal: sudo cpan install Mozilla :: CA

He must solve this problem.