Python HTTPS requests (urllib2) to some sites fail on Ubuntu 12.04 without proxy

Question

I have an little app I wrote in Python and it used to work... until yesterday, when it suddenly started giving me an error in a HTTPS connection. I don't remember if there was an update, but both Python 2.7.3rc2 and Python 3.2 are failing just the same.

I googled it and found out that this happens when people are behind a proxy, but I'm not (and nothing have changed in my network since the last time it worked). My syster's computer running windows and Python 2.7.2 has no problems (in the same network).

>>> url = 'https://www.mediafire.com/api/user/get_session_token.php'
>>> response = urllib2.urlopen(url).read()
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1215, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>

What's wrong? Any help is appreciated.

PS.: Older python versions don't work either, not in my system and not in a live session from USB, but DO work in a Ubuntu 11.10 live session.

Answer 1

This appears to be related to the addition of TLS 1.1 and 1.2 support to the version of OpenSSL found in 12.04. The connection failure can be reproduced with the OpenSSL command line tool:

$ openssl s_client -connect www.mediafire.com:443
CONNECTED(00000003)
140491065808544:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 320 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

The connection succeeds if I force the connection to use TLS 1.0 with the -tls1 command line argument.

I would suggest you file a bug report about this problem here:

https://bugs.launchpad.net/ubuntu/+filebug

Answer 2

You can avoid modifying the httplib.py file by modifying your HTTPSConnection object:

import httplib, ssl, socket

conn = httplib.HTTPSConnection(URL.hostname)
sock = socket.create_connection((conn.host, conn.port), conn.timeout, conn.source_address)
conn.sock = ssl.wrap_socket(sock, conn.key_file, conn.cert_file, ssl_version=ssl.PROTOCOL_TLSv1)
conn.request('POST', URL.path + URL.query)

The request method creates a new socket only if connection.sock is not defined. Creating your own one adding the ssl_version parameter will make the request method use it. Then everything else works as usual.

I was having the same issue and this works for me.

Regards

Answer 3

EDIT httplib.py (/usr/lib/pythonX.X/httplib.py on Linux)

FIND HTTPSConnection class declaration

  class HTTPSConnection(HTTPConnection):
....

Inside class code CHANGE line

self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)

TO

self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, ssl_version=ssl.PROTOCOL_TLSv1)

Then httplib HTTPS request should work

import httplib
from urlparse import urlparse
url = XXX
URL = urlparse(url)
connection = httplib.HTTPSConnection(URL.hostname)
connection.request('POST', URL.path + URL.query)
response = connection.getresponse()

Answer 4

For python novices like me, here is the way to override httplib the easiest way. At the top of your python script, include these lines:

import httplib
from httplib import HTTPConnection, HTTPS_PORT
import ssl

class HTTPSConnection(HTTPConnection):
    "This class allows communication via SSL."
    default_port = HTTPS_PORT

    def __init__(self, host, port=None, key_file=None, cert_file=None,
            strict=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
            source_address=None):
        HTTPConnection.__init__(self, host, port, strict, timeout,
                source_address)
        self.key_file = key_file
        self.cert_file = cert_file

    def connect(self):
        "Connect to a host on a given (SSL) port."
        sock = socket.create_connection((self.host, self.port),
                self.timeout, self.source_address)
        if self._tunnel_host:
            self.sock = sock
            self._tunnel()
        # this is the only line we modified from the httplib.py file
        # we added the ssl_version variable
        self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, ssl_version=ssl.PROTOCOL_TLSv1)

#now we override the one in httplib
httplib.HTTPSConnection = HTTPSConnection
# ssl_version corrections are done

From here on, you can use urllib or whatever you use just like you normally would.

Note: This is for python 2.7. For a python 3.x solution, you need to override the HTTPSConnection class found in http.client. I leave that as an exercise for the reader. :-)