25

I have an little app I wrote in Python and it used to work... until yesterday, when it suddenly started giving me an error in a HTTPS connection. I don't remember if there was an update, but both Python 2.7.3rc2 and Python 3.2 are failing just the same.

I googled it and found out that this happens when people are behind a proxy, but I'm not (and nothing have changed in my network since the last time it worked). My syster's computer running windows and Python 2.7.2 has no problems (in the same network).

>>> url = 'https://www.mediafire.com/api/user/get_session_token.php'
>>> response = urllib2.urlopen(url).read()
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1215, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>

What's wrong? Any help is appreciated.

PS.: Older python versions don't work either, not in my system and not in a live session from USB, but DO work in a Ubuntu 11.10 live session.

5
  • 1
    Does it happen for every SSL site you try to contact, or just the one? If it doesn't occur for every site, then could you tell us what site is causing the problem? Mar 26, 2012 at 4:03
  • Well, I'm not an experienced programmer myself, and I'm trying to read a page from a site's API, and that's the only call that requires SSL, so I don't know if I was doing it right in the first place. I've been using it like a normal urllib.urlopen(url).read() call and it was working. Could you please give me another site's address or a python script that would answer this question?
    – Pablo
    Mar 26, 2012 at 5:54
  • Oh, I forgot to mention: the site is Mediafire. It's its get_session_token call that is causing the problem.
    – Pablo
    Mar 26, 2012 at 6:12
  • I was able to reproduce this with that site. I've updated your question to include the site in question. I suspect that this is a problem with OpenSSL, since wget fails too. Mar 26, 2012 at 6:38
  • This happens with stream.twitter.com for me at the time of writing.
    – MarkR
    Mar 20, 2013 at 11:30

7 Answers 7

17

This appears to be related to the addition of TLS 1.1 and 1.2 support to the version of OpenSSL found in 12.04. The connection failure can be reproduced with the OpenSSL command line tool:

$ openssl s_client -connect www.mediafire.com:443
CONNECTED(00000003)
140491065808544:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 320 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

The connection succeeds if I force the connection to use TLS 1.0 with the -tls1 command line argument.

I would suggest you file a bug report about this problem here:

https://bugs.launchpad.net/ubuntu/+filebug

3
  • 2
    Thank you! I reported a bug. Please, see if you can add any relevant info to it: bugs.launchpad.net/ubuntu/+source/openssl/+bug/965371
    – Pablo
    Mar 26, 2012 at 15:06
  • 1
    How does this help him work around the problem in Python?
    – Cerin
    Sep 23, 2013 at 20:46
  • 2
    @Cerin: it isolated the problem as an OpenSSL bug rather than something in Python, and directed him to use the bug tracker. That problem has since been fixed. Sep 24, 2013 at 0:47
13

For python novices like me, here is the way to override httplib the easiest way. At the top of your python script, include these lines:


import httplib
from httplib import HTTPConnection, HTTPS_PORT
import ssl

class HTTPSConnection(HTTPConnection):
    "This class allows communication via SSL."
    default_port = HTTPS_PORT

    def __init__(self, host, port=None, key_file=None, cert_file=None,
            strict=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
            source_address=None):
        HTTPConnection.__init__(self, host, port, strict, timeout,
                source_address)
        self.key_file = key_file
        self.cert_file = cert_file

    def connect(self):
        "Connect to a host on a given (SSL) port."
        sock = socket.create_connection((self.host, self.port),
                self.timeout, self.source_address)
        if self._tunnel_host:
            self.sock = sock
            self._tunnel()
        # this is the only line we modified from the httplib.py file
        # we added the ssl_version variable
        self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, ssl_version=ssl.PROTOCOL_TLSv1)

#now we override the one in httplib
httplib.HTTPSConnection = HTTPSConnection
# ssl_version corrections are done

From here on, you can use urllib or whatever you use just like you normally would.

Note: This is for python 2.7. For a python 3.x solution, you need to override the HTTPSConnection class found in http.client. I leave that as an exercise for the reader. :-)

5
  • 2
    I really like this solution, it avoids modifying any system libraries or other hackery.
    – MarkR
    Mar 20, 2013 at 11:28
  • 4
    Fails using Python 2.7.4 on Ubuntu 12.04: NameError: name 'socket' is not defined. --- You'll need to add "import socket" as well. Jul 24, 2013 at 20:34
  • Works great on Ubuntu 13.04. Thanks!
    – dharmatech
    Aug 31, 2013 at 22:04
  • 2
    There is no reason to only patch httplib. People may use other SSL sockets. One could patch ssl instead as in my answer below.
    – temoto
    Sep 16, 2013 at 8:36
  • This gives me the error BadStatusLine: ''
    – Cerin
    Sep 23, 2013 at 21:08
8

You can avoid modifying the httplib.py file by modifying your HTTPSConnection object:

import httplib, ssl, socket

conn = httplib.HTTPSConnection(URL.hostname)
sock = socket.create_connection((conn.host, conn.port), conn.timeout, conn.source_address)
conn.sock = ssl.wrap_socket(sock, conn.key_file, conn.cert_file, ssl_version=ssl.PROTOCOL_TLSv1)
conn.request('POST', URL.path + URL.query)

The request method creates a new socket only if connection.sock is not defined. Creating your own one adding the ssl_version parameter will make the request method use it. Then everything else works as usual.

I was having the same issue and this works for me.

Regards

0
7

The problem is in ssl, it has nothing to do with HTTP, so why patching httplib if you can patch ssl. The following code should fix all SSL sockets including, but not limited to HTTPS, for Python 2.6+ (built in ssl, did not try with pyopenssl).

import functools
import ssl

old_init = ssl.SSLSocket.__init__

@functools.wraps(old_init)
def ubuntu_openssl_bug_965371(self, *args, **kwargs):
  kwargs['ssl_version'] = ssl.PROTOCOL_TLSv1
  old_init(self, *args, **kwargs)

ssl.SSLSocket.__init__ = ubuntu_openssl_bug_965371
1
  • Good answer. Nice, elegant way to solve the problem.
    – chnrxn
    May 10, 2015 at 13:39
3

EDIT httplib.py (/usr/lib/pythonX.X/httplib.py on Linux)

FIND HTTPSConnection class declaration

  class HTTPSConnection(HTTPConnection):
....

Inside class code CHANGE line

self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)

TO

self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, ssl_version=ssl.PROTOCOL_TLSv1)

Then httplib HTTPS request should work

import httplib
from urlparse import urlparse
url = XXX
URL = urlparse(url)
connection = httplib.HTTPSConnection(URL.hostname)
connection.request('POST', URL.path + URL.query)
response = connection.getresponse()
1
  • 3
    It really isn;t right to edit a system file like that. Instead, redefine any definitions that need to be changed, by redefining them in your code.
    – nanofarad
    Jan 13, 2013 at 0:47
2

This issue is likely due to SSLv2 being disabled on the web server, but Python 2.x tries to establish a connection with PROTOCOL_SSLv23 by default.

Here's the link to my answer for a similar issue on Stack Overflow - https://stackoverflow.com/a/24166498/41957

Update: this is functionally the same as @temoto's answer above.

4
  • TypeError: unbound method __init__() must be called with SSLSocket instance as first argument (got _socketobject instance instead)
    – sureshvv
    May 8, 2015 at 8:14
  • Hmm, partial() doesn't work for class methods. Will post a better solution shortly.
    – chnrxn
    May 10, 2015 at 2:47
  • @sureshvv, if you can help to check the solution it will be appreciated.
    – chnrxn
    May 10, 2015 at 9:19
  • @temeto's answer worked.
    – sureshvv
    May 12, 2015 at 6:59
1

A simple fix that worked for me was to override SSL's default protocol:

import ssl
ssl.PROTOCOL_SSLv23 = ssl.PROTOCOL_TLSv1
1
  • It's hackish, but it works rather well in today's context. Ever since the poodle vulnerability has been discovered, TLSv1 pretty much became the only acceptable version on the Internet.
    – chnrxn
    May 10, 2015 at 4:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .