Open Persian URLs with urllib2

I am trying to open the URL http: // الاعلي-للاتصالات.قطر / ar / news-events / event / future-internet-privacy with urllib2.urlopen, but it always reports an error.

A similar thing happens with http: // الاعلي-للاتصالات.قطر / ar ... other pages (Chinese) open normally.

Any ideas point me to the correct way to open these URLs?

urllib2.urlopen("http://الاعلي-للاتصالات.قطر/ar/news-events/event/future-internet-privacy").read() urllib2.urlopen('http://الاعلي-للاتصالات.قطر').read() 

[Changed] error:

 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.6/urllib2.py", line 391, in open response = self._open(req, data) File "/usr/lib/python2.6/urllib2.py", line 409, in _open '_open', req) File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain result = func(*args) File "/usr/lib/python2.6/urllib2.py", line 1170, in http_open return self.do_open(httplib.HTTPConnection, req) File "/usr/lib/python2.6/urllib2.py", line 1142, in do_open h.request(req.get_method(), req.get_selector(), req.data, headers) File "/usr/lib/python2.6/httplib.py", line 914, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.6/httplib.py", line 951, in _send_request self.endheaders() File "/usr/lib/python2.6/httplib.py", line 908, in endheaders self._send_output() File "/usr/lib/python2.6/httplib.py", line 780, in _send_output self.send(msg) File "/usr/lib/python2.6/httplib.py", line 759, in send self.sock.sendall(str) 

I also tried using u'http: // الاعلي-للاتصالات.قطر '. encode ('utf-8'), but the result could not be opened.

+4
source share
1 answer

As @Donal says, the URL should be punycoded . Fortunately, Python includes this already. Here is a sample Python code

 domain = "الاعلي-للاتصالات.قطر" domain_unicode = unicode(domain, "utf8") domain_idna = domain_unicode.encode("idna") urllib2.urlopen("http://" + domain_idna).read() 

Hope this helps.

+8
source

Source: https://habr.com/ru/post/1381485/


All Articles