yahoo test error connection reset

[cpcloud]

we really need to fix this...

https://travis-ci.org/cpcloud/pandas/jobs/8321921
https://travis-ci.org/cpcloud/pandas/jobs/8322345
https://travis-ci.org/jreback/pandas/jobs/8448538
https://travis-ci.org/pydata/pandas/jobs/8449118 (HTTP error here)

[jreback]

I thought the "@network" was supposed to catch these? are these not IOError?

[cpcloud]

no the error generated is socket.error which is basically a light wrapper around one of the C errno numbers

[jreback]

which apparently is not caught by IOError!
so socket.error should prob be added as a default list of things to catch in '@network'?

[cpcloud]

so wish python 2 had this

[jreback]

should we be catcing OSError then?

[cpcloud]

here's why that might be problem

In [1]: open('asdfasdf', 'r')
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-1-efa01a311135> in <module>()
----> 1 open('asdfasdf', 'r')

IOError: [Errno 2] No such file or directory: 'asdfasdf'

In [2]: os.path.getsize('asd09g8')
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-0a206a03b7df> in <module>()
----> 1 os.path.getsize('asd09g8')

/home/phillip/.virtualenvs/pandas/lib/python2.7/genericpath.pyc in getsize(filename)
     47 def getsize(filename):
     48     """Return the size of a file, reported by os.stat()."""
---> 49     return os.stat(filename).st_size
     50
     51

OSError: [Errno 2] No such file or directory: 'asd09g8'

[cpcloud]

of course python3 has FileNotFoundError ARG

[cpcloud]

in python3 socket.error is an alias for OSError so i guess we should catch it then...that should work too for connection errors...pr coming in 100..99..98

[jreback]

These are network calls though, the point is to catch them (obviously if we can find a narrow exception would be good)

[cpcloud]

but if there happens to be a file opening error in a python2 test labelled networrk it won't be caught

[jreback]

so let's add the socket exceptions then (in addition to IOError)

[cpcloud]

i did

_network_error_classes = IOError, OSError

try:
    ConnectionError
except NameError:
    _network_error_classes += socket.error,

[cpcloud]

but probably better is

_network_error_classes = IOError,

try:
    _network_error_classes += ConnectionError,
except NameError:
    _network_error_classes += socket.error,

[cpcloud]

this should work in 2.x, 3.x since 3.2 follows py2 conventions regarding exceptions

[jreback]

are there other socket errors that would want to catch?

[cpcloud]

ssl possibly

[jreback]

I guess throw it in and we see more..then easy to add....

[cpcloud]

all exceptions in socket are subclasses of IOError so that's reassuring

In [32]: print x[0].map(lambda x: getattr(socket, x.split('.')[1]))[10:].map(lambda x: map(lambda y: y.__name__, x.mro())).values
[ ['error', 'IOError', 'EnvironmentError', 'StandardError', 'Exception', 'BaseException', 'object']
 ['gaierror', 'error', 'IOError', 'EnvironmentError', 'StandardError', 'Exception', 'BaseException', 'object']
 ['herror', 'error', 'IOError', 'EnvironmentError', 'StandardError', 'Exception', 'BaseException', 'object']
 ['SSLError', 'error', 'IOError', 'EnvironmentError', 'StandardError', 'Exception', 'BaseException', 'object']]

[cpcloud]

here's how i got x

In [33]: psearch socket.*error*
socket.SO_ERROR
socket.SSL_ERROR_EOF
socket.SSL_ERROR_INVALID_ERROR_CODE
socket.SSL_ERROR_SSL
socket.SSL_ERROR_SYSCALL
socket.SSL_ERROR_WANT_CONNECT
socket.SSL_ERROR_WANT_READ
socket.SSL_ERROR_WANT_WRITE
socket.SSL_ERROR_WANT_X509_LOOKUP
socket.SSL_ERROR_ZERO_RETURN
socket.error
socket.gaierror
socket.herror
socket.sslerror

In [34]: # copy to clipboard via tmux

In [35]: x = read_clipboard(header=None)

[cpcloud]

missed one socket.timeout which is also a subclass of IOError which begs the question why aren't these being caught?

[cpcloud]

i'm not convinced that my PR is necessary, why would IOErrors be caught but not socket.error?

[cpcloud]

let me see if i can somehow trigger this error across a vagrant machine

[jtratner]

Should this be the responsibility of the function to catch these errors? Or
is it supposed to bubble up.

Also, if you hit this error with internet on and then don't with internet
off - then it could be an issue with the calling function rather than
anything internal. Post merge, I ran this sans - internet and only see
errors in two functions that return None or non-complete objects with
IOError.
On Jun 21, 2013 6:16 PM, "Phillip Cloud" [email protected] wrote:

let me see if i can somehow trigger this error across a vagrant machine

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3982#issuecomment-19843883
.

[jreback]

I thought the new function of the network decorator is to by default catch any network related error that is specified and then skip ?

[cpcloud]

it ran successfully with and without internet

[jtratner]

So there's a separate socket error? When I tested, it was raising socket
error as IO... Is it platform dependent?

Also, if you do this, maybe we should create a function that checks whether
something is a network error or maybe context manager... Then could use it
elsewhere.
On Jun 21, 2013 8:25 PM, "Phillip Cloud" [email protected] wrote:

it ran successfully with and without internet

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3982#issuecomment-19847467
.

[cpcloud]

yes the socket module has a handful of exception classes, all subclasses of IOError but the one generated here wasn't being caught by the decorator

[jtratner]

Probably one of the tests decorated by with_connectivity_check. Those try
to ping google (or another url) and if they succeed, they then raise the
error instead of catching it. Just swap out with network. If it's
actually not working that's more problematic. (I notice, for example, that
one test catches and prints the socket errors but actually raises an
AttributeError.

On Fri, Jun 21, 2013 at 11:35 PM, Phillip Cloud [email protected]:

yes the socket module has a couple of exception classes, all subclasses
of IOError but that wasn't being caught by the decorator

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3982#issuecomment-19850454
.

[jreback]

#3985 closes this, right?

[jtratner]

@jreback I think it's #4002 that fixes this (adding in the closing(urlopen(url)) syntax so that it doesn't leave unclosed urlopen results.

[jreback]

I see you moved #4002 to 0.12, this too? trying to close the remaining issues.....

[cpcloud]

still failing: https://travis-ci.org/cpcloud/pandas/jobs/8478753

[jtratner]

@cpcloud Just realized that I didn't set network and with_connectivity_check to close the url...maybe that's contributing?

[cpcloud]

@jtratner 👍 nice work on the network testing wiki i went over there to write some stuff and you already did it!

[jtratner]

no prob

On Wed, Jun 26, 2013 at 9:32 PM, Phillip Cloud [email protected]:

@jtratner https://github.com/jtratner [image: 👍] nice work on the
network testing wiki i went over there to write some stuff and you already
did it!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3982#issuecomment-20091878
.