EDGAR - test issue - stalled build

[femtotrader]

Hello,

there is an issue with CI because of EDGAR tests
see for example https://travis-ci.org/pydata/pandas-datareader/builds/155920545

No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated

Pinging @jtkiley #147

Any idea ?

Kind regards

[jtkiley]

This is an issue with the data source. Sometimes the SEC's FTP server just stops sending data without notification, so it doesn't raise an exception. I typically see it when I'm using it heavily from home (this throttling doesn't seem to apply at all to computers on .edu networks), or when I was making several commits that trigger tests (i.e. making it throttle Travis).

I'd try closing and reopening the PR to bump the test to run again. That usually seemed to work for me if I gave it a couple hours (presumably dropping off the throttling list on the FTP server).

This issue is one reason that I tried to minimize the number of tests (and test multiple things when possible), even though that's not usually ideal.

[femtotrader]

ftplib.FTP have a timeout parameter that could help.

[femtotrader]

Still failling, timeout doesn't help!

https://travis-ci.org/pydata/pandas-datareader/jobs/158398050

[jtkiley]

These pass repeatedly on my university network computer, so I'm still reasonably sure it's a SEC issue with non-.edu domains.

I emailed the SEC to see if there's a way I can code around the throttling and/or if they can add travis-ci.org to their whitelist.

[femtotrader]

Thanks @jtkiley

[femtotrader]

Any news @jtkiley ?
Pinging @sinhrks

[jtkiley]

The reply I received said that FTP does not actually have any throttling, but that they do have a lot of issues where multi-threaded jobs on AWS saturate their server. To be fair, I'm a little skeptical that there's not some kind of preference shown to .edu domains (whether it's obvious to them or not), as it behaves much differently for me from the university network, though, admittedly, I may not have a big enough sample size to be sure that I didn't just get lucky.

I suppose the better question at this point is what to do about it. We already have them skip when it actually throws an error, but I'm not sure what our options are when the connection just stalls. For what it's worth, I can run the tests successfully over and over from the university network.

Thoughts?

[femtotrader]

See #269

[gfyoung]

Update: we've disabled all EDGAR tests for the time being due to other issues with data returned. Thus, this should no longer be an issue ATM.

[bashtage]

EDGAR has been deprecated.