DISCUSS: move online data reader subpackages to separate project?

[jorisvandenbossche]

Opening an issue to discuss this: should we think about moving the functionality to read online data sources to a separate package?

This was mentioned recently here #8842 (comment) and #8631 (comment)

Some reasons why we would want to move it:

• This often needs much more urgent releases than pandas to fix changes in the remote sources. So with separate release cycle, this can speed up propagation of bugfixes to users
• More general: to limit the scope of pandas as the core project, and to encourage that some people take ownership of this subproject (which is easier as a separate project?), publicize it separately, ..

Some questions that come to mind:

• What should we move? (everything in remote_data.rst?)
  • Everything in io.data? (the DataReader function for Yahoo and Google Finance, FRED and Fama/French, and the Options class
  • Also io.wb (certain World Bank data) and io.ga (google analytics interface)?
• Should we move it all to one separate project, or to multiple?
• Are there outside of pandas already some packages like this? (possibilities to merge with that/collaborate/..)
• Who would be interested to take this up? To be a maintainer of such a package?

Pinging some of the people have worked on these subpackages (certainly add others if you know):
@dstephens99 @MichaelWS @kdiether @cpcloud @vincentarelbundock @jnmclarty
@jreback @immerrr @TomAugspurger @hayd @jtratner

[immerrr]

I was trying to think through the workflow of such migration and it hit me that there's one thing all those subprojects to be moved away get for free when being under the umbrella of pandas: the utilities aka the boring stuff, like setup scripts, docs, tests, benchmarks & ci infrastructure. Maybe we could think of a way to somehow move crucial parts of those into a subrepo(-s) so that a fix in one place can be easily propagated everywhere.

Also, as a food for thought, I think there are a lot of non-remote io packages that can benefit from moving away and being provided as plugins, e.g. fresh and unstable stata, clipboard, maybe excel. If you ask me, I'd say pretty much every library that works through a 3rdparty dependency could be worth separating and being maintained closer to that dependency package by a person having experience with it. One organizational bonus of such separation would be that pandas will have a lot less optional dependencies.

[jreback]

I am not in favor of moving any core IO packages eg stata,clipboard,excel

one of the mains points of pandas is that it provides a consistent interface with IO
not sure having these as separate packages will help that and just add another dependency where except for excel don't actually have any deps

[immerrr]

FTR, clipboard does have deps, they are simply hidden under a layer of pandas.util.clipboard:

• gtk, PyQT4 and PySide modules:

$ grep import pandas/util/clipboard.py 
#   import pyperclip
import platform, os
    import ctypes
                import gtk
                    import PyQt4 as qt4
                    import PyQt4.QtCore
                    import PyQt4.QtGui
                        import PySide as qt4
                        import PySide.QtCore
                        import PySide.QtGui

• pbcopy,pbpaste, xclip,xsel commands

$ grep "os\.\(popen\|system\)" pandas/util/clipboard.py 
    outf = os.popen('pbcopy', 'w')
    outf = os.popen('pbpaste', 'r')
    outf = os.popen('xclip -selection c', 'w')
    outf = os.popen('xclip -selection c -o', 'r')
    outf = os.popen('xsel -i', 'w')
    outf = os.popen('xsel -o', 'r')
    xclipExists = os.system('which xclip > /dev/null') == 0
        xselExists = os.system('which xsel > /dev/null') == 0

[jreback]

@immerrr hah..I knew you were going to say that. However, these are system level for the most part and generally included on linux/mac/windows (except the PyQt). So not that big of a deal.

[immerrr]

Also, it depends on what do you call "core". I always thought of pandas core being containers+time utils+split/apply/combine+high perf queries/computations. The rest could (should) just feed data in or out via a high-throughput API.

That being said, I agree that unified API is good, so it should be sorted out before starting to pull pandas codebase apart. And yes, for practicality reasons, most frequently used io modules could be considered core, too, I just don't have the data to figure out which of them are most popular.

[jorisvandenbossche]

Maybe we should limit the discussion first to the remote data readers? As the other io modules are indeed much more fundamental -> this would require a more general discussion on the scope of pandas as the core project.
And the remote data could also be a good exercise to see how smooth this goes / what obstacles there are (things that would also come up in a broader discussion), and that already will be difficult enough to sort out now:

• the things @immerrr raises on setup scripts, docs, tests, benchmarks & ci infrastructure. Very good points I think. How can we facilitate this from the pandas side?
• A better defined internal API? (packages that build upon pandas will want to use certain utility functions, such as functions in core.common)

[jreback]

I think absolutely these sub-projects should use the existing tests (and pandas testing infrastructure).
Doc building actually could be on independently via readthedocs (no cython in these projects).
Pandas could maintain linkage from its docs to the sub-projects. Though I think they should physically be separate.

[immerrr]

I think absolutely these sub-projects should use the existing tests

It's not obvious from the phrasing, I hope you mean that they should re-use the infrastructure, e.g. build matrix, but neither they shouldn't run tests of core pandas nor core pandas should run tests of its "plugin" packages. Which brings me to another issue which I'll describe in separate gh issue.

Speaking of build matrix, I'd expect the latter to be tested against pandas master and at least one last pandas major release. Or maybe two. The second one would ensure that when running for a fix for last-minute incompatible change in newly released core version one doesn't break the latest major release as of several days ago and don't force users to upgrade both pandas-io-foo and pandas in lockstep. But then again, if one wants to use a bleeding edge version of pandas-io-foo package they should be able to do the same for pandas itself.

Another issue is that I don't know of a way to include one yaml file inside the other which stacks with the fact that travis.yml should reside in repository root. I'm not yet sure I see how to make travis/appveyor.yml generic enough to require minimal (ideally, none whatsoever?) attention after project is bootstrapped and yet rely on some common submodule so that exact versions can be changed with a single commit to some "shared" repo and a single submodule update. Symlinking travis.yml to a submodule might do, but I'm not sure if that works on windows.

[femtotrader]

Hello,

I've been working on a more object oriented approach about DataReader than actual code.

so it can use urlopen or requests (for now I'm only using requests as my goal was to use requests-cache) #8713

see this https://github.com/femtotrader/pandas_datareaders
Sorry, I wasn't aware of this "issue".

This is just a friendly "fork", please excuse me.

I would be very pleased to have your feedback about this.

[hayd]

+1, it sucks that someone (stuck in the dark ages) using 0.10 or something can't use data readers (if something's changed in the api since then) without upgrading and potentially breaking/changing lots of their code.

I suspect for the most part we're not using any wild/depreciating commands, that aren't tested elsewhere in pandas, the datafeed codebase... so IMO testing against master isn't actually that important (and not doing so makes things much easier), just add the pandas version - e.g. 0.16.rc1 - to travis for the package right before a pandas release... see if anything breaks. We could potentially provide a way to test all these packages from within pandas (for a while we should keep the tests around in pandas), but tbh I don't think this is so important: a failure right before a release would be a big edge case (that's why we have RCs) and all you'd have to do is fix up the package and release a new version.

For the backwards compat these have to dependancies of (and depend on*) pandas right?

I think the easiest is to add this right here in the pydata umbrella group, or create another group here; that way it can seem more "official".

*though it may be interesting to lift that direction, and have pandas as a soft dependancy (like https://github.com/changhiskhan/poseidon does).

Does this make sense?

---

@femtotrader did you copy and paste the classes and tests from pandas or is this something different? (I worry a little about change requests dep at the same time as migrating, but +1 on using requests).

[femtotrader]

@hayd this is something different as (except Yahoo Options) in data.py https://github.com/pydata/pandas/blob/master/pandas/io/data.py almost everything is function... no class. Some people considers that we shouldn't depend on requests. I think that using these classes we can both depend on requests if we want or use (urllib or urllib2) urlopen.

[jnmclarty]

One important consideration is the recruiting effects of newer contributors. I have points on each of the three, speaking as a rookie myself. I comment on the docs as well, as @jreback pulled that into this thread.

Carve out Documentation

• Barriers to Entry Reduced. Stripping out the need to compile the most recent commit, to help fix typos, or clarity, makes complete sense. I would be more likely to contribute to docs for 0.16, if I could leave 0.14 installed.

My recommendation : If anything gets carved out, documentation should be the highest priority.

Carve out Data Sources

• Barriers to Entry Mixed. For new maintainers, barriers are higher, as mentioned above. For new contributors, entry barriers are lower, for similar reasons as the Documentation.
• As other's have said, +1 to finding a way to use the existing pandas framework for the boring stuff.

My recommendation : Add a note to the docs advising that maintainers are wanted. And, as they step up, one by one they can be carved out. For instance, I could likely learn, and volunteer, to be a maintainer for the World Bank stuff, but I wouldn't necessarily want to be a maintainer for the other stuff.

Carve out IO

• Barriers to Entry Increased. My (potentially wrong) perception would be that helping on something like this would be more complicated, if it was a separate project. I would be dissuaded from trying. I would be intimidated by the implications of version mismatch. Eg. "For this commit, how do I test against all the necessary, and next, versions of pandas? That sounds hard."

My recommendation : It has it's benefits, but I think the costs outweigh them. I think the "darkages" reference above has merit, but it is trumped by the complicatedness that pandas would become.

[hayd]

@jnmclarty The problem with carving out docs is IMO that docs and code are "hard-linked" (changes in the code api means changing the docs... at the same time), I think @jreback is talking about docs for these sub-projects (not pandas docs in general/core).

Note: you can change the docs without compiling, you can even do it within github!

I envision users of older pandas (dark ages) being able to monkey-patch (depending on their code):

# import pandas.io.data as web  # previous way (and still works in newer pandas, but with latest data package)
import pandas_data_readers as web  # becomes, to work in older pandas, to use latest data package

This reminds me of @jreback's presentation "pandas is pydata middleware":

Ripping it out and seeing what happens: https://github.com/hayd/pandas_data_readers (it works without change for pandas 0.13-0.15, there's a few compat import issues pre-0.13 see https://travis-ci.org/hayd/pandas_data_readers/builds/43218562).

Edit: may be easy to also break apart the compat module - would make fixing earlier version easy - not sure if that's feasible though.