CLN/ENH/BLD: Remove need for 2to3 for Python 3. #4384

jtratner · 2013-07-27T20:58:22Z

Many changes to make codebase compatible in 2 and 3. For range, zip, map, etc. tried to favor iterators over needing to use lists.

Changes:

No more 2to3 in setup.py
merges util/compat and util/py3compat into pandas/compat
incorporates useful parts of the six library into compat (+ adds SIX to LICENSES)
defaults to using iterators in both Python 2 and Python 3 for range, zip, map, and filter
Adds lrange, lzip, lmap, and lfilter, which wrap corresponding iterator methods with lists
Improved type checks (where appropriate --> many need to be rigid to work with pandas)
various other utilities to be Py2/3 compatible (iteritems, iterkeys, itervalues, etc.)
deprecates iterkv with a warning (no longer necessary because library no longer uses 2to3)
compatibility wrapper around dateutil for handling unicode when version <= 2.0 + 1.5 dateutil build in Travis

Switches everything to use special iteritems, so can deprecate as discussed in #4372.

This iteritems use "iteritems" method if available and otherwise uses items.

Checklist of packages to run through 2to3 to check:

Other tasks:

alias __bool__ to __nonzero__

cpcloud · 2013-07-27T21:36:12Z

I think GitHub might explode if I click on the files changed tab 💥

cpcloud · 2013-07-27T21:37:00Z

I think filter is also an iterator in Python 3 IIRC

jtratner · 2013-07-27T21:51:06Z

@cpcloud it is...still coming up on my list :P

jtratner · 2013-07-27T21:53:48Z

@cpcloud actually, only used in one place in library and it's used as an iterator anyways. still need to check for itertools changes.

cpcloud · 2013-07-27T21:57:15Z

great. i think i might use filter in eval-3393, i'll check

jtratner · 2013-07-27T22:16:45Z

@cpcloud just put from six.moves import filter at the top of the file and you'll be fine. It just uses itertools.ifilter under the hood, so slap a list on it otherwise. Very simple

cpcloud · 2013-07-27T22:18:11Z

cool thanks! pandas is going to rule the world soon.

cpcloud · 2013-07-27T22:23:00Z

i hope python 3 testing is going to be faster after this .... those 2to3 runs seem like they take FOREVER

jtratner · 2013-07-27T22:25:59Z

@cpcloud I think it makes the entire test suite a little faster...not sure.

cpcloud · 2013-07-27T22:46:16Z

nice looks like travis build times are 3-4 mins faster that is cool

jtratner · 2013-07-27T22:53:16Z

@cpcloud does Cython have any changes for Python 3?

cpcloud · 2013-07-27T22:56:49Z

I believe all of those issues are taken care of during translation from the Cython to C. I don't see why any Cython would need to be changed. Are you getting compilation errors?

jtratner · 2013-07-27T23:29:07Z

@cpcloud guess I can empirically see that that's not true...just gone through so many changes (and I can't get py3 pandas to work on my mac).

jtratner · 2013-07-27T23:36:09Z

@cpcloud I'm getting all these weird stata reader errors. Do you have any ideas on what might be causing them?

cpcloud · 2013-07-28T00:03:16Z

@jtratner can u link me to a build with those errors?

jtratner · 2013-07-28T00:04:52Z

@cpcloud check out the first few errors on here: https://travis-ci.org/jtratner/pandas/jobs/9563671

cpcloud · 2013-07-28T00:27:21Z

have you tried stepping with a debugger?

jtratner · 2013-07-28T00:28:54Z

@cpcloud I've been having trouble getting pandas with Python 3 to install on my mac, which is why I haven't been able to step through.

jtratner · 2013-07-28T00:29:14Z

@cpcloud but eventually I'll figure out how to get it to work

cpcloud · 2013-07-28T00:30:11Z

ah ok... it's weird because i think you actually have to setup.py install it for 2to3 to run which is annoying whenever you make a change...you have to keep doing it

jtratner · 2013-07-28T00:33:34Z

@cpcloud well, the current build has 2to3 disabled, so you don't actually need to do that. I'm trying a separate build with 2to3 enabled to see if it resolves the problem.

cpcloud · 2013-07-28T00:34:47Z

indeed, i hadn't pulled down your PR in my py3.2 venv yet 😄

cpcloud · 2013-07-28T00:35:22Z

anyway my dev version works so just keep at it i think you'll get it ... i would offer to clone my env but i'm not using mac

jtratner · 2013-07-28T00:36:27Z

@cpcloud okay, pretty sure I can figure it out. It doesn't fail with any of the stata errors with 2to3, so that means I just have to apply some of those fixes and it will work :).

cpcloud · 2013-07-28T00:55:07Z

for some reason the path_or_buf.read() calls are not reading anything...

cpcloud · 2013-07-28T00:55:22Z

(in _read_header)

jtratner · 2013-07-28T01:00:19Z

@cpcloud thanks. And I found all the non-stata errors are resolved by 2to3'ing pandas/core.

cpcloud · 2013-07-28T01:14:20Z

ahh so the problem is actually not in _read_header you need to convert the calls to np.where(map(...)) to np.where(list(map())) then you're golden...i just tried it, it works

cpcloud · 2013-07-28T01:14:33Z

that's in the data method

cpcloud · 2013-07-28T01:14:46Z

i love getting to the bottom of things!

cpcloud · 2013-07-30T01:22:07Z

sorry i wrote that on the wrong thread...see the eval thread!

jreback · 2013-07-30T01:39:00Z

@jtratner

rebases all worked (with a little TLC..) except for 1 thing:

FAIL: test_iterkv_deprecation (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/jreback/pandas/build/lib.linux-x86_64-3.3/pandas/tests/test_frame.py", line 10316, in test_iterkv_deprecation
    self.mixed_float.iterkv()
  File "/usr/local/lib/python3.3/contextlib.py", line 55, in __exit__
    next(self.gen)
  File "/mnt/home/jreback/pandas/build/lib.linux-x86_64-3.3/pandas/util/testing.py", line 1058, in assert_produces_warning
    % expected_warning.__name__)
AssertionError: Did not see expected warning of class 'DeprecationWarning'.
``
which works when I run only tests/test_frame.py but not all tests...
?

jtratner · 2013-07-30T01:42:39Z

Maybe I put it in as future warning by accident? Or did we lose the
commit/change that added the deprecation?
On Jul 29, 2013 9:39 PM, "jreback" [email protected] wrote:

@jtratner https://github.com/jtratner

rebases all worked (with a little TLC..) except for 1 thing:

FAIL: test_iterkv_deprecation (pandas.tests.test_frame.TestDataFrame)

Traceback (most recent call last):
File "/mnt/home/jreback/pandas/build/lib.linux-x86_64-3.3/pandas/tests/test_frame.py", line 10316, in test_iterkv_deprecation
self.mixed_float.iterkv()
File "/usr/local/lib/python3.3/contextlib.py", line 55, in exit
next(self.gen)
File "/mnt/home/jreback/pandas/build/lib.linux-x86_64-3.3/pandas/util/testing.py", line 1058, in assert_produces_warning
% expected_warning.name)
AssertionError: Did not see expected warning of class 'DeprecationWarning'.
``
which works when I run only tests/test_frame.py but not all tests...
?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/4384#issuecomment-21764382
.

jtratner · 2013-07-30T01:51:16Z

@jreback oh didn't see your last line, I have an idea...let me look.

jtratner · 2013-07-30T01:53:47Z

@jreback which branch?

jreback · 2013-07-30T01:54:35Z

msgpack3 (simpler one)

jtratner · 2013-07-30T02:04:37Z

Why does it require pytest?

======================================================================
ERROR: Failure: ImportError (No module named 'pytest')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/failure.py", line 38, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/loader.py", line 413, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 160, in load_module
    return load_source(name, filename, file)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 109, in load_source
    return _LoadSourceCompatibility(name, pathname, file).load_module(name)
  File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1023, in load_module
  File "<frozen importlib._bootstrap>", line 1004, in load_module
  File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 869, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "/Users/jtratner/projects/python/pandas3/pandas/tests/test_msgpack/test_except.py", line 4, in <module>
    from pytest import raises
ImportError: No module named 'pytest'

======================================================================
ERROR: Failure: ImportError (No module named 'pytest')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/failure.py", line 38, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/loader.py", line 413, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 160, in load_module
    return load_source(name, filename, file)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 109, in load_source
    return _LoadSourceCompatibility(name, pathname, file).load_module(name)
  File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1023, in load_module
  File "<frozen importlib._bootstrap>", line 1004, in load_module
  File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 869, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "/Users/jtratner/projects/python/pandas3/pandas/tests/test_msgpack/test_obj.py", line 4, in <module>
    from pytest import raises
ImportError: No module named 'pytest'

======================================================================
ERROR: Failure: ImportError (No module named 'pytest')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/failure.py", line 38, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/loader.py", line 413, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 160, in load_module
    return load_source(name, filename, file)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 109, in load_source
    return _LoadSourceCompatibility(name, pathname, file).load_module(name)
  File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1023, in load_module
  File "<frozen importlib._bootstrap>", line 1004, in load_module
  File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 869, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "/Users/jtratner/projects/python/pandas3/pandas/tests/test_msgpack/test_pack.py", line 6, in <module>
    from pytest import raises, xfail
ImportError: No module named 'pytest'

======================================================================
ERROR: Failure: ImportError (No module named 'pytest')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/failure.py", line 38, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/loader.py", line 413, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 160, in load_module
    return load_source(name, filename, file)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 109, in load_source
    return _LoadSourceCompatibility(name, pathname, file).load_module(name)
  File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1023, in load_module
  File "<frozen importlib._bootstrap>", line 1004, in load_module
  File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 869, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "/Users/jtratner/projects/python/pandas3/pandas/tests/test_msgpack/test_sequnpack.py", line 7, in <module>
    from pytest import raises
ImportError: No module named 'pytest'

======================================================================
FAIL: test_iterkv_deprecation (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jtratner/projects/python/pandas3/pandas/tests/test_frame.py", line 10316, in test_iterkv_deprecation
    self.mixed_float.iterkv()
  File "/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/contextlib.py", line 55, in __exit__
    next(self.gen)
  File "/Users/jtratner/projects/python/pandas3/pandas/util/testing.py", line 1058, in assert_produces_warning
    % expected_warning.__name__)
AssertionError: Did not see expected warning of class 'DeprecationWarning'.

----------------------------------------------------------------------
Ran 3537 tests in 166.243s

jreback · 2013-07-30T02:06:25Z

its there from where when msgpack tests originally created; been meaning to take it out

jtratner · 2013-07-30T02:06:34Z

@jreback easy to put raises-equivalent into util/testing

Also, you should change this line:

    unpacker = Unpacker(six.BytesIO(b'foobar'), read_size=3)

to the BytesIO from pandas.compat

jreback · 2013-07-30T02:10:20Z

what about the rest of them that directly import from six?

jtratner · 2013-07-30T02:12:09Z

hold on, I have a diff for you.

jreback · 2013-07-30T02:15:39Z

I fixed them all....how do I do raw bytes? b'foo' ?

jtratner · 2013-07-30T02:19:13Z

yeah, that's cross-compatible. I just posted a commit on jtratner/msgpack3 that fixes everything. Main thing is you need to pass string encoding in test_seq.py and wrap around chr

jtratner · 2013-07-30T02:19:28Z

(hadn't seen your note before that...)

jtratner · 2013-07-30T02:19:44Z

binarydata = [bytes(chr(i), 'utf-8') for i in range(256)]

jtratner · 2013-07-30T02:24:34Z

nvm, my example doesn't work for that

cpcloud · 2013-07-30T02:58:25Z

actually assert_raises is in nose.tools

jreback · 2013-07-30T02:59:04Z

jreback@d6c8279

look at test_msgpack/test_seq.py

I had to hack a bit, encoding in py3, but passing str in py2...seems to work, but never could figure out why

jtratner · 2013-07-30T03:08:11Z

@jreback well, in Python 3, everything is unicode, whereas in py2 everything is encoded by default, so using bytes requires setting explicit coding. In python 2 bytes is exactly the same as str.

jreback · 2013-07-30T13:02:21Z

@jtratner I resolved that issue with assert_produces_warning....

I was calling iterkv (in the msgpack ).....which was working correctly. I suspect that the warnings filter somehow was not turned back on correctly though in assert_produces_warning....because this happened before the test failure (of the test for calling iterkv)....

IOW I was calling iterkv accidently, which produced a warning (but I think globally warnings are somehow turned off for DeprecationWarning).....

then when it hit the specific test, warnings were still turned off!

weird

jtratner · 2013-07-30T16:12:02Z

warnings are incredibly finicky. I'm -1 on checking for things like
DeprecationWarnings, especially because I think they don't appear in a
standard interpreter.

jreback · 2013-07-30T16:37:58Z

agreed.....I in fact usually when I test something like this I make it raise while I am testing (which in this case would have found the cases where I didn't fix the deprecated routine)....

so

maybe need something that is a sub-class of DeprecationWarning that assert_produces_warning can really 'catch' (but in normal usage just spews a warning)?

just thinking out loud here

cpcloud · 2013-07-30T16:52:33Z

there's no runtime switch to turn on deprecation warnings?

jreback · 2013-08-05T23:57:18Z

http://ondrejcertik.blogspot.com/2013/08/how-to-support-both-python-2-and-3.html?m=1

I guess we had an easier time than SymPy

jtratner · 2013-08-06T00:07:11Z

python-modernize was an amazingly helpful tool in this - it wraps 2to3 in some useful ways. Running it once gets you 90% of the way there. The big issue for me was that I wanted to use iterators as much as possible so 2.X wouldn't have unnecessary list calls - that took the majority of the time (as well as integrating the six library). I don't know whether it had a performance boost, but if I hadn't cared about that, it probably would've been easier.

There were two particularly hard parts:

Cross-compatible urllib/httplib imports and exceptions.
That weird unicode error we found and fixed with StringIO.

jtratner deleted the better-python3-compat branch July 30, 2013 01:53

This was referenced Aug 6, 2013

2to3 refactor broke test_perf.py #4474

Closed

2to3 refactor broke vbenches #4477

Closed

jreback mentioned this pull request Aug 15, 2013

0.6.0 Roadmap statsmodels/statsmodels#1019

Closed

argriffing mentioned this pull request Jan 6, 2015

For Python2+Python3 compatibility, add some from __future__ imports. cvxpy/cvxpy#164

Closed

CLN/ENH/BLD: Remove need for 2to3 for Python 3. #4384

CLN/ENH/BLD: Remove need for 2to3 for Python 3. #4384

Conversation

jtratner commented Jul 27, 2013

cpcloud commented Jul 27, 2013

cpcloud commented Jul 27, 2013

jtratner commented Jul 27, 2013

jtratner commented Jul 27, 2013

cpcloud commented Jul 27, 2013

jtratner commented Jul 27, 2013

cpcloud commented Jul 27, 2013

cpcloud commented Jul 27, 2013

jtratner commented Jul 27, 2013

cpcloud commented Jul 27, 2013

jtratner commented Jul 27, 2013

cpcloud commented Jul 27, 2013

jtratner commented Jul 27, 2013

jtratner commented Jul 27, 2013

cpcloud commented Jul 28, 2013

jtratner commented Jul 28, 2013

cpcloud commented Jul 28, 2013

jtratner commented Jul 28, 2013

jtratner commented Jul 28, 2013

cpcloud commented Jul 28, 2013

jtratner commented Jul 28, 2013

cpcloud commented Jul 28, 2013

cpcloud commented Jul 28, 2013

jtratner commented Jul 28, 2013

cpcloud commented Jul 28, 2013

cpcloud commented Jul 28, 2013

jtratner commented Jul 28, 2013

cpcloud commented Jul 28, 2013

cpcloud commented Jul 28, 2013

cpcloud commented Jul 28, 2013

cpcloud commented Jul 30, 2013

jreback commented Jul 30, 2013

jtratner commented Jul 30, 2013

FAIL: test_iterkv_deprecation (pandas.tests.test_frame.TestDataFrame)

jtratner commented Jul 30, 2013

jtratner commented Jul 30, 2013

jreback commented Jul 30, 2013

jtratner commented Jul 30, 2013

jreback commented Jul 30, 2013

jtratner commented Jul 30, 2013

jreback commented Jul 30, 2013

jtratner commented Jul 30, 2013

jreback commented Jul 30, 2013

jtratner commented Jul 30, 2013

jtratner commented Jul 30, 2013

jtratner commented Jul 30, 2013

jtratner commented Jul 30, 2013

cpcloud commented Jul 30, 2013

jreback commented Jul 30, 2013

jtratner commented Jul 30, 2013

jreback commented Jul 30, 2013

jtratner commented Jul 30, 2013

jreback commented Jul 30, 2013

cpcloud commented Jul 30, 2013

jreback commented Aug 5, 2013

jtratner commented Aug 6, 2013