A couple of thoughts, mainly about dtypes.

• the extension dtypes in main types are parameterized, which is quite a good thing. This would require some factory functions to generate these at run-time.
  • Categorical really should be parameterized with the actual categories, then the impl of a Categorical is quite easy. (and we dont have to do the is_dtype_equal thingy all over the place)
  • datetime w/tz & timedelta similarly (if we want to support other units)
  • strings can also use this for having an encoding
  • ideal these will be sub-class of the base dtype
• how do these dtypes interact with Python/numpy dtypes? e.g. comparisons / convertibility
• show these be actual types (eg. inherit from PyTypeObject)?
• how 'separable' is this dtype meant to be? e.g. in theory this should be a separate library that say numpy/python could actually build upon (or maybe just is just a working impl that someone else can build on).

OT: what is your editor / debug setup for c++?

Darwin 069-jreback.local 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64

[jreback-~/pandas] python setup.py build_ext --inplace
running build_ext
cmake  -DPYTHON_EXECUTABLE=/Users/jreback/miniconda/bin/python   /Users/jreback/pandas
-- Toolchain build.
-- Downloading and extracting dependencies.
CMake Error at cmake_modules/toolchain.cmake:42 (message):
  Toolchain bootstrap failed.
Call Stack (most recent call first):
  CMakeLists.txt:24 (include)


-- Configuring incomplete, errors occurred!
error: command 'cmake' failed with exit status 1

[jreback-~/pandas] cmake --version
cmake version 3.3.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).

what am I missing?

Sorry, I haven't begun to address the build toolchain problem, so for the moment I've been developing using the portable C++ toolchain we're using for several projects being developed at Cloudera https://github.com/cloudera/native-toolchain .

I'm not sure what to do here. We'll want to make it easy to add new C++ libraries that dynamically link to libpandas, but they'll need to get installed on the user's system somehow. conda? homebrew? What about Windows?

Let me come up with some instructions that will work for you without necessarily requiring you to install the native-toolchain I linked above (though that may be the easiest thing).

I need to get Travis CI passing anyway

to your other comments

how do these dtypes interact with Python/numpy dtypes? e.g. comparisons / convertibility
show these be actual types (eg. inherit from PyTypeObject)?

From pandas's point of view, there will only be pandas::DataType instances. If you want to interact with some other data, it will need to be properly marshalled / wrapped to a pandas data type and object in libpandas. So comparison will happen after mapping NumPy dtypes to pandas types (which is already implicitly happening, but because of the fragmentation between NumPy and non-NumPy types it's a bit messy).

how 'separable' is this dtype meant to be? e.g. in theory this should be a separate library that say
numpy/python could actually build upon (or maybe just is just a working impl that someone else
can build on).

At least at the outset not very separable (making it so would be a lot of work with little benefit), but with some engineering effort the code could be made reusable.

show these be actual types (eg. inherit from PyTypeObject)?

No — I think it's better to have a clean and flexible C++ API without the burdens of Python's C API object construction protocol and only wrap the objects in Python types when the user is interacting with them.

@jreback I removed some of the currently-unneeded google thirdparty libraries. I'll add a thirdparty directory with googletest and get that building so Travis CI will work and we don't need to rely on an external toolchain for the moment. I'd like to add libraries like RE2, snappy, and lz4 to libpandas so we will need to figure out a dependency-hell avoidance strategy

Phew OK I got the build working on OS X. Please let me know if

python setup.py build_ext --inplace

does not work for you so I can investigate further. I'll fix up the Travis CI build when I can and set it to only test the native core stuff for now

ok, build procedure works for me (on osx)! (and your example above).

[jreback-~/pandas] nosetests pandas/internals/tests/ -v
test_convert_numpy_int_dtypes (pandas.internals.tests.test_proto.TestLibPandas) ... ok
test_convert_primitive_numpy_arrays_basics (pandas.internals.tests.test_proto.TestLibPandas) ... ok
test_integer_get_set_unset_na (pandas.internals.tests.test_proto.TestLibPandas) ... ok
test_isnull_natype_interop (pandas.internals.tests.test_proto.TestLibPandas) ... ok
test_libpandas_decrefs (pandas.internals.tests.test_proto.TestLibPandas) ... ok

----------------------------------------------------------------------
Ran 5 tests in 0.018s

OK

where do you want to put a list of API wishlist things for these structures?

e.g.

isnull on the Array, and a cached-property on ImmutableArray
is_monotonoic_increasing/is_monotonic_decreasing as well

I just started things on #11970

I'll start making the (long) list in a Google document that I'll link on that issue. I'll see about getting Travis passing on this branch and merge this into the integration branch.

If you start hacking on this with me, this may be useful (also for my own reference since I make plenty of mistakes): https://google.github.io/styleguide/cppguide.html

@wesm gr8. mainly acting as a 'user'/feature requestor for the moment

OMG the build passed. @jreback have any idea why I had to twiddle $APT_ARGS to get Travis to work?

I'm going to squash and merge this into the integration branch and move onto the next experiments later this week

I guess that makes the confirmation go away on the apt-get installs

we don't really use the included stuff too much except for very basics and use default compiler

Gotcha. I just squashed and merged this PR into libpandas-native-core branch

See afc53f0