ENH: allow saving of meta-data via CArrays to support wide tables #11788
Conversation
Wide table
And, make sure that non_index_axes is preserved if it was written by legacy code.
| @@ -4606,6 +4606,22 @@ def test_read_nokey(self): | |||
| df.to_hdf(path, 'df2', mode='a') | |||
| self.assertRaises(ValueError, read_hdf, path) | |||
|
|
|||
| def test_legacy_non_index_axes(self): | |||
| filename = tm.get_data_path('legacy_hdf/legacy_non_index_axes_0.17.1.h5') | |||
| with HDFStore(filename, 'r') as store: | |||
|
so, seeing that we are now generating legacy files for hdf5, I think its time to expand this generator to do it; you can see how we do this for msgpack/pickle. need to expand this to write several types of files for hdf5 (e.g. Series,DataFrame,Panel), with varying datatypes (you can prob use a lot of what is there now). Then we just import this to compare, makes creation of legacy files real easy. |
jreback
changed the title
|
@MJuddBooth can you update |
|
Yes, I re-factored a bit before the holidays and all the tests looked good but year-end work stopped me before I could finish and push. I can get back to it today or tomorrow. |
|
closing, but pls reopen if you'd like to finish this up. Its a nice change. |
|
@MJuddBooth Any chance you would be able to finish this up? I know it has been two years... |
|
Oh my. I need to upgrade my fork to 0.23 so I will try to revisit this week. |
|
Yay 🍾 ! Thanks a lot! I am happy to help by the way if there is something delegable. |
|
@MJuddBooth I'm quite interested in this feature being rolled out and it seems you're already pretty far into the implementation. Do you have any updates on it? Please let me know if I can help in any way! |
|
Yeah, seems it's dead again @MJuddBooth ? |
supersedes #10243
closes #6245
Here is the code as it stands now. Appending MultiIndex axes now works. The one test that still fails is test_append_frame_column_oriented: writing the DateTimeIndex as columns fails because the index is converted to an array of timestamps so np.array creates an object array. If it could be passed down unmolested np.array() would do the right thing. I don't yet have a good idea how to fix that.
I added a test for reading the old format, let me know if you think it's sufficient coverage. I've been testing the reading more broadly against my own data.