Basic JSON normalization/flattening

[wesm]

closes #1067.

This isn't a fast function but has quite a bit of functionality. Take a look at the unit tests. Comments on the API welcome and this could use a few iterations. Various ad hoc things like concatenating "key paths" to metadata

[wesm]

@nehalecky can i get some feedback?

[nehalecky]

@wesm, hey! Yes, I will roll this out to a direct use case I have and let you know. I was hoping to do so today, so I'll get back soon. ty.

[nehalecky]

Really busy yesterday, I am going to try and get to this today. :)

[jreback]

@nehalecky did you get a chance to try out?

we adding this to 0.12?

[nehalecky]

Hey @jreback. I am finishing one last deadline (due today). My very next project will benefit much from this functionality, so I will be testing it out. I have a deadline for this early next week, and will certainly have reviewed this by then. If that works, great and either way I will let you know my thoughts. :) THANKS!

[cpcloud]

maybe push to 0.13? @wesm?

[wesm]

yes let's push to 0.13 so it can see some work

[ghost]

Warning: cute.

def jqj(data,rule):
     from sh import jq
     from sh import ErrorReturnCode
     import json   
     jq = jq.bake('-M')
     try:
         return json.loads(jq(rule,_in=json.dumps(data)).stdout)
     except ErrorReturnCode as e:
         raise RuntimeError("Jq exited with an error\n%s" % str(e))
     data = dict(dwarves={'names':['dancer','prancer','grinch']},tale='rambo II')
     rule = '[{name: .dwarves.names[],tale}]'
     jqj(data,rule)

[{u'name': u'dancer', u'tale': u'rambo II'},
 {u'name': u'prancer', u'tale': u'rambo II'},
 {u'name': u'grinch', u'tale': u'rambo II'}]

jq

[cpcloud]

That's pretty nifty

[jreback]

@nehalecky @cpcloud @wesm try to merge to 0.13?

[jreback]

@nehalecky have a look?

[jreback]

@wesm, merge this?

[wesm]

Sure, it's good to go by me.

[jreback]

ok..let me rebase....and can do

[nehalecky]

Sorry all, life + work is all really busy. In startup's requirements change quickly! Regarding the PR, glad to see it's been merged and included in 0.13. I'm eager to work with it, and when I do, I will provide feedback.

Thank for all your efforts!

[jreback]

@nehalecky gr8...try it out....it may not be super fast....(optimization PR's always welcome)...b.ut should be good on moderately sized sets

[cpcloud]

xref: http://stackoverflow.com/questions/21494030/create-a-pandas-dataframe-from-deeply-nested-json

@y-p thanks for that snippet above

[ghost]

I've looked at this again recently for other reasons and found plenty of existing,
dedicated languages for doing this. Basically, XML has XSLT for doing transformations
and lots of people came up with the idea of doing the same for JSON.

If you're interested google: jaql, jsonpath, jsoniq or jsont.

Appreciate your care for attribution.