VIS/ENH Hexbin plot

[TomAugspurger]

This is just 10 minutes of copy-paste cargo-culting to gauge interest, I haven't tested anything yet.

In [1]: df = pd.DataFrame(np.random.randn(1000, 2))

In [2]: df.plot(kind='hexbin', x=0, y=1)
Out[2]: <matplotlib.axes.AxesSubplot at 0x10eb76e10>

It's not terribly difficult to do this on your own, so my feeling wouldn't be hurt at all if people are -1 on this :)

EDIT: oops I branched from my to_frame branch. I'll clean up the commits.

[jtratner]

looks interesting

[TomAugspurger]

Colorbar on by default?

Also, apparently the default matplotlib colormap, jet, is widely despised. I'm looking around for a better default.

[TomAugspurger]

Thoughts on a default color pallet? Seaborn seems to use cubehelix (screenshot)

This may be helpful. @olgabot seems to use different cmaps for different data values. That may be a bit too much for pandas.

EDIT: oh I also think this should spend some time incubating on master. Once the RC branch is forked, I'll add some release notes for .14;

[TomAugspurger]

Oh, also, I hate the matplotlib argument names for the value (C) and reduction function reduce_C_function.

A brief overview:

by default (C=None), compute a histogram in each bin. Essentially each (x, y) point in a bin has a value of 1, and the reduction function is sum (i.e. count since every value is 1.)

If you specify C (another column of df for pandas), then each point is a 3 tuple (x, y, c): (x, y) still specify what bin you end up in, and c determines the value passed to the reduction function reduce_C_function

What's the policy on respecting other libraries' function arguments. Other pandas plotting functions seem to use colormap instead of matplotlib's cmap.

If we're willing to change, I'd suggest renaming C to value and reduce_C_function to reduce or reduce_func

Here's an example to play with if that helps:

df = DataFrame({"A": np.random.uniform(size=20),
                "B": np.random.uniform(size=20),
                "C": np.arange(20) + np.random.uniform(size=20)})

ax = df.plot(kind='hexbin', x='A', y='B', gridsize=10)  # histogram by default
ax = df.plot(kind='hexbin', x='A', y='B', C='C', reduce_C_function=np.max)

So in the second case the color of a bin will be determined by the maximum value of C (from column C in df) in that bin.

[olgabot]

While cubehelix solves the saturation problem, it doesn't solve the issue that the rainbow colormap is harmful and does not help with interpretation of the heatmap. The linked paper gives great examples how the changing colors introduce patterns in the data that don't actually exist and can lead to over-interpretation of color changes.

I don't think using different colormaps for different data ranges is that complicated, in prettyplotlib it's a simple check of the data's max and min and using a divergent heatmap like blue-red if there's negative and positive values, or a sequential heatmap if there's only positive or negative values. These colormaps are built into matplotlib

I'm still working on my matplotlib install. I have some code for a clustered heatmap as well (like heatmap.2 in R) that would use but it depends on scipy, so I'm guessing I'd have to implement hierarchical clustering myself to reduce pandas's dependencies. Plus as a data science/bioinformatics purist I'd want it to have optimal leaf ordering, too. This is the default in MATLAB, but not in R or scipy.

As for renaming, I agree that the matplotlib arguments are cryptic but I personally prefer consistency over introducing new standards: http://xkcd.com/927/

[mwaskom]

I set the matplotlib default colormap to cubehelix in seaborn because I wanted to banish jet and that seemed like a reasonably non-crappy alternative that is at least somewhat adaquate for most data. I don't think the hue shifts are as big a problem in cubehelix as in rainbow maps and jet because they're pretty gradual and accompanied by a shift in lightness/saturation.

But in general I agree that there's no good one-size-fits-all solution to colormaps, and it's better to adapt to the data. For the corrplot function the default colormap is coolwarm, which I tend to prefer a bit to RdBu as the extreme values aren't quite as dark.

[TomAugspurger]

Thanks for the feedback! I'll think on this, but adapting to the data would be complicated by the business with C and reduce_C_function. That's all handled on the matplotlib side, so I won't know the range of the values when I hand everything off to matplotlib's hexbin.

The default behavior is to just do counts, so things will always be positive in that case. It might make sense just to pick the default with that case in mind, and assume that people doing fancier things can choose an appropriate color palette.

[TomAugspurger]

@olgabot scipy is a soft dependency for pandas, so that should be OK. I think the Gaussian KDE plot also depends on scipy.

[mwaskom]

Ah, yeah, if the data are counts then it probably makes sense to use a sequential map; any of the colorbrewer ones should work.

[TomAugspurger]

I think this is ready to go. Thanks for the suggestions on the colormaps. I'm going with BuGr as the default since the default aggregation is by counts, which will always be nonnegative. If you're using a different aggregation then you should choose the appropriate colormap.

One thing I'm a tad concerned about is the docstring on df.plot. I thew in a Notes with things specific to hexbin. I wonder if we should put explanations in a separate function like scipy.integrate.quad_explain() does. Or I could remove that and just point people to the html docs.

Let me know when your ready and I'll rebase and squash.

[jreback]

@TomAugspurger you want to throw in 0.13?

pls change the release notes, rebase, and squash...

[jreback]

if you think the API might change, then just label it experimental (if you want)

[jreback]

does this have an associated issue?

[TomAugspurger]

No issue associated. I just put the PR number in the release notes. I'm pretty confident the API won't need adjusting, but I guess labeling it experimental is the safe thing to do.

Ready when you and Travis say it's good!

[jreback]

perfect

@cpcloud @jtratner @y-p any comments?

[ghost]

This looks nice.

@TomAugspurger, do you know if binhex plots are available via yhat/ggplot ?
If not, it'd be great to help them grow the library, they're building it on top of pandas already.

[jreback]

looks ready to merge...any objections? @TomAugspurger @y-p @jtratner ?

[ghost]

Merging new features between RC and final?

[jreback]

seems non-invasive to me

[TomAugspurger]

I'm not in a rush to get this in if that's a problem.

[jreback]

@TomAugspurger push this to 0.14, @y-p?

[TomAugspurger]

Fine by me. Once .13 is out I'll make the doc changes and ping you when I get that done (could be a little while).

[ghost]

Yup, and to stay on track we should be more aggressive about tagging 0.13 final when it's time
and moving on to 0.14, release channels can be async to that. Just a headsup to pandas-dev
when bug queue is empty and green across platforms.

[ghost]

@TomAugspurger, so is this plot available out of the box with seaborn which is built on top of pandas?

[olgabot]

Not yet:
http://stanford.edu/~mwaskom/software/seaborn/plotting_distributions.html

---

Olga Botvinnik
PhD Program in Bioinformatics and Systems Biology
Gene Yeo Laboratory | Sanford Consortium for Regenerative Medicine
University of California, San Diego
olgabotvinnik.com
blog.olgabotvinnik.com
github.com/olgabot

On Mon, Dec 16, 2013 at 10:38 AM, y-p [email protected] wrote:

@TomAugspurger https://github.com/TomAugspurger, so is this plot
available out of the box with seaborn which is built on top of pandas?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/5478#issuecomment-30687094
.

[ghost]

I resisted other PRs for more sophisticated plots but this is closer to home, should be ok.

[TomAugspurger]

I've been a tad worried about inconsistency in what vis PRs are accepted as well, so I haven't wanted to push to get this in. That said I think this should go in since I see hexbin plots as a drop-in replacement for scatter plots when you have a bunch of points.

Plus Wes tweeted about it, and we can't make him a liar...

[jreback]

hahah.!

ok....

pls rebase and put in release notes (0.13.1)..thxs

[jreback]

moved to 0.13.1 section

[jreback]

Can you move this to 0.14.0 section?

[jreback]

move to v0.13.1.txt

[jreback]

maybe use the docs example here (its kind of cool) (and possibly put a link to the doc section as well)

[ghost]

I still think 0.14.0... don't make me out to be a softie.

[TomAugspurger]

I threw everything in .14, so this should be good to sit until we start on that.

I'm having trouble building the docs right now (on master too, not just this branch). I'll try to track down what's wrong with my environment and then I'll let you know that everything builds correctly.

[TomAugspurger]

ping @jreback if you're ready to merge.

[jreback]

@TomAugspurger trivial release notes change..then good 2 go

[TomAugspurger]

@jreback Fixed that. I removed the experimental tag too.

[jreback]

gr8 thanks @TomAugspurger !