.. currentmodule:: pandas
.. ipython:: python :suppress: import numpy as np import random import os np.random.seed(123456) from pandas import * options.display.max_rows=15 import pandas as pd randn = np.random.randn randint = np.random.randint np.set_printoptions(precision=4, suppress=True)
This is a repository for short and sweet examples and links for useful pandas recipes. We encourage users to add to this documentation.
This is a great First Pull Request (to add interesting links and/or put short code inline for existing links)
These are some neat pandas idioms
How to split a frame with a boolean criterion?
How to select from a frame with complex criteria?
Select rows closest to a user-defined number
How to reduce a sequence (e.g. of Series) using a binary operator
The :ref:`indexing <indexing>` docs.
Indexing using both row labels and conditionals, see here
Use loc for label-oriented slicing and iloc positional slicing, see here
Extend a panel frame by transposing, adding a new dimension, and transposing back to the original dimensions, see here
Mask a panel by using np.where and then reconstructing the panel with the new masked values
here
Using ~ to take the complement of a boolean array, see
here
Efficiently creating columns using applymap
The :ref:`multindexing <indexing.hierarchical>` docs.
Creating a multi-index from a labeled frame
Performing arithmetic with a multi-index that needs broadcasting
Slicing a multi-index with xs #2
Setting portions of a multi-index with xs
Partial Selection, the need for sortedness
Prepending a level to a multiindex
The :ref:`panelnd<dsintro.panelnd>` docs.
The :ref:`missing data<missing_data>` docs.
Fill forward a reversed timeseries
.. ipython:: python
df = pd.DataFrame(np.random.randn(6,1), index=pd.date_range('2013-08-01', periods=6, freq='B'), columns=list('A'))
df.ix[3,'A'] = np.nan
df
df.reindex(df.index[::-1]).ffill()
The :ref:`grouping <groupby>` docs.
Apply to different items in a group
Replacing values with groupby means
Sort by group with aggregation
Create multiple aggregated columns
Create a value counts column and reassign back to the DataFrame
Rolling Computation window based on values instead of counts
The :ref:`Pivot <reshaping.pivot>` docs.
Frequency table like plyr in R
Turning embedded lists into a multi-index frame
Rolling apply with a DataFrame returning a Series
Rolling apply with a DataFrame returning a Scalar
Turn a matrix with hours in columns and days in rows into a continuous row sequence in the form of a time series. How to rearrange a python pandas DataFrame?
Dealing with duplicates when reindexing a timeseries to a specified frequency
The :ref:`Resample <timeseries.resampling>` docs.
TimeGrouping of values grouped across time
Using TimeGrouper and another grouping to create subgroups, then apply a custom function
Resampling with custom periods
Resample intraday frame without adding new days
The :ref:`Concat <merging.concatenation>` docs. The :ref:`Join <merging.join>` docs.
Join with a criteria based on the values
The :ref:`Plotting <visualization>` docs.
Setting x-axis major and minor labels
Plotting multiple charts in an ipython notebook
Annotate a time-series plot #2
Generate Embedded plots in excel files using Pandas, Vincent and xlsxwriter
Performance comparison of SQL vs HDF5
The :ref:`CSV <io.read_csv_table>` docs
Reading only certain rows of a csv chunk-by-chunk
Reading the first few lines of a frame
Reading a file that is compressed but not by gzip/bz2 (the native compressed formats which read_csv understands).
This example shows a WinZipped file, but is a general application of opening the file within a context manager and
using that handle to read.
See here
Reading CSV with Unix timestamps and converting to local timezone
Write a multi-row index CSV without writing duplicates
The :ref:`SQL <io.sql>` docs
Reading from databases with SQL
The :ref:`Excel <io.excel>` docs
Reading from a filelike handle
Reading HTML tables from a server that cannot handle the default request header
The :ref:`HDFStores <io.hdf5>` docs
Simple Queries with a Timestamp Index
Managing heterogeneous data using a linked multiple table hierarchy
Merging on-disk tables with millions of rows
Deduplicating a large store by chunks, essentially a recursive reduction operation. Shows a function for taking in data from csv file and creating a store by chunks, with date parsing as well. See here
Creating a store chunk-by-chunk from a csv file
Appending to a store, while creating a unique index
Reading in a sequence of files, then providing a global unique index to a store while appending
Troubleshoot HDFStore exceptions
Setting min_itemsize with strings
Using ptrepack to create a completely-sorted-index on a store
Storing Attributes to a group node
.. ipython:: python
df = DataFrame(np.random.randn(8,3))
store = HDFStore('test.h5')
store.put('df',df)
# you can store an arbitrary python object via pickle
store.get_storer('df').attrs.my_attribute = dict(A = 10)
store.get_storer('df').attrs.my_attribute
.. ipython:: python
:suppress:
store.close()
os.remove('test.h5')
Pandas readily accepts numpy record arrays, if you need to read in a binary
file consisting of an array of C structs. For example, given this C program
in a file called main.c compiled with gcc main.c -std=gnu99 on a
64-bit machine,
#include <stdio.h>
#include <stdint.h>
typedef struct _Data
{
int32_t count;
double avg;
float scale;
} Data;
int main(int argc, const char *argv[])
{
size_t n = 10;
Data d[n];
for (int i = 0; i < n; ++i)
{
d[i].count = i;
d[i].avg = i + 1.0;
d[i].scale = (float) i + 2.0f;
}
FILE *file = fopen("binary.dat", "wb");
fwrite(&d, sizeof(Data), n, file);
fclose(file);
return 0;
}the following Python code will read the binary file 'binary.dat' into a
pandas DataFrame, where each element of the struct corresponds to a column
in the frame:
import numpy as np
from pandas import DataFrame
names = 'count', 'avg', 'scale'
# note that the offsets are larger than the size of the type because of
# struct padding
offsets = 0, 8, 16
formats = 'i4', 'f8', 'f4'
dt = np.dtype({'names': names, 'offsets': offsets, 'formats': formats},
align=True)
df = DataFrame(np.fromfile('binary.dat', dt))Note
The offsets of the structure elements may be different depending on the architecture of the machine on which the file was created. Using a raw binary file format like this for general data storage is not recommended, as it is not cross platform. We recommended either HDF5 or msgpack, both of which are supported by pandas' IO facilities.
Numerical integration (sample-based) of a time series
The :ref:`Timedeltas <timeseries.timedeltas>` docs.
Create timedeltas with date differences
Adding days to dates in a dataframe
To globally provide aliases for axis names, one can define these 2 functions:
.. ipython:: python
def set_axis_alias(cls, axis, alias):
if axis not in cls._AXIS_NUMBERS:
raise Exception("invalid axis [%s] for alias [%s]" % (axis, alias))
cls._AXIS_ALIASES[alias] = axis
def clear_axis_alias(cls, axis, alias):
if axis not in cls._AXIS_NUMBERS:
raise Exception("invalid axis [%s] for alias [%s]" % (axis, alias))
cls._AXIS_ALIASES.pop(alias,None)
set_axis_alias(DataFrame,'columns', 'myaxis2')
df2 = DataFrame(randn(3,2),columns=['c1','c2'],index=['i1','i2','i3'])
df2.sum(axis='myaxis2')
clear_axis_alias(DataFrame,'columns', 'myaxis2')