{{ header }}
This is a short introduction to pandas, geared mainly for new users. You can see more complex recipes in the :ref:`Cookbook<cookbook>`.
Customarily, we import as follows:
.. ipython:: python import numpy as np import pandas as pd
Pandas provides two types of classes for handling data:
- :class:`Series`: a one-dimensional labeled array holding data of any type
- such as integers, strings, Python objects etc.
- :class:`DataFrame`: a two-dimensional data structure that holds data like a two-dimension array or a table with rows and columns.
See the :ref:`Intro to data structures section <dsintro>`.
Creating a :class:`Series` by passing a list of values, letting pandas create a default :class:`RangeIndex`.
.. ipython:: python s = pd.Series([1, 3, 5, np.nan, 6, 8]) s
Creating a :class:`DataFrame` by passing a NumPy array with a datetime index using :func:`date_range` and labeled columns:
.. ipython:: python
dates = pd.date_range("20130101", periods=6)
dates
df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list("ABCD"))
df
Creating a :class:`DataFrame` by passing a dictionary of objects where the keys are the column labels and the values are the column values.
.. ipython:: python
df2 = pd.DataFrame(
{
"A": 1.0,
"B": pd.Timestamp("20130102"),
"C": pd.Series(1, index=list(range(4)), dtype="float32"),
"D": np.array([3] * 4, dtype="int32"),
"E": pd.Categorical(["test", "train", "test", "train"]),
"F": "foo",
}
)
df2
The columns of the resulting :class:`DataFrame` have different :ref:`dtypes <basics.dtypes>`:
.. ipython:: python df2.dtypes
If you're using IPython, tab completion for column names (as well as public attributes) is automatically enabled. Here's a subset of the attributes that will be completed:
.. ipython:: @verbatim In [1]: df2.<TAB> # noqa: E225, E999 df2.A df2.bool df2.abs df2.boxplot df2.add df2.C df2.add_prefix df2.clip df2.add_suffix df2.columns df2.align df2.copy df2.all df2.count df2.any df2.combine df2.append df2.D df2.apply df2.describe df2.applymap df2.diff df2.B df2.duplicated
As you can see, the columns A, B, C, and D are automatically
tab completed. E and F are there as well; the rest of the attributes have been
truncated for brevity.
See the :ref:`Essentially basics functionality section <basics>`.
Use :meth:`DataFrame.head` and :meth:`DataFrame.tail` to view the top and bottom rows of the frame respectively:
.. ipython:: python df.head() df.tail(3)
Display the :attr:`DataFrame.index` or :attr:`DataFrame.columns`:
.. ipython:: python df.index df.columns
Return a NumPy representation of the underlying data with :meth:`DataFrame.to_numpy` without the index or column labels:
.. ipython:: python df.to_numpy()
Note
NumPy arrays have one dtype for the entire array while pandas DataFrames
have one dtype per column. When you call :meth:`DataFrame.to_numpy`, pandas will
find the NumPy dtype that can hold all of the dtypes in the DataFrame.
If the common data type is object, :meth:`DataFrame.to_numpy` will require
copying data.
.. ipython:: python df2.dtypes df2.to_numpy()
:func:`~DataFrame.describe` shows a quick statistic summary of your data:
.. ipython:: python df.describe()
Transposing your data:
.. ipython:: python df.T
:meth:`DataFrame.sort_index` sorts by an axis:
.. ipython:: python df.sort_index(axis=1, ascending=False)
:meth:`DataFrame.sort_values` sorts by values:
.. ipython:: python df.sort_values(by="B")
Note
While standard Python / NumPy expressions for selecting and setting are intuitive and come in handy for interactive work, for production code, we recommend the optimized pandas data access methods, :meth:`DataFrame.at`, :meth:`DataFrame.iat`, :meth:`DataFrame.loc` and :meth:`DataFrame.iloc`.
See the indexing documentation :ref:`Indexing and Selecting Data <indexing>` and :ref:`MultiIndex / Advanced Indexing <advanced>`.
For a :class:`DataFrame`, passing a single label selects a columns and
yields a :class:`Series` equivalent to df.A:
.. ipython:: python df["A"]
For a :class:`DataFrame`, passing a slice : selects matching rows:
.. ipython:: python df[0:3] df["20130102":"20130104"]
See more in :ref:`Selection by Label <indexing.label>` using :meth:`DataFrame.loc` or :meth:`DataFrame.at`.
Selecting a row matching a label:
.. ipython:: python df.loc[dates[0]]
Selecting all rows (:) with a select column labels:
.. ipython:: python df.loc[:, ["A", "B"]]
For label slicing, both endpoints are included:
.. ipython:: python df.loc["20130102":"20130104", ["A", "B"]]
Selecting a single row and column label returns a scalar:
.. ipython:: python df.loc[dates[0], "A"]
For getting fast access to a scalar (equivalent to the prior method):
.. ipython:: python df.at[dates[0], "A"]
See more in :ref:`Selection by Position <indexing.integer>` using :meth:`DataFrame.iloc` or :meth:`DataFrame.iat`.
Select via the position of the passed integers:
.. ipython:: python df.iloc[3]
Integer slices acts similar to NumPy/Python:
.. ipython:: python df.iloc[3:5, 0:2]
Lists of integer position locations:
.. ipython:: python df.iloc[[1, 2, 4], [0, 2]]
For slicing rows explicitly:
.. ipython:: python df.iloc[1:3, :]
For slicing columns explicitly:
.. ipython:: python df.iloc[:, 1:3]
For getting a value explicitly:
.. ipython:: python df.iloc[1, 1]
For getting fast access to a scalar (equivalent to the prior method):
.. ipython:: python df.iat[1, 1]
Select rows where df.A is greater than 0.
.. ipython:: python df[df["A"] > 0]
Selecting values from a :class:`DataFrame` where a boolean condition is met:
.. ipython:: python df[df > 0]
Using :func:`~Series.isin` method for filtering:
.. ipython:: python df2 = df.copy() df2["E"] = ["one", "one", "two", "three", "four", "three"] df2 df2[df2["E"].isin(["two", "four"])]
Setting a new column automatically aligns the data by the indexes:
.. ipython:: python
s1 = pd.Series([1, 2, 3, 4, 5, 6], index=pd.date_range("20130102", periods=6))
s1
df["F"] = s1
Setting values by label:
.. ipython:: python df.at[dates[0], "A"] = 0
Setting values by position:
.. ipython:: python df.iat[0, 1] = 0
Setting by assigning with a NumPy array:
.. ipython:: python :okwarning: df.loc[:, "D"] = np.array([5] * len(df))
The result of the prior setting operations:
.. ipython:: python df
A where operation with setting:
.. ipython:: python df2 = df.copy() df2[df2 > 0] = -df2 df2
For NumPy data types, np.nan represents missing data. It is by
default not included in computations. See the :ref:`Missing Data section
<missing_data>`.
Reindexing allows you to change/add/delete the index on a specified axis. This returns a copy of the data:
.. ipython:: python df1 = df.reindex(index=dates[0:4], columns=list(df.columns) + ["E"]) df1.loc[dates[0] : dates[1], "E"] = 1 df1
:meth:`DataFrame.dropna` drops any rows that have missing data:
.. ipython:: python df1.dropna(how="any")
:meth:`DataFrame.fillna` fills missing data:
.. ipython:: python df1.fillna(value=5)
:func:`isna` gets the boolean mask where values are nan:
.. ipython:: python pd.isna(df1)
See the :ref:`Basic section on Binary Ops <basics.binop>`.
Operations in general exclude missing data.
Calculate the mean value for each column:
.. ipython:: python df.mean()
Calculate the mean value for each row:
.. ipython:: python df.mean(axis=1)
Operating with another :class:`Series` or :class:`DataFrame` with a different index or column
will align the result with the union of the index or column labels. In addition, pandas
automatically broadcasts along the specified dimension and will fill unaligned labels with np.nan.
.. ipython:: python s = pd.Series([1, 3, 5, np.nan, 6, 8], index=dates).shift(2) s df.sub(s, axis="index")
:meth:`DataFrame.agg` and :meth:`DataFrame.transform` applies a user defined function that reduces or broadcasts its result respectively.
.. ipython:: python df.agg(lambda x: np.mean(x) * 5.6) df.transform(lambda x: x * 101.2)
See more at :ref:`Histogramming and Discretization <basics.discretization>`.
.. ipython:: python s = pd.Series(np.random.randint(0, 7, size=10)) s s.value_counts()
:class:`Series` is equipped with a set of string processing methods in the str
attribute that make it easy to operate on each element of the array, as in the
code snippet below. See more at :ref:`Vectorized String Methods
<text.string_methods>`.
.. ipython:: python s = pd.Series(["A", "B", "C", "Aaba", "Baca", np.nan, "CABA", "dog", "cat"]) s.str.lower()
pandas provides various facilities for easily combining together :class:`Series` and :class:`DataFrame` objects with various kinds of set logic for the indexes and relational algebra functionality in the case of join / merge-type operations.
See the :ref:`Merging section <merging>`.
Concatenating pandas objects together row-wise with :func:`concat`:
.. ipython:: python df = pd.DataFrame(np.random.randn(10, 4)) df # break it into pieces pieces = [df[:3], df[3:7], df[7:]] pd.concat(pieces)
Note
Adding a column to a :class:`DataFrame` is relatively fast. However, adding a row requires a copy, and may be expensive. We recommend passing a pre-built list of records to the :class:`DataFrame` constructor instead of building a :class:`DataFrame` by iteratively appending records to it.
:func:`merge` enables SQL style join types along specific columns. See the :ref:`Database style joining <merging.join>` section.
.. ipython:: python
left = pd.DataFrame({"key": ["foo", "foo"], "lval": [1, 2]})
right = pd.DataFrame({"key": ["foo", "foo"], "rval": [4, 5]})
left
right
pd.merge(left, right, on="key")
:func:`merge` on unique keys:
.. ipython:: python
left = pd.DataFrame({"key": ["foo", "bar"], "lval": [1, 2]})
right = pd.DataFrame({"key": ["foo", "bar"], "rval": [4, 5]})
left
right
pd.merge(left, right, on="key")
By "group by" we are referring to a process involving one or more of the following steps:
- Splitting the data into groups based on some criteria
- Applying a function to each group independently
- Combining the results into a data structure
See the :ref:`Grouping section <groupby>`.
.. ipython:: python
df = pd.DataFrame(
{
"A": ["foo", "bar", "foo", "bar", "foo", "bar", "foo", "foo"],
"B": ["one", "one", "two", "three", "two", "two", "one", "three"],
"C": np.random.randn(8),
"D": np.random.randn(8),
}
)
df
Grouping by a column label, selecting column labels, and then applying the :meth:`~pandas.core.groupby.DataFrameGroupBy.sum` function to the resulting groups:
.. ipython:: python
df.groupby("A")[["C", "D"]].sum()
Grouping by multiple columns label forms :class:`MultiIndex`.
.. ipython:: python df.groupby(["A", "B"]).sum()
See the sections on :ref:`Hierarchical Indexing <advanced.hierarchical>` and :ref:`Reshaping <reshaping.stacking>`.
.. ipython:: python
arrays = [
["bar", "bar", "baz", "baz", "foo", "foo", "qux", "qux"],
["one", "two", "one", "two", "one", "two", "one", "two"],
]
index = pd.MultiIndex.from_arrays(arrays, names=["first", "second"])
df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=["A", "B"])
df2 = df[:4]
df2
The :meth:`~DataFrame.stack` method "compresses" a level in the DataFrame's columns:
.. ipython:: python stacked = df2.stack(future_stack=True) stacked
With a "stacked" DataFrame or Series (having a :class:`MultiIndex` as the
index), the inverse operation of :meth:`~DataFrame.stack` is
:meth:`~DataFrame.unstack`, which by default unstacks the last level:
.. ipython:: python stacked.unstack() stacked.unstack(1) stacked.unstack(0)
See the section on :ref:`Pivot Tables <reshaping.pivot>`.
.. ipython:: python
df = pd.DataFrame(
{
"A": ["one", "one", "two", "three"] * 3,
"B": ["A", "B", "C"] * 4,
"C": ["foo", "foo", "foo", "bar", "bar", "bar"] * 2,
"D": np.random.randn(12),
"E": np.random.randn(12),
}
)
df
:func:`pivot_table` pivots a :class:`DataFrame` specifying the values, index and columns
.. ipython:: python pd.pivot_table(df, values="D", index=["A", "B"], columns=["C"])
pandas has simple, powerful, and efficient functionality for performing resampling operations during frequency conversion (e.g., converting secondly data into 5-minutely data). This is extremely common in, but not limited to, financial applications. See the :ref:`Time Series section <timeseries>`.
.. ipython:: python
rng = pd.date_range("1/1/2012", periods=100, freq="s")
ts = pd.Series(np.random.randint(0, 500, len(rng)), index=rng)
ts.resample("5Min").sum()
:meth:`Series.tz_localize` localizes a time series to a time zone:
.. ipython:: python
rng = pd.date_range("3/6/2012 00:00", periods=5, freq="D")
ts = pd.Series(np.random.randn(len(rng)), rng)
ts
ts_utc = ts.tz_localize("UTC")
ts_utc
:meth:`Series.tz_convert` converts a timezones aware time series to another time zone:
.. ipython:: python
ts_utc.tz_convert("US/Eastern")
Adding a non-fixed duration (:class:`~pandas.tseries.offsets.BusinessDay`) to a time series:
.. ipython:: python rng rng + pd.offsets.BusinessDay(5)
pandas can include categorical data in a :class:`DataFrame`. For full docs, see the :ref:`categorical introduction <categorical>` and the :ref:`API documentation <api.arrays.categorical>`.
.. ipython:: python
df = pd.DataFrame(
{"id": [1, 2, 3, 4, 5, 6], "raw_grade": ["a", "b", "b", "a", "a", "e"]}
)
Converting the raw grades to a categorical data type:
.. ipython:: python
df["grade"] = df["raw_grade"].astype("category")
df["grade"]
Rename the categories to more meaningful names:
.. ipython:: python
new_categories = ["very good", "good", "very bad"]
df["grade"] = df["grade"].cat.rename_categories(new_categories)
Reorder the categories and simultaneously add the missing categories (methods under :meth:`Series.cat` return a new :class:`Series` by default):
.. ipython:: python
df["grade"] = df["grade"].cat.set_categories(
["very bad", "bad", "medium", "good", "very good"]
)
df["grade"]
Sorting is per order in the categories, not lexical order:
.. ipython:: python
df.sort_values(by="grade")
Grouping by a categorical column with observed=False also shows empty categories:
.. ipython:: python
df.groupby("grade", observed=False).size()
See the :ref:`Plotting <visualization>` docs.
We use the standard convention for referencing the matplotlib API:
.. ipython:: python
import matplotlib.pyplot as plt
plt.close("all")
The plt.close method is used to close a figure window:
.. ipython:: python
ts = pd.Series(np.random.randn(1000), index=pd.date_range("1/1/2000", periods=1000))
ts = ts.cumsum()
@savefig series_plot_basic.png
ts.plot();
Note
When using Jupyter, the plot will appear using :meth:`~Series.plot`. Otherwise use matplotlib.pyplot.show to show it or matplotlib.pyplot.savefig to write it to a file.
:meth:`~DataFrame.plot` plots all columns:
.. ipython:: python
df = pd.DataFrame(
np.random.randn(1000, 4), index=ts.index, columns=["A", "B", "C", "D"]
)
df = df.cumsum()
plt.figure();
df.plot();
@savefig frame_plot_basic.png
plt.legend(loc='best');
See the :ref:`IO Tools <io>` section.
:ref:`Writing to a csv file: <io.store_in_csv>` using :meth:`DataFrame.to_csv`
.. ipython:: python
df = pd.DataFrame(np.random.randint(0, 5, (10, 5)))
df.to_csv("foo.csv")
:ref:`Reading from a csv file: <io.read_csv_table>` using :func:`read_csv`
.. ipython:: python
pd.read_csv("foo.csv")
.. ipython:: python
:suppress:
import os
os.remove("foo.csv")
Writing to a Parquet file:
.. ipython:: python
:okwarning:
df.to_parquet("foo.parquet")
Reading from a Parquet file Store using :func:`read_parquet`:
.. ipython:: python
:okwarning:
pd.read_parquet("foo.parquet")
.. ipython:: python
:suppress:
os.remove("foo.parquet")
Reading and writing to :ref:`Excel <io.excel>`.
Writing to an excel file using :meth:`DataFrame.to_excel`:
.. ipython:: python
df.to_excel("foo.xlsx", sheet_name="Sheet1")
Reading from an excel file using :func:`read_excel`:
.. ipython:: python
pd.read_excel("foo.xlsx", "Sheet1", index_col=None, na_values=["NA"])
.. ipython:: python
:suppress:
os.remove("foo.xlsx")
If you are attempting to perform a boolean operation on a :class:`Series` or :class:`DataFrame` you might see an exception like:
.. ipython:: python
:okexcept:
if pd.Series([False, True, False]):
print("I was true")
See :ref:`Comparisons<basics.compare>` and :ref:`Gotchas<gotchas>` for an explanation and what to do.