(01) Intro to Python/(21) Intro to Pandas (5) - Accessing and Changing Specific Observations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to Pandas (5) - Accessing and Changing Specific Observations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the last lesson we saw how to rename and drop columns, and to set the index in a DataFrame.\n",
    "\n",
    "In this lesson we'll learn about positional and label-based selection and how to use this to make changes to specific observations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Module Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#plotly.offline doesn't push your charts to the clouds\n",
    "import plotly.offline as pyo\n",
    "#allows us to create the Data and Figure objects\n",
    "from plotly.graph_objs import *\n",
    "#plotly.plotly pushes your charts to the cloud  \n",
    "import plotly.plotly as py\n",
    "\n",
    "#pandas is a data analysis library\n",
    "import pandas as pd\n",
    "from pandas import DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Positional and Label-based Selection\n",
    "\n",
    "First of all, I'm going to read some data into a DataFrame. I also need to make another DataFrame which has a label-based index rather than a positional index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseRateData = pd.read_csv(\"http://www.richard-muir.com/data/public/csv/BoEBaseRate.csv\")\n",
    "\n",
    "baseRateData_r = baseRateData.rename(columns = {'VALUE' : 'Value', 'DATE' : 'Date'})\n",
    "baseRateData_r.set_index(baseRateData_r['Date'], inplace=True)\n",
    "baseRateData_r.drop(['Date'], axis = 1, inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at these DataFrames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseRateData.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseRateData_r.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selecting observations in a DataFrame\n",
    "\n",
    "We can select observations from a DataFrame by using <code>df.loc</code> and <code>df.iloc</code>.\n",
    "\n",
    "- <code>df.loc</code> selects the observations by their index label\n",
    "- <code>df.iloc</code> selects the observations by their position\n",
    "\n",
    "Here, I'm using <code>df.loc</code> to select the first 10 rows from <code>baseRateData</code>. Note that <code>df.loc</code> doesn't work like a list slice in Python; rather than stopping before the specified number, we include that observation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseRateData.loc[:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If I try to use <code>df.loc</code> on <code>baseRateData_r</code>, this won't work because we have changed the index label:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseRateData_r.loc[:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead I have to pass the row index label which I want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseRateData_r.loc[:'15/01/1975']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "But <code>df.iloc</code> works the same on both DataFrames because in <code>baseRateData</code>, the index is equal to the position - <code>df.iloc</code> works on the ordinal position of the rows.\n",
    "\n",
    "Confusingly, <code>df.iloc</code> works in the same way as list and string slicing, stopping just before the specified position:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseRateData.iloc[:9]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseRateData_r.iloc[:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For both <code>df.loc</code> and <code>df.iloc</code>, we can take a slice from the middle of the DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseRateData_r.loc['06/01/1975':'13/01/1975']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "baseRateData.iloc[4:6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also combine the column names with <code>df.loc</code> and <code>df.iloc</code> to get 2D slices of a DataFrame.\n",
    "\n",
    "Remember that <code>df.loc</code> works on the labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseRateData.loc[5:13, 'DATE']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But <code>df.iloc</code> operates on the index; the columns are numerically indexed (in the same way as the rows):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "baseRateData.iloc[5:13, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changing Data in a DataFrame\n",
    "\n",
    "So now we can select individual rows and columns in a DataFrame by the index label or position. We can use this knowledge to make changes to specific observations within the DataFrame.\n",
    "\n",
    "Imagine that we were told that the first twenty rows of our data were incorrect; they should have been 1.15 instead of 11.5. Let's make some changes!\n",
    "\n",
    "First of all, I'm using <code>df.loc</code> to select the first 20 rows by label and only the 'VALUE' column. It's just a simple matter of setting the value which we want these observations to take:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseRateData.loc[:19, 'VALUE'] = 1.15\n",
    "\n",
    "baseRateData.head(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also do it with <code>df.iloc</code>. Remember that the slicing is slightly different...\n",
    "\n",
    "I'll change it instead to 2.15 so we can prove it works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "baseRateData.iloc[:20, 0] = 2.15\n",
    "\n",
    "baseRateData.head(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What have we learnt this lesson?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lesson we've seen how to access rows and columns by their label and position, and to use this positional selection to make changes to the data in the DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have any questions, please ask in the comments section or email <a href=\"mailto:me@richard-muir.com\">me@richard-muir.com</a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}