lessons/01_preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d3e7ea21-6437-48e8-a9e4-3bdc05f709c9",
   "metadata": {},
   "source": [
    "# Python Text Analysis: Preprocessing\n",
    "\n",
    "* * * \n",
    "\n",
    "<div class=\"alert alert-success\">  \n",
    "    \n",
    "### Learning Objectives \n",
    "    \n",
    "* Learn common steps for preprocessing text data, as well as specific operations for preprocessing Twitter data.\n",
    "* Know commonly used NLP packages and what they are capable of.\n",
    "* Understand tokenizers, and how they have changed since the advent of Large Language Models.\n",
    "</div>\n",
    "\n",
    "### Icons Used in This Notebook\n",
    "🔔 **Question**: A quick question to help you understand what's going on.<br>\n",
    "🥊 **Challenge**: Interactive excersise. We'll work through these in the workshop!<br>\n",
    "⚠️ **Warning:** Heads-up about tricky stuff or common mistakes.<br>\n",
    "🎬 **Demo**: Showing off something more advanced – so you know what Python can be used for!<br> \n",
    "\n",
    "### Sections\n",
    "1. [Preprocessing](#section1)\n",
    "2. [Tokenization](#section2)\n",
    "\n",
    "In this three-part workshop series, we'll learn the building blocks for performing text analysis in Python. These techniques lie in the domain of Natural Language Processing (NLP). NLP is a field that deals with identifying and extracting patterns of language, primarily in written texts. Throughout the workshop series, we'll interact with various packages for performing text analysis: starting from simple string methods to specific NLP packages, such as `nltk`, `spaCy`, and more recent ones on Large Language Models (`BERT`).\n",
    "\n",
    "Now, let's have these packages properly installed before diving into the materials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d442e4c7-e926-493d-a64e-516616ad915a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment the following lines to install packages/model\n",
    "# %pip install NLTK\n",
    "# %pip install transformers\n",
    "# %pip install spaCy\n",
    "# !python -m spacy download en_core_web_sm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df5b8f8e-4e69-426e-a202-ec48b325e89a",
   "metadata": {},
   "source": [
    "<a id='section1'></a>\n",
    "\n",
    "# Preprocessing\n",
    "\n",
    "In Part 1 of this workshop, we'll address the first step of text analysis. Our goal is to convert the raw, messy text data into a consistent format. This process is often called **preprocessing**, **text cleaning**, or **text normalization**.\n",
    "\n",
    "You'll notice that at the end of preprocessing, our data is still in a format that we can read and understand. In Parts 2 and 3, we will begin our foray into converting the text data into a numerical representation—a format that can be more readily handled by computers. \n",
    "\n",
    "🔔 **Question**: Let's pause for a minute to reflect on **your** previous experiences working on text data. \n",
    "- What is the format of the text data you have interacted with (plain text, CSV, or XML)?\n",
    "- Where does it come from (structured corpus, scraped from the web, survey data)?\n",
    "- Is it messy (i.e., is the data formatted consistently)?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b35911a-3b3f-4a48-a7d1-9882aab04851",
   "metadata": {},
   "source": [
    "## Common Processes\n",
    "\n",
    "Preprocessing is not something we can accomplish with a single line of code. We often start by familiarizing ourselves with the data, and along the way, we gain a clearer understanding of the granularity of preprocessing we want to apply.\n",
    "\n",
    "Typically, we begin by applying a set of commonly used processes to clean the data. These operations don't substantially alter the form or meaning of the data; they serve as a standardized procedure to reshape the data into a consistent format.\n",
    "\n",
    "The following processes, for examples, are commonly applied to preprocess English texts of various genres. These operations can be done using built-in Python functions, such as `string` methods, and Regular Expressions. \n",
    "- Lowercase the text\n",
    "- Remove punctuation marks\n",
    "- Remove extra whitespace characters\n",
    "- Remove stop words\n",
    "\n",
    "After the initial processing, we may choose to perform task-specific processes, the specifics of which often depend on the downstream task we want to perform and the nature of the text data (i.e., its stylistic and linguistic features).  \n",
    "\n",
    "Before we jump into these operations, let's take a look at our data!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec5d7350-9a1e-4db9-b828-a87fe1676d8d",
   "metadata": {},
   "source": [
    "### Import the Text Data\n",
    "\n",
    "The text data we'll be working with is a CSV file. It contains tweets about U.S. airlines, scrapped from Feb 2015. \n",
    "\n",
    "Let's read the file `airline_tweets.csv` into dataframe with `pandas`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3d1ff64b-53ad-4eca-b846-3fda20085c43",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pandas\n",
    "import pandas as pd\n",
    "\n",
    "# File path to data\n",
    "csv_path = '../data/airline_tweets.csv'\n",
    "\n",
    "# Specify the separator\n",
    "tweets = pd.read_csv(csv_path, sep=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e397ac6a-c2ba-4cce-8700-b36b38026c9d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>airline_sentiment_confidence</th>\n",
       "      <th>negativereason</th>\n",
       "      <th>negativereason_confidence</th>\n",
       "      <th>airline</th>\n",
       "      <th>airline_sentiment_gold</th>\n",
       "      <th>name</th>\n",
       "      <th>negativereason_gold</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>text</th>\n",
       "      <th>tweet_coord</th>\n",
       "      <th>tweet_created</th>\n",
       "      <th>tweet_location</th>\n",
       "      <th>user_timezone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>570306133677760513</td>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>cairdin</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica What @dhepburn said.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:35:52 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Eastern Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>570301130888122368</td>\n",
       "      <td>positive</td>\n",
       "      <td>0.3486</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jnardino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica plus you've added commercials t...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:15:59 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>570301083672813571</td>\n",
       "      <td>neutral</td>\n",
       "      <td>0.6837</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>yvonnalynn</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica I didn't today... Must mean I n...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:15:48 -0800</td>\n",
       "      <td>Lets Play</td>\n",
       "      <td>Central Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>570301031407624196</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>Bad Flight</td>\n",
       "      <td>0.7033</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jnardino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica it's really aggressive to blast...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:15:36 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>570300817074462722</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>Can't Tell</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jnardino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica and it's a really big bad thing...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:14:45 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             tweet_id airline_sentiment  airline_sentiment_confidence  \\\n",
       "0  570306133677760513           neutral                        1.0000   \n",
       "1  570301130888122368          positive                        0.3486   \n",
       "2  570301083672813571           neutral                        0.6837   \n",
       "3  570301031407624196          negative                        1.0000   \n",
       "4  570300817074462722          negative                        1.0000   \n",
       "\n",
       "  negativereason  negativereason_confidence         airline  \\\n",
       "0            NaN                        NaN  Virgin America   \n",
       "1            NaN                     0.0000  Virgin America   \n",
       "2            NaN                        NaN  Virgin America   \n",
       "3     Bad Flight                     0.7033  Virgin America   \n",
       "4     Can't Tell                     1.0000  Virgin America   \n",
       "\n",
       "  airline_sentiment_gold        name negativereason_gold  retweet_count  \\\n",
       "0                    NaN     cairdin                 NaN              0   \n",
       "1                    NaN    jnardino                 NaN              0   \n",
       "2                    NaN  yvonnalynn                 NaN              0   \n",
       "3                    NaN    jnardino                 NaN              0   \n",
       "4                    NaN    jnardino                 NaN              0   \n",
       "\n",
       "                                                text tweet_coord  \\\n",
       "0                @VirginAmerica What @dhepburn said.         NaN   \n",
       "1  @VirginAmerica plus you've added commercials t...         NaN   \n",
       "2  @VirginAmerica I didn't today... Must mean I n...         NaN   \n",
       "3  @VirginAmerica it's really aggressive to blast...         NaN   \n",
       "4  @VirginAmerica and it's a really big bad thing...         NaN   \n",
       "\n",
       "               tweet_created tweet_location               user_timezone  \n",
       "0  2015-02-24 11:35:52 -0800            NaN  Eastern Time (US & Canada)  \n",
       "1  2015-02-24 11:15:59 -0800            NaN  Pacific Time (US & Canada)  \n",
       "2  2015-02-24 11:15:48 -0800      Lets Play  Central Time (US & Canada)  \n",
       "3  2015-02-24 11:15:36 -0800            NaN  Pacific Time (US & Canada)  \n",
       "4  2015-02-24 11:14:45 -0800            NaN  Pacific Time (US & Canada)  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show the first five rows\n",
    "tweets.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae3b339f-45cf-465d-931c-05f9096fd510",
   "metadata": {},
   "source": [
    "The dataframe has one row per tweet. The text of tweet is shown in the `text` column.\n",
    "- `text` (`str`): the text of the tweet.\n",
    "\n",
    "Other metadata we are interested in include: \n",
    "- `airline_sentiment` (`str`): the sentiment of the tweet, labeled as \"neutral,\" \"positive,\" or \"negative.\"\n",
    "- `airline` (`str`): the airline that is tweeted about.\n",
    "- `retweet count` (`int`): how many times the tweet was retweeted."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "302c695b-4bd1-4151-9cb9-ef5253eb16df",
   "metadata": {},
   "source": [
    "Let's take a look at some of the tweets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b690daab-7be5-4b8f-8af0-a91fdec4ec4f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@VirginAmerica What @dhepburn said.\n",
      "@VirginAmerica plus you've added commercials to the experience... tacky.\n",
      "@VirginAmerica I didn't today... Must mean I need to take another trip!\n"
     ]
    }
   ],
   "source": [
    "print(tweets['text'].iloc[0])\n",
    "print(tweets['text'].iloc[1])\n",
    "print(tweets['text'].iloc[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8adc05fa-ad30-4402-ab56-086bcb09a166",
   "metadata": {},
   "source": [
    "🔔 **Question**: What have you noticed? What are the stylistic features of tweets?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3460393-00a6-461c-b02a-9e98f9b5d1af",
   "metadata": {},
   "source": [
    "### Lowercasing\n",
    "\n",
    "While we acknowledge that a word's casing is informative, we often don't work in contexts where we can properly utilize this information.\n",
    "\n",
    "More often, the subsequent analysis we perform is **case-insensitive**. For instance, in frequency analysis, we want to account for various forms of the same word. Lowercasing the text data aids in this process and simplifies our analysis.\n",
    "\n",
    "We can easily achieve lowercasing with the string method [`.lower()`](https://docs.python.org/3/library/stdtypes.html#str.lower); see [documentation](https://docs.python.org/3/library/stdtypes.html#string-methods) for more useful functions.\n",
    "\n",
    "Let's apply it to the following example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "58a95d90-3ef1-4bff-9cfe-d447ed99f252",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@VirginAmerica I was scheduled for SFO 2 DAL flight 714 today. Changed to 24th due weather. Looks like flight still on?\n"
     ]
    }
   ],
   "source": [
    "# Print the first example\n",
    "first_example = tweets['text'][108]\n",
    "print(first_example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c66d91c0-6eed-4591-95fc-cd2eae2e0d41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "==================================================\n",
      "@virginamerica i was scheduled for sfo 2 dal flight 714 today. changed to 24th due weather. looks like flight still on?\n",
      "==================================================\n",
      "@VIRGINAMERICA I WAS SCHEDULED FOR SFO 2 DAL FLIGHT 714 TODAY. CHANGED TO 24TH DUE WEATHER. LOOKS LIKE FLIGHT STILL ON?\n"
     ]
    }
   ],
   "source": [
    "# Check if all characters are in lowercase\n",
    "print(first_example.islower())\n",
    "print(f\"{'=' * 50}\")\n",
    "\n",
    "# Convert it to lowercase\n",
    "print(first_example.lower())\n",
    "print(f\"{'=' * 50}\")\n",
    "\n",
    "# Convert it to uppercase\n",
    "print(first_example.upper())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bf0d8c8-bd6c-47ef-b305-09ac61d07d4d",
   "metadata": {},
   "source": [
    "### Eliminar Caracteres de Espaciado Extra\n",
    "\n",
    "A veces nos encontramos con textos que contienen espacios en blanco innecesarios, como espacios, tabulaciones y caracteres de nueva línea, lo cual es particularmente común cuando el texto proviene de páginas web. Antes de profundizar en los detalles, presentemos brevemente las Expresiones Regulares (regex) y el paquete `re`. \n",
    "\n",
    "Las expresiones regulares son una forma poderosa de buscar patrones específicos de cadenas en grandes corpus de texto. Tienen una curva de aprendizaje notoriamente empinada, pero pueden ser muy eficientes cuando logramos dominarlas. Muchos paquetes de procesamiento de lenguaje natural (NLP) dependen en gran medida de las expresiones regulares. Los probadores de regex, como [regex101](https://regex101.com), son herramientas útiles tanto para entender como para crear expresiones regulares.\n",
    "\n",
    "Nuestro objetivo en este taller no es ofrecer una inmersión profunda (ni siquiera superficial) en regex; en su lugar, queremos exponerlos a ellas para que estén mejor preparados para hacer inmersiones más profundas en el futuro.\n",
    "\n",
    "El siguiente ejemplo es un poema de William Wordsworth. Como muchos poemas, el texto puede contener saltos de línea extra (es decir, caracteres de nueva línea, `\\n`) que queremos eliminar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d1bd73f1-a30f-4269-a05e-47cfff7b496f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# File path to the poem\n",
    "text_path = '../data/poem_wordsworth.txt'\n",
    "\n",
    "# Read the poem in\n",
    "with open(text_path, 'r') as file:\n",
    "    text = file.read()\n",
    "    file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a693dd9-9706-40b3-863f-f568020245f7",
   "metadata": {},
   "source": [
    "Como pueden ver, el poema está formateado como una cadena continua de texto con saltos de línea al final de cada línea, lo que lo hace difícil de leer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7e78a75a-8e15-4bcb-a416-783aa7f60ef3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"I wandered lonely as a cloud\\n\\n\\nI wandered lonely as a cloud\\nThat floats on high o'er vales and hills,\\nWhen all at once I saw a crowd,\\nA host, of golden daffodils;\\nBeside the lake, beneath the trees,\\nFluttering and dancing in the breeze.\\n\\nContinuous as the stars that shine\\nAnd twinkle on the milky way,\\nThey stretched in never-ending line\\nAlong the margin of a bay:\\nTen thousand saw I at a glance,\\nTossing their heads in sprightly dance.\\n\\nThe waves beside them danced; but they\\nOut-did the sparkling waves in glee:\\nA poet could not but be gay,\\nIn such a jocund company:\\nI gazed—and gazed—but little thought\\nWhat wealth the show to me had brought:\\n\\nFor oft, when on my couch I lie\\nIn vacant or in pensive mood,\\nThey flash upon that inward eye\\nWhich is the bliss of solitude;\\nAnd then my heart with pleasure fills,\\nAnd dances with the daffodils.\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47cce993-c315-4aaa-87fe-149de8607f65",
   "metadata": {},
   "source": [
    "Una función útil que podemos usar para mostrar el poema correctamente es `.splitlines()`. Como su nombre indica, divide una secuencia de texto larga en una lista de líneas cada vez que encuentra un carácter de nueva línea."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ddeade7a-065d-49e6-bdd3-87a8ea8f6e6e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I wandered lonely as a cloud',\n",
       " '',\n",
       " '',\n",
       " 'I wandered lonely as a cloud',\n",
       " \"That floats on high o'er vales and hills,\",\n",
       " 'When all at once I saw a crowd,',\n",
       " 'A host, of golden daffodils;',\n",
       " 'Beside the lake, beneath the trees,',\n",
       " 'Fluttering and dancing in the breeze.',\n",
       " '',\n",
       " 'Continuous as the stars that shine',\n",
       " 'And twinkle on the milky way,',\n",
       " 'They stretched in never-ending line',\n",
       " 'Along the margin of a bay:',\n",
       " 'Ten thousand saw I at a glance,',\n",
       " 'Tossing their heads in sprightly dance.',\n",
       " '',\n",
       " 'The waves beside them danced; but they',\n",
       " 'Out-did the sparkling waves in glee:',\n",
       " 'A poet could not but be gay,',\n",
       " 'In such a jocund company:',\n",
       " 'I gazed—and gazed—but little thought',\n",
       " 'What wealth the show to me had brought:',\n",
       " '',\n",
       " 'For oft, when on my couch I lie',\n",
       " 'In vacant or in pensive mood,',\n",
       " 'They flash upon that inward eye',\n",
       " 'Which is the bliss of solitude;',\n",
       " 'And then my heart with pleasure fills,',\n",
       " 'And dances with the daffodils.']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Split the single string into a list of lines\n",
    "text.splitlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44d3825b-0857-44e1-bf6a-d8c7a9032704",
   "metadata": {},
   "source": [
    "Volvamos a nuestros datos de tweets para un ejemplo.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "53a81ea9-65c4-474a-8530-35393555d1be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"@VirginAmerica seriously would pay $30 a flight for seats that didn't have this playing.\\nit's really the only bad thing about flying VA\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print the second example\n",
    "second_example = tweets['text'][5]\n",
    "second_example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aef55865-36fd-4c06-a765-530cf3b53096",
   "metadata": {},
   "source": [
    "En este caso, realmente no queremos dividir el tweet en una lista de cadenas. Aún esperamos una sola cadena de texto, pero nos gustaría eliminar el salto de línea completamente de la cadena.\n",
    "\n",
    "El método de cadenas `.strip()` hace eficazmente el trabajo de eliminar los espacios al principio y al final del texto. Sin embargo, no funcionará en nuestro ejemplo, ya que el carácter de nueva línea está en el medio de la cadena."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b933503b-4370-4dc4-b287-6dc2f9cdb1d4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"@VirginAmerica seriously would pay $30 a flight for seats that didn't have this playing.\\nit's really the only bad thing about flying VA\""
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Strip only removed blankspace at both ends\n",
    "second_example.strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b99b80b4-804f-460f-a2d5-adbd654902b3",
   "metadata": {},
   "source": [
    "Aquí es donde las expresiones regulares (regex) podrían ser realmente útiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ceac9714-7053-4b2e-affb-71f8c3d2dcd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5f08d20-ba81-4e48-9e2a-5728148005b3",
   "metadata": {},
   "source": [
    "Ahora, con las expresiones regulares (regex), esencialmente la estamos llamando para que coincida con un patrón que hemos identificado en los datos de texto, y queremos realizar algunas operaciones sobre la parte coincidente: extraerla, reemplazarla por algo más o eliminarla por completo. Por lo tanto, el funcionamiento de regex podría desglosarse en los siguientes pasos:\n",
    "\n",
    "- Identificar y escribir el patrón en regex (`r'PATTERN'`)\n",
    "- Escribir el reemplazo para el patrón (`'REPLACEMENT'`)\n",
    "- Llamar a la función específica de regex (por ejemplo, `re.sub()`)\n",
    "\n",
    "En nuestro ejemplo, el patrón que estamos buscando es `\\s`, que es el nombre corto de regex para cualquier carácter de espacio en blanco (incluidos `\\n` y `\\t`). También agregamos un cuantificador `+` al final: `\\s+`. Esto significa que nos gustaría capturar una o más ocurrencias del carácter de espacio en blanco."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1248d227-1149-4014-94a5-c05592a27a7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write a pattern in regex\n",
    "blankspace_pattern = r'\\s+'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc075c2e-1a1d-4393-a3ea-8ad7c118364b",
   "metadata": {},
   "source": [
    "El reemplazo para uno o más caracteres de espacio en blanco es exactamente un solo espacio, que es el límite de palabra canónico en inglés. Cualquier espacio adicional se reducirá a un solo espacio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c55cb2f1-f4ca-4b79-900c-f65ec303ddac",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write a replacement for the pattern identfied\n",
    "blankspace_repl = ' '"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc12e3d1-728a-429b-9c83-4dcc88590bc4",
   "metadata": {},
   "source": [
    "Finalmente, pongamos todo junto usando la función [`re.sub()`](https://docs.python.org/3.11/library/re.html#re.sub), lo que significa que queremos sustituir un patrón por un reemplazo. La función recibe tres argumentos: el patrón, el reemplazo y la cadena a la que queremos aplicar la función."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5249b24b-7111-4569-be29-c40efa5e148e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@VirginAmerica seriously would pay $30 a flight for seats that didn't have this playing. it's really the only bad thing about flying VA\n"
     ]
    }
   ],
   "source": [
    "# Replace whitespace(s) with ' '\n",
    "clean_text = re.sub(pattern = blankspace_pattern, \n",
    "                    repl = blankspace_repl, \n",
    "                    string = second_example)\n",
    "print(clean_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a895fbe3-a034-4124-94af-72a528913c51",
   "metadata": {},
   "source": [
    "Ta-da!  El carácter de nueva línea ya no está allí."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7087dc0c-5fef-4f1c-8662-7cbc8a978f34",
   "metadata": {},
   "source": [
    "### Eliminar los Signos de Puntuación\n",
    "\n",
    "A veces solo estamos interesados en analizar **caracteres alfanuméricos** (es decir, las letras y los números), en cuyo caso podríamos querer eliminar los signos de puntuación.\n",
    "\n",
    "El módulo `string` contiene una lista de signos de puntuación predefinidos. Vamos a imprimirlos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "70e8502b-b703-45e0-8852-0c3210363440",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n"
     ]
    }
   ],
   "source": [
    "# Load in a predefined list of punctuation marks\n",
    "from string import punctuation\n",
    "print(punctuation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91119c9e-431c-42cb-afea-f7e607698929",
   "metadata": {},
   "source": [
    "En la práctica, para eliminar estos caracteres de puntuación, podemos simplemente iterar sobre el texto y eliminar los caracteres que se encuentren en la lista, como se muestra a continuación en la función `remove_punct`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "237d868d-339d-4bbe-9a3b-20fa5fbdf231",
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_punct(text):\n",
    "    '''Remove punctuation marks in input text'''\n",
    "    \n",
    "    # Select characters not in puncutaion\n",
    "    no_punct = []\n",
    "    for char in text:\n",
    "        if char not in punctuation:\n",
    "            no_punct.append(char)\n",
    "\n",
    "    # Join the characters into a string\n",
    "    text_no_punct = ''.join(no_punct)   \n",
    "    \n",
    "    return text_no_punct"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4fc768b-c2dd-4386-8212-483c4485e4be",
   "metadata": {},
   "source": [
    "Aplicamos la función al siguiente ejemplo.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7596c465-3d85-4b72-a853-f2151bcd91df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@VirginAmerica why are your first fares in May over three times more than other carriers when all seats are available to select???\n",
      "==================================================\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'VirginAmerica why are your first fares in May over three times more than other carriers when all seats are available to select'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print the third example\n",
    "third_example = tweets['text'][20]\n",
    "print(third_example)\n",
    "print(f\"{'=' * 50}\")\n",
    "\n",
    "# Apply the function \n",
    "remove_punct(third_example)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "853a4b83-f503-4405-aedd-66bbc088e3e7",
   "metadata": {},
   "source": [
    "Intentémoslo con otro tweet. ¿Qué has notado?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5b3c2f60-fc92-4326-bad6-5ad04be50476",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@VirginAmerica trying to add my boy Prince to my ressie. SF this Thursday @VirginAmerica from LAX http://t.co/GsB2J3c4gM\n",
      "==================================================\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'VirginAmerica trying to add my boy Prince to my ressie SF this Thursday VirginAmerica from LAX httptcoGsB2J3c4gM'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print another tweet\n",
    "print(tweets['text'][100])\n",
    "print(f\"{'=' * 50}\")\n",
    "\n",
    "# Apply the function\n",
    "remove_punct(tweets['text'][100])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1af02ce5-b674-4cb4-8e08-7d7416963f9c",
   "metadata": {},
   "source": [
    "¿Qué pasa con el siguiente ejemplo?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6f8c3947-e6b8-42fe-8a58-15e4b6c60005",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Weve got quite a bit of punctuation here dont we Python DLab'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print a text with contraction\n",
    "contraction_text = \"We've got quite a bit of punctuation here, don't we?!? #Python @D-Lab.\"\n",
    "\n",
    "# Apply the function\n",
    "remove_punct(contraction_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62574c66-db3f-4500-9c3b-cea2f3eb2a30",
   "metadata": {},
   "source": [
    "⚠️ **Advertencia:** En muchos casos, queremos eliminar los signos de puntuación **después** de la tokenización, lo cual discutiremos en un momento. ¡Esto nos dice que el **orden** del preprocesamiento es importante!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58c6b85e-58e7-4f56-9b4a-b60c85b394ba",
   "metadata": {},
   "source": [
    "## 🥊 Desafío 1: Preprocesamiento con Múltiples Pasos\n",
    "\n",
    "Hasta ahora hemos aprendido algunas operaciones de preprocesamiento, ¡vamos a combinarlas en una función! Esta función sería útil para referirse a ella si alguna vez trabajas con datos de texto en inglés desordenados y deseas preprocesarlos con una sola función.\n",
    "\n",
    "Los datos de texto de ejemplo para el desafío 1 se muestran a continuación. Escribe una función que:\n",
    "- Ponga el texto en minúsculas\n",
    "- Elimine los signos de puntuación\n",
    "- Elimine los caracteres de espacio en blanco extra\n",
    "\n",
    "¡Siéntete libre de reutilizar los códigos que hemos usado anteriormente!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "deb10cba-239e-4856-b56d-7d5eb850c9b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "This is a text file that has some extra blankspace at the start and end. Blankspace is a catch-all term for spaces, tabs, newlines, and a bunch of other things that computers distinguish but to us all look like spaces, tabs and newlines.\n",
      "\n",
      "\n",
      "The Python method called \"strip\" only catches blankspace at the start and end of a string. But it won't catch it in       the middle,\t\tfor example,\n",
      "\n",
      "in this sentence.\t\tOnce again, regular expressions will\n",
      "\n",
      "help\t\tus    with this.\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "challenge1_path = '../data/example1.txt'\n",
    "\n",
    "with open(challenge1_path, 'r') as file:\n",
    "    challenge1 = file.read()\n",
    "    \n",
    "print(challenge1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e2480823-65dd-4f52-a7b3-6d9b10d87912",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def clean_text(text):\n",
    "\n",
    "    # Step 1: Lowercase\n",
    "    text = ...\n",
    "\n",
    "    # Step 2: Use remove_punct to remove punctuation marks\n",
    "    text = ...\n",
    "\n",
    "    # Step 3: Remove extra whitespace characters\n",
    "    text = ...\n",
    "\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "dc603506-0adb-45d7-bb6f-62958c054fdd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Uncomment to apply the above function to challenge 1 text \n",
    "# clean_text(challenge1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67c159cb-8eaa-4c30-b8ff-38a712d2bb0f",
   "metadata": {},
   "source": [
    "## Procesos Específicos para Tareas\n",
    "\n",
    "Ahora que entendemos las operaciones comunes de preprocesamiento, aún hay algunas operaciones adicionales a considerar. Nuestros datos de texto pueden requerir una mayor normalización dependiendo del idioma, la fuente y el contenido de los datos.\n",
    "\n",
    "Por ejemplo, si estamos trabajando con documentos financieros, podríamos querer estandarizar los símbolos monetarios convirtiéndolos en dígitos. En nuestros datos de tweets, hay numerosos hashtags y URLs. Estos pueden ser reemplazados por marcadores de posición para simplificar el análisis posterior.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2936cea-74e9-40c2-bfbe-6ba8129330de",
   "metadata": {},
   "source": [
    "### 🎬 **Demostración**: Eliminar Hashtags y URLs\n",
    "\n",
    "Aunque las URLs, los hashtags y los números son informativos por derecho propio, a menudo no nos importa necesariamente el significado exacto de cada uno de ellos.\n",
    "\n",
    "Si bien podríamos eliminarlos por completo, a menudo es informativo saber que **existe** una URL o un hashtag. En la práctica, reemplazamos las URLs y los hashtags individuales por un \"símbolo\" que conserva el hecho de que estas estructuras existen en el texto. Es común usar simplemente las cadenas \"URL\" y \"HASHTAG\".\n",
    "\n",
    "Dado que estos tipos de texto suelen seguir una estructura regular, son un buen caso para usar expresiones regulares. Apliquemos estos patrones a los datos de tweets.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "03c0dc37-a013-4f0a-b72f-a1f64dc6c1bd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@VirginAmerica @virginmedia I'm flying your #fabulous #Seductive skies again! U take all the #stress away from travel http://t.co/ahlXHhKiyn\n"
     ]
    }
   ],
   "source": [
    "# Print the example tweet \n",
    "url_tweet = tweets['text'][13]\n",
    "print(url_tweet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "4ef61bea-ea11-468d-8176-a2f63659d204",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"@VirginAmerica @virginmedia I'm flying your #fabulous #Seductive skies again! U take all the #stress away from travel  URL \""
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# URL \n",
    "url_pattern = r'(http|ftp|https):\\/\\/([\\w_-]+(?:(?:\\.[\\w_-]+)+))([\\w.,@?^=%&:\\/~+#-]*[\\w@?^=%&\\/~+#-])'\n",
    "url_repl = ' URL '\n",
    "re.sub(url_pattern, url_repl, url_tweet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "ea8e0f2a-460e-4088-aa89-dc2a8bc6f7fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"@VirginAmerica @virginmedia I'm flying your HASHTAG  HASHTAG  skies again! U take all the HASHTAG  away from travel http://t.co/ahlXHhKiyn\""
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Hashtag\n",
    "hashtag_pattern = r'(?:^|\\s)[＃#]{1}(\\w+)'\n",
    "hashtag_repl = ' HASHTAG '\n",
    "re.sub(hashtag_pattern, hashtag_repl, url_tweet)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71d68d49-4923-49c0-9113-b844dc7546b9",
   "metadata": {},
   "source": [
    "<a id='section2'></a>\n",
    "\n",
    "# Tokenization\n",
    "\n",
    "## Tokenizers Before LLMs\n",
    "\n",
    "One of the most important steps in text analysis is tokenization. This is the process of breaking a long sequence of text into word tokens. With these tokens available, we are ready to perform word-level analysis. For instance, we can filter out tokens that don't contribute to the core meaning of the text.\n",
    "\n",
    "In this section, we'll introduce how to perform tokenization using `nltk`, `spaCy`, and a Large Language Model (`bert`). The purpose is to expose you to different NLP packages, help you understand their functionalities, and demonstrate how to access key functions in each package.\n",
    "\n",
    "### `nltk`\n",
    "\n",
    "The first package we'll be using is called **Natural Language Toolkit**, or `nltk`. \n",
    "\n",
    "Let's install a couple modules from the package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "441d81f8-361e-4273-bd36-91a272f4a38a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "64b327cc-5c77-4fdc-9aaf-17d7f0761237",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment the following lines to install these modules\n",
    "# nltk.download('wordnet')\n",
    "# nltk.download('stopwords')\n",
    "# nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e79b699-c3a5-489f-9b3c-95653aba34d6",
   "metadata": {},
   "source": [
    "`nltk` has a function called `word_tokenize`. It requires one argument, which is the text to be tokenized, and it returns a list of tokens for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "7b5d6944-c641-4fac-a239-5947a496371c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@VirginAmerica Really missed a prime opportunity for Men Without Hats parody, there. https://t.co/mWpG7grEZP\n"
     ]
    }
   ],
   "source": [
    "# Load word_tokenize \n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "# Print the example\n",
    "text = tweets['text'][7]\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "95fde2a3-e4e2-4e61-ad54-e4d5d0a6ba71",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['@',\n",
       " 'VirginAmerica',\n",
       " 'Really',\n",
       " 'missed',\n",
       " 'a',\n",
       " 'prime',\n",
       " 'opportunity',\n",
       " 'for',\n",
       " 'Men',\n",
       " 'Without',\n",
       " 'Hats',\n",
       " 'parody',\n",
       " ',',\n",
       " 'there',\n",
       " '.',\n",
       " 'https',\n",
       " ':',\n",
       " '//t.co/mWpG7grEZP']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply the NLTK tokenizer\n",
    "nltk_tokens = word_tokenize(text)\n",
    "nltk_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80ead039-7721-4b22-8590-0d7824631675",
   "metadata": {},
   "source": [
    "Here we are, with a list of tokens identified by `nltk`. Let's take a minute to inspect them! \n",
    "\n",
    "🔔 **Question**: Do word boundaries decided by `nltk` make sense to you? Pay attention to the twitter handle and the URL in the example tweet. \n",
    "\n",
    "You may feel that accessing functions in `nltk` is pretty straightforward. The function we used above was imported from the `nltk.tokenize` module, which as the name suggests, primarily does the job of tokenization. \n",
    "\n",
    "Underlyingly, `nltk` has [a collection of modules](https://www.nltk.org/api/nltk.html) that fulfill different purposes, to name a few:\n",
    "\n",
    "| NLTK module   | Fucntion                  | Link                                                         |\n",
    "|---------------|---------------------------|--------------------------------------------------------------|\n",
    "| nltk.tokenize | Tokenization              | [Documentation](https://www.nltk.org/api/nltk.tokenize.html) |\n",
    "| nltk.corpus   | Retrieve built-in corpora | [Documentation](https://www.nltk.org/nltk_data/)             |\n",
    "| nltk.tag      | Part-of-speech tagging    | [Documentation](https://www.nltk.org/api/nltk.tag.html)      |\n",
    "| nltk.stem     | Stemming                  | [Documentation](https://www.nltk.org/api/nltk.stem.html)     |\n",
    "| ...           | ...                       | ...                                                          |\n",
    "\n",
    "Let's import `stopwords` from the `nltk.corpus` module, which hosts a range of built-in corpora. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "84bbfced-8803-41ca-9cae-49bdadf8c000",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load predefined stop words from nltk\n",
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dee971a1-1189-4cb6-8317-4836f54c3ae2",
   "metadata": {},
   "source": [
    "Let's specificy that we want to retrieve English stop words. The function simply returns a list of stop words, mostly function words, that `nltk` identifies. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "6009e1df-b720-4d22-a162-7fd250a58672",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'about', 'above', 'after', 'again', 'against', 'ain', 'all', 'am', 'an']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print the first 10 stopwords\n",
    "stop = stopwords.words('english')\n",
    "stop[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c3ec908-de6c-42c5-a370-f1b1df0032b3",
   "metadata": {},
   "source": [
    "### `spaCy`\n",
    "Other than `nltk`, we have another widely-used package called `spaCy`. \n",
    "\n",
    "`spaCy` has its own processing pipeline. It takes in a string of text, runs the `nlp` pipeline on it, and stores the processed text and its annotations in an object called `doc`. The `nlp` pipeline always performs tokenization, as well as [other text analysis components](https://spacy.io/usage/processing-pipelines#custom-components) requested by the user. These components are pretty similar to modules in `nltk`. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6a0facd-4b75-41ac-920c-5ea044f7ae2e",
   "metadata": {},
   "source": [
    "<img src='../images/spacy.png' alt=\"spacy pipeline\" width=\"700\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3ef1eaf-2790-4928-b094-943f2803c6a0",
   "metadata": {},
   "source": [
    "Note that we always start by initializing the `nlp` pipeline, depending on the language of the text. Here, we are loading a pretrained language model for English: `en_core_web_sm`. The name suggests that it is a lightweight model trained on some text data (e.g., blogs); see model descriptions [here](https://spacy.io/models/en#en_core_web_sm).\n",
    "\n",
    "This is the first time we encounter the concept of **pretraining**, though you may have heard it elsewhere. In the context of NLP, pretraining means that the model has been trained on a vast amount of data. As a result, it comes with a certain \"knowledge\" of word structure and grammar of the language.\n",
    "\n",
    "Therefore, when we apply the model to our own data, we can expect it to be reasonably accurate in performing various annotation tasks, e.g., tagging a word's part of speech, identifying the syntactic head of a phrase, and etc. \n",
    "\n",
    "Let's dive in! We'll first need to load the pretrained language model we installed earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "524dfc02-aa8f-4888-9f81-74a570db72b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57d669c3-2f5a-41b6-893b-ea1d438b3a48",
   "metadata": {},
   "source": [
    "The `nlp` pipeline, by default, includes a set of components, which we can access via the `.pipe_names` attribute. \n",
    "\n",
    "You may notice that it dosen't include the tokenizer. Don't worry! Tokenizer is a special component that the pipeline always includes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "6d581ca5-43f8-4ef9-b099-2fc92c324581",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tok2vec', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer', 'ner']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Retrieve components included in NLP pipeline\n",
    "nlp.pipe_names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1e37f91-d174-4101-bfc6-2859cb0fe5cc",
   "metadata": {},
   "source": [
    "Let's run the `nlp` pipeline on our example tweet data, and assign it to a variable `doc`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "618e8558-625d-4546-8109-63f9bae9790f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply the pipeline to example tweet\n",
    "doc = nlp(tweets['text'][7])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54325d60-5c5c-488d-baf2-7eed4de2c031",
   "metadata": {},
   "source": [
    "Under the hood, the `doc` object contains the tokens (created by the tokenizer) and their annotations (created by other components), which are [linguistic features](\n",
    "https://spacy.io/usage/linguistic-features) useful for text analysis. We retrieve the token and its annotations by accessing corresponding attributes. \n",
    "\n",
    "| Attribute      | Annotation                              | Link                                                                      |\n",
    "|----------------|-----------------------------------------|---------------------------------------------------------------------------|\n",
    "| token.text     | The token in verbatim text              | [Documentation](https://spacy.io/api/token#attributes)                    |\n",
    "| token.is_stop  | Whether the token is a stop word        | [Documentation](https://spacy.io/api/attributes#_title)                   |\n",
    "| token.is_punct | Whether the token is a punctuation mark | [Documentation](https://spacy.io/api/attributes#_title)                   |\n",
    "| token.lemma_   | The base form of the token              | [Documentation](https://spacy.io/usage/linguistic-features#lemmatization) |\n",
    "| token.pos_     | The simple POS-tag of the token         | [Documentation](https://spacy.io/usage/linguistic-features#pos-tagging)   |\n",
    "| ...            | ...                                     | ...                                                                       |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e9f23c8-a157-44a7-a6ec-6894aec1a595",
   "metadata": {},
   "source": [
    "Let's first get the tokens themselves! We'll iterate over the `doc` object and retrieve the text of each token. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "4c71efee-e6cf-46c4-9198-593304f6560d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['@VirginAmerica',\n",
       " 'Really',\n",
       " 'missed',\n",
       " 'a',\n",
       " 'prime',\n",
       " 'opportunity',\n",
       " 'for',\n",
       " 'Men',\n",
       " 'Without',\n",
       " 'Hats',\n",
       " 'parody',\n",
       " ',',\n",
       " 'there',\n",
       " '.',\n",
       " 'https://t.co/mWpG7grEZP']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the verbatim texts of tokens\n",
    "spacy_tokens = [token.text for token in doc]\n",
    "spacy_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "f4fc23f0-c699-45e6-ad62-e131036d601f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['@',\n",
       " 'VirginAmerica',\n",
       " 'Really',\n",
       " 'missed',\n",
       " 'a',\n",
       " 'prime',\n",
       " 'opportunity',\n",
       " 'for',\n",
       " 'Men',\n",
       " 'Without',\n",
       " 'Hats',\n",
       " 'parody',\n",
       " ',',\n",
       " 'there',\n",
       " '.',\n",
       " 'https',\n",
       " ':',\n",
       " '//t.co/mWpG7grEZP']"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the NLTK tokens\n",
    "nltk_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0ace59e-40e0-42b3-9f2b-d30ac94dccab",
   "metadata": {},
   "source": [
    "🔔 **Question**: Let's pause for a minute to compare the tokens generated by `nltk` and `spaCy`. What have you noticed?\n",
    "\n",
    "Remember we can also access various annotations of these okens. For instance, one annotation `spaCy` offers is that it conveniently encodes whether a token is a stop word. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "626af687-e986-4c97-af86-edf7dbd22c3e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[False,\n",
       " True,\n",
       " False,\n",
       " True,\n",
       " False,\n",
       " False,\n",
       " True,\n",
       " False,\n",
       " True,\n",
       " False,\n",
       " False,\n",
       " False,\n",
       " True,\n",
       " False,\n",
       " False]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Retrieve the is_stop annotation\n",
    "spacy_stops = [token.is_stop for token in doc]\n",
    "\n",
    "# The results are boolean values\n",
    "spacy_stops"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b6548b6-7e89-4f42-b8cb-bf7c93b34eb4",
   "metadata": {},
   "source": [
    "## 🥊 Challenge 2: Remove Stop Words\n",
    "\n",
    "We have known how `nltk` and `spaCy` work as NLP packages. We've also demostrated how to identify stop words with each package. \n",
    "\n",
    "Let's write **two** functions to remove stop words from our text data. \n",
    "\n",
    "- Complete the function for stop words removal using `nltk`\n",
    "    - The starter code requires two arguments: the raw text input and a list of predefined stop words\n",
    "- Complete the function for stop words removal using `spaCy`\n",
    "    - The starter code requires one argument: the raw text input\n",
    " \n",
    "A friendly reminder before we dive in: both functions take raw text as input—that's a signal to perform tokenization on the raw text first!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b24b5370-392f-420d-8c9e-78146d0fca29",
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_stopword_nltk(raw_text, stopword):\n",
    "    \n",
    "    # Step 1: Tokenization with nltk\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Step 2: Filter out tokens in the stop word list\n",
    "    # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65b5bbda-0af5-49a9-8f5e-77d61ab217e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_stopword_spacy(raw_text):\n",
    "\n",
    "    # Step 1: Apply the nlp pipeline\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Step 2: Filter out tokens that are stop words\n",
    "    # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4c3b0da-2223-4d7f-9014-696498e804e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove_stopword_nltk(text, stop)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f83538ba-6bf1-49ca-90ec-b6532b1ffcb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove_stopword_spacy(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3a6b1ec-87cc-4a08-a5dd-0210a9c56f0b",
   "metadata": {},
   "source": [
    "## 🎬 **Demo**: Powerful Features from `spaCy`\n",
    "\n",
    "`spaCy`'s nlp pipeline includes a number of linguistic annotations, which could be very useful for text analysis. \n",
    "\n",
    "For instance, we can access more annotations such as the lemma, the part-of-speech tag and its meaning, and whether the token looks like URLs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "eb6c7d93-51a3-4fb8-8321-cb672f4f1b8f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@VirginAmerica           | @VirginAmerica           | PROPN        | proper noun  | 0            |\n",
      "Really                   | really                   | ADV          | adverb       | 0            |\n",
      "missed                   | miss                     | VERB         | verb         | 0            |\n",
      "a                        | a                        | DET          | determiner   | 0            |\n",
      "prime                    | prime                    | ADJ          | adjective    | 0            |\n",
      "opportunity              | opportunity              | NOUN         | noun         | 0            |\n",
      "for                      | for                      | ADP          | adposition   | 0            |\n",
      "Men                      | Men                      | PROPN        | proper noun  | 0            |\n",
      "Without                  | without                  | ADP          | adposition   | 0            |\n",
      "Hats                     | Hats                     | PROPN        | proper noun  | 0            |\n",
      "parody                   | parody                   | NOUN         | noun         | 0            |\n",
      ",                        | ,                        | PUNCT        | punctuation  | 0            |\n",
      "there                    | there                    | ADV          | adverb       | 0            |\n",
      ".                        | .                        | PUNCT        | punctuation  | 0            |\n",
      "https://t.co/mWpG7grEZP  | https://t.co/mWpG7grEZP  | PROPN        | proper noun  | 1            |\n"
     ]
    }
   ],
   "source": [
    "# Print tokens and their annotations\n",
    "for token in doc:\n",
    "    print(f\"{token.text:<24} | {token.lemma_:<24} | {token.pos_:<12} | {spacy.explain(token.pos_):<12} | {token.like_url:<12} |\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17388e0c-88b6-4cd9-8d2b-adb7f10b5330",
   "metadata": {},
   "source": [
    "As you can imagine, it is typical for this dataset to contain place names and airport codes. It would be cool if we are able to identify them and extract them from tweets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "cb489f78-fbb2-497b-a36d-3400c00c9b9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@JetBlue Vegas, San Francisco, Baltimore, San Diego and Philadelphia so far! I'm a very frequent business traveler.\n",
      "==================================================\n",
      "@VirginAmerica Flying LAX to SFO and after looking at the awesome movie lineup I actually wish I was on a long haul.\n"
     ]
    }
   ],
   "source": [
    "# Print example tweets with place names and airport codes\n",
    "tweet_city = tweets['text'][8273]\n",
    "tweet_airport = tweets['text'][502]\n",
    "print(tweet_city)\n",
    "print(f\"{'=' * 50}\")\n",
    "print(tweet_airport)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "013990a5-5e07-4a45-9427-fcd33840d3b8",
   "metadata": {},
   "source": [
    "We can use the \"ner\" ([Named Entity Recognition](https://en.wikipedia.org/wiki/Named-entity_recognition)) component to identify entities and their categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "b9e63519-5991-49fa-9a5d-be9f9b408ad1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vegas           | 9          | 14         | GPE       \n",
      "San Francisco   | 16         | 29         | GPE       \n",
      "Baltimore       | 31         | 40         | GPE       \n",
      "San Diego       | 42         | 51         | GPE       \n",
      "Philadelphia    | 56         | 68         | GPE       \n"
     ]
    }
   ],
   "source": [
    "# Print entities identified from the text\n",
    "doc_city = nlp(tweet_city)\n",
    "for ent in doc_city.ents:\n",
    "    print(f\"{ent.text:<15} | {ent.start_char:<10} | {ent.end_char:<10} | {ent.label_:<10}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b933ed0-7018-450c-b0a6-fb76cb6d5be9",
   "metadata": {},
   "source": [
    "We can also use `displacy` to highlight entities identified in the text, and at the same time, annotate the entity category. \n",
    "\n",
    "In the following example, we have four `GPE` (i.e., geopolitical entities, usually countries and cities) identified. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "7a5a5219-af1f-445c-a35b-7c49d739a91b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">@JetBlue \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Vegas\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       ", \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    San Francisco\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       ", \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Baltimore\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       ", \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    San Diego\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Philadelphia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " so far! I'm a very frequent business traveler.</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize the identified entities\n",
    "from spacy import displacy\n",
    "displacy.render(doc_city, style='ent', jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5d7953b-04a1-46d0-817a-db29edd8c83b",
   "metadata": {},
   "source": [
    "Let's give it a try with another example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "9246c7e3-8990-4d47-a355-deb63dbd1cc7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@VirginAmerica  | 0          | 14         | CARDINAL  \n",
      "Flying LAX      | 15         | 25         | ORG       \n",
      "SFO             | 29         | 32         | ORG       \n"
     ]
    }
   ],
   "source": [
    "# Print entities identified from the text\n",
    "doc_airport = nlp(tweet_airport)\n",
    "for ent in doc_airport.ents:\n",
    "     print(f\"{ent.text:<15} | {ent.start_char:<10} | {ent.end_char:<10} | {ent.label_:<10}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1df38472-3193-44c5-9b2f-e311ce9d42e0",
   "metadata": {},
   "source": [
    "Interesting that airport codes are identified as `ORG`—organizations, and the tweet handle as `CARDINAL`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "7e4bf382-4c57-4b78-a37f-fc1f6cb1d565",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    @VirginAmerica\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Flying LAX\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " to \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    SFO\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " and after looking at the awesome movie lineup I actually wish I was on a long haul.</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize the identified entities\n",
    "displacy.render(doc_airport, style='ent', jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "467d6f28-effc-4fe1-90e9-47c89dc5492d",
   "metadata": {},
   "source": [
    "## Tokenizers Since LLMs\n",
    "\n",
    "So far, we've seen what tokenization looks like with two widely-used NLP packages. They work quite well in some settings, but not others. Recall that `nltk` struggles with URLs. Now, imagine the data we have is even messier, containing misspellings, recently coined words, foreign names, and etc (collectively called \"out of vocabulary\" or OOV words). In such circumstances, we might need a more powerful model to handle these complexities.\n",
    "\n",
    "In fact, tokenization schemes change substantially with **Large Language Models** (LLMs), which are models trained on an enormous amount of data from mixed sources. With that magnitude of data, LLMs are better at chunking a longer sequence into tokens and tokens into **subtokens**. These subtokens can be morphological units of a word, such as an affix, but they can also be parts of a word where the model sets a \"meaningful\" boundary. \n",
    "\n",
    "In this section, we will demonstrate tokenization in **BERT** (Bidirectional Encoder Representations from Transformers), which utilizes a tokenization algorithm called [**WordPiece**](https://huggingface.co/learn/nlp-course/en/chapter6/6). \n",
    "\n",
    "We will load the tokenizer of BERT from the package `transformers`, which hosts a number of Transformer-based LLMs (e.g., BERT). We won't go into the architecture of Transformer in this workshop, but feel free to check out the D-lab workshop on [GPT Fundamentals](https://github.com/dlab-berkeley/GPT-Fundamentals)!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5e1509b-5b9e-456d-909f-a6b5099c48c8",
   "metadata": {},
   "source": [
    "### WordPiece Tokenization\n",
    "\n",
    "Note that BERT comes in a variety of versions. The one we will explore today is `bert-base-uncased`. This model has a moderate size (referred to as `base`) and is case-insensitive, meaning the input text will be lowercased by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7927226c-d04e-4117-9c49-3d355611b209",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Load BERT tokenizer in\n",
    "from transformers import BertTokenizer\n",
    "\n",
    "# Initialize the tokenizer \n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "128cfb38-274e-4d75-9d14-8744020fe49c",
   "metadata": {},
   "source": [
    "The tokenizer has multiple functions, as we will see in a minute. Now we want to access the `.tokenize()` function from the tokenizer. \n",
    "\n",
    "Let's tokenize an example tweet below. What have you noticed?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "62649193-bb00-4ae8-9102-bce3d1dfb6c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text: @VirginAmerica Just DM'd. Same issue persisting.\n",
      "==================================================\n",
      "Tokens: ['@', 'virgin', '##ame', '##rica', 'just', 'd', '##m', \"'\", 'd', '.', 'same', 'issue', 'persist', '##ing', '.']\n",
      "Number of tokens: 15\n"
     ]
    }
   ],
   "source": [
    "# Select an example tweet from dataframe\n",
    "text = tweets['text'][194]\n",
    "print(f\"Text: {text}\")\n",
    "print(f\"{'=' * 50}\")\n",
    "\n",
    "# Apply tokenizer\n",
    "tokens = tokenizer.tokenize(text)\n",
    "print(f\"Tokens: {tokens}\")\n",
    "print(f\"Number of tokens: {len(tokens)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc4f780e-207c-4d3d-b1b6-063c6d118945",
   "metadata": {},
   "source": [
    "The double \"hashtag\" symbols (`##`) refer to a subword token—a segment separated from the previous token.\n",
    "\n",
    "🔔 **Question**: Do these subwords make sense to you? \n",
    "\n",
    "One significant development with LLMs is that each token is assigned an ID from its vocabulary. Our computer does not understand text in its raw form, so each token is translated into an ID. These IDs are the inputs that the model accesses and operates on.\n",
    "\n",
    "Tokens and IDs can be converted bidirectionally, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "3112a63d-82b6-4fc0-a904-ab86f8740653",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ID of just is: 2074\n",
      "Token 2074 is: just\n"
     ]
    }
   ],
   "source": [
    "# Get the input ID of the word \n",
    "print(f\"ID of just is: {tokenizer.vocab['just']}\")\n",
    "\n",
    "# Get the text of the input ID\n",
    "print(f\"Token 2074 is: {tokenizer.decode([2074])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b9fd13d-f26e-480c-b43e-2b5fbc4898cd",
   "metadata": {},
   "source": [
    "Let's convert tokens to input IDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "8d125e1d-2560-4136-829c-b1c11e34636c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of input IDs: 15\n",
      "Input IDs of text: [1030, 6261, 14074, 14735, 2074, 1040, 2213, 1005, 1040, 1012, 2168, 3277, 29486, 2075, 1012]\n"
     ]
    }
   ],
   "source": [
    "# Convert a list of tokens to a list of input IDs\n",
    "input_ids = tokenizer.convert_tokens_to_ids(tokens)\n",
    "print(f\"Number of input IDs: {len(input_ids)}\")\n",
    "print(f\"Input IDs of text: {input_ids}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f25a1a14-a6db-414b-a1a8-c6be73c81a15",
   "metadata": {},
   "source": [
    "### Special Tokens\n",
    "\n",
    "In addition to the tokens and subtokens discussed above, BERT also makes use of three special tokens: `SEP`, `CLS`, and `UNK`. The `SEP` token acts as a sentence terminator, commonly known as an `EOS` (End of Sentence) token. The `UNK` token represents any token that is not found in the vocabulary, hence \"unknown\" tokens. The `CLS` token is added to the beginning of the sentence. It originates from text classification tasks (e.g., spam detection), where reseachers found it useful to have a token that aggregates the information of the entire sentence for classification purposes.\n",
    "\n",
    "When we apply `tokenizer()` directly to our text data, we are asking BERT to **encode** the text for us. This involves multiple steps: \n",
    "- Tokenize the text\n",
    "- Add special tokens\n",
    "- Convert tokens to input IDs\n",
    "- Other model-specific processes\n",
    "  \n",
    "Let's print them out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "21479c25-7a9a-4fac-ba09-1812575b8170",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of input IDs: 17\n",
      "IDs from tokenizer: [101, 1030, 6261, 14074, 14735, 2074, 1040, 2213, 1005, 1040, 1012, 2168, 3277, 29486, 2075, 1012, 102]\n"
     ]
    }
   ],
   "source": [
    "# Get the input IDs by providing the key \n",
    "input_ids_from_tokenizer = tokenizer(text)['input_ids']\n",
    "print(f\"Number of input IDs: {len(input_ids_from_tokenizer)}\")\n",
    "print(f\"IDs from tokenizer: {input_ids_from_tokenizer}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb6ea98e-7374-488a-a804-46cab166125c",
   "metadata": {},
   "source": [
    "It looks like we have two more tokens added: 101 and 102. \n",
    "\n",
    "Let's convert them to texts!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "82a18dc1-ca0d-4d5b-8ac6-f56cb44bf8e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The 101st token: [CLS]\n",
      "The 102nd token: [SEP]\n"
     ]
    }
   ],
   "source": [
    "# Convert input IDs to texts\n",
    "print(f\"The 101st token: {tokenizer.convert_ids_to_tokens(101)}\")\n",
    "print(f\"The 102nd token: {tokenizer.convert_ids_to_tokens(102)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b16d4e10-e96c-432a-a6c9-2c992d00fcde",
   "metadata": {},
   "source": [
    "As you can see, our text example is now a list of vocabulary IDs. In addtion to that, BERT adds the sentence terminator `SEP` and the beginning `CLS` token to the original text. BERT's tokenizer encodes tons of texts likewise; and afterwards, they are ready for further processes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac56be32-2a5f-441e-8283-de3e60705c0b",
   "metadata": {},
   "source": [
    "## 🥊 Challenge 3: Find the Word Boundary\n",
    "\n",
    "Now that we know tokenization in BERT often returns subwords. Let's try a few more examples. \n",
    "\n",
    "- What do you think is the correct boundary for splitting the following words into subwords?\n",
    "- What other examples have you tested?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "f1140b19-398c-44dd-829c-c922b0e6f7eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tokens(string):\n",
    "    '''Tokenzie the input string with BERT'''\n",
    "    tokens = tokenizer.tokenize(string)\n",
    "    return print(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "0c07c71e-5be2-4d91-9c6f-26de4145307d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['dl', '##ab']\n",
      "['co', '##vid']\n",
      "['hug', '##ga', '##ble']\n",
      "['37', '##8']\n"
     ]
    }
   ],
   "source": [
    "# Abbreviations\n",
    "get_tokens('dlab')\n",
    "\n",
    "# OOV\n",
    "get_tokens('covid')\n",
    "\n",
    "# Prefix\n",
    "get_tokens('huggable')\n",
    "\n",
    "# Digits\n",
    "get_tokens('378')\n",
    "\n",
    "# YOUR EXAMPLE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4acb7cb9-6e60-4e0a-8cc6-21d57237e835",
   "metadata": {},
   "source": [
    "We will wrap up Part 1 with this (hopefully) thought-provoking challenge. LLMs often come with a much more sophisticated tokenization scheme, but there is ongoing discussion about their limitations in real-world applications. The reference section includes a few blog posts discussing this problem. Feel free to explore further if this sounds like an interesting question to you!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7943ed9-70de-4f4a-b1bb-b2896d05e618",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "1. A tutorial introducing the tokenization scheme in BERT: [The huggingface NLP course on wordpiece tokenization](https://huggingface.co/learn/nlp-course/chapter6/6?fw=pt)\n",
    "2. A specific example of \"failure\" in tokenization: [Weaknesses of wordpiece tokenization: Findings from the front lines of NLP at VMware.](https://medium.com/@rickbattle/weaknesses-of-wordpiece-tokenization-eb20e37fec99)\n",
    "3. How does BERT decide boundaries between subtokens: [Subword tokenization in BERT](https://tinkerd.net/blog/machine-learning/bert-tokenization/#subword-tokenization)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce0812a7-f033-46ed-bc7b-67109c369e6c",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "## ❗ Key Points\n",
    "\n",
    "* Preprocessing includes multiple steps, some of them are more common to text data regardlessly, and some are task-specific. \n",
    "* Both `nltk` and `spaCy` could be used for tokenization and stop word removal. The latter is more powerful in providing various linguistic annotations. \n",
    "* Tokenization works differently in BERT, which often involves breaking down a whole word into subwords. \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}