02_Lab_1_Data_analysis_visualisation_solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introductory applied machine learning (INFR10069)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 1: Data analysis and visualisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab we work with a spam filtering dataset. We will perform exploratory data analysis, visualisation and, finally, we learn how to perform classification tasks using Naive Bayes. For this, we will use the the packages introduced in Lab 1, and `scikit-learn` package (`sklearn`): a machine learning library for Python which works with numpy array, and pandas DataFrame objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Please Note**: Throughout this lab we make reference to [`methods`](https://en.wikipedia.org/wiki/Method_%28computer_programming%29) for specific objects e.g. \"make use of the predict method of the MultinomialNB classifier\". If you get confused, refer to the documentation and just ctrl+f for the object concerned:\n",
    "* [Scikit-learn API documentation](http://scikit-learn.org/stable/modules/classes.html) \n",
    "* [Seaborn API documentation](https://seaborn.github.io/api.html)\n",
    "* [Matplotlib Pyplot documentation](http://matplotlib.org/1.5.3/api/pyplot_summary.html)\n",
    "* [Pandas API documentation](http://pandas.pydata.org/pandas-docs/stable/api.html)\n",
    "* [Numpy documentation](http://docs.scipy.org/doc/numpy/reference/)\n",
    "\n",
    "There are also tonnes of great examples online; googling key words with the word \"example\" will serve you well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need to import the packages (run all the code cells as you read along):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import packages\n",
    "from __future__ import division, print_function # Imports from __future__ since we're running Python 2\n",
    "import os\n",
    "import numpy as np \n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Clarification*:\n",
    "\n",
    "* The `%matplotlib inline` command is a special ipython [built in magic command](http://ipython.readthedocs.io/en/stable/interactive/magics.html) which forces the matplotlib plots to be rendered within the notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spambase dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Spambase](http://archive.ics.uci.edu/ml/datasets/Spambase) dataset consists of tagged emails from a single email account. You should read through the description available for this data to get a feel for what you're dealing with. We have downloaded the dataset for you.\n",
    "\n",
    "You will find the dataset located at `./datasets/spambase.csv` (the `datasets` directory is adjacent to this file). Execute the cell below to load the csv into in a pandas DataFrame object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load the dataset\n",
    "data_path = os.path.join(os.getcwd(), 'datasets', 'spambase.csv')\n",
    "spambase = pd.read_csv(data_path, delimiter = ',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have now loaded the data. Let's get a feeling of what the data looks like by using the `head()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "spambase.head(5) # Display the 5 first rows of the dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** Display the number of attributes in the dataset (i.e. number of columns)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "spambase.shape[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Display the number of observations (i.e. number of rows)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "spambase.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** Display the mean and standard deviation of each attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "spambase.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now want to *remove* some of the attributes from our data. There are various reasons for wanting to do so, for instance we might think that these are not relevant to the task we want to perform (i.e. e-mail classification) or they might have been contaminated with noise during the data collection process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** Delete the `capital_run_length_average`, `capital_run_length_longest` and  `capital_run_length_total` attributes. *Hint*: You should make use of the [`drop`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop.html) method. *Tip*: some pandas methods have the argument `inplace` which you can use to determine whether they alter the object they are called upon and return nothing, or return a new object. This is particularly useful if you are dealing with huge datasets where you would typically want to operate `inplace`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "spambase.drop([\"capital_run_length_average\", \"capital_run_length_longest\", \n",
    "                          \"capital_run_length_total\"], axis=1, inplace=True)\n",
    "## or, less efficiently\n",
    "# spambase = spambase.drop([\"capital_run_length_average\", \"capital_run_length_longest\", \n",
    "#                           \"capital_run_length_total\"], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Display the new number of attributes. Does it look like what you expected?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "spambase.shape[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The remaining attributes represent relative frequencies of various important words and characters in emails. This is true for all attributes except `is_spam` which represents whether the e-mail was annotated as spam or not. So each e-mail is represented by a 55 dimensional vector representing whether or not a particular word exists in an e-mail. This is the so called [bag of words](http://en.wikipedia.org/wiki/Bag_of_words_model) representation and is clearly a very crude approximation since it does not take into account the order of the words in the emails."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 3 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's get a feeling of the distribution of ham (i.e. valid) vs. spam emails. We can do this by using a [countplot](https://seaborn.github.io/generated/seaborn.countplot.html?highlight=countplot#seaborn.countplot) in seaborn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** Produce a seaborn [countplot](https://seaborn.github.io/generated/seaborn.countplot.html?highlight=countplot#seaborn.countplot) object that shows the distribution of ham/spam e-mails. Assign it to a variable (e.g. `ax` to emphasise it is a [matplotlib.axes.Axes](http://matplotlib.org/api/axes_api.html#axes) object)\n",
    "  \n",
    "**b)** In the same cell, modify the labels on the x axis (`xticklabels`) to `Ham` and `Spam` (by default they should be set to `0.0` and `1.0`). *Hint: Axes objects have a [`set_xticklabels`](http://matplotlib.org/api/axes_api.html#matplotlib.axes.Axes.set_xticklabels) method!* \n",
    "  \n",
    "**c)** Finally, again in the same cell, remove the `is_spam` label from the x axis (`xlabel`) since it does not add any information to the graph\n",
    "\n",
    "You may notice `<matplotlib.text.Text at ...memory_location...>` printed by the ipython notebook. This is just because the notebook is inferring how to display the last object in the cell. To explicitly plot the Axes object, use the `matplotlib.pyplot.show()` method at the very end of the cell, i.e. `plt.show()` (we imported the `matplotlib.pyplot` module as `plt` above)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "ax = sns.countplot(x='is_spam', data=spambase)\n",
    "ax.set_xticklabels(['Ham', 'Spam'])\n",
    "plt.xlabel('')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to simplify the problem by transforming our dataset. We will replace all numerical values which represent word frequencies with a binary value representing whether each word was present in a document or not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 4 ==========\n",
    "\n",
    "**a)** Crate a new dataframe called `spambase_binary` from `spambase`. *Hint*: Look into the [`copy`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.copy.html) method in pandas. *Tip*: Be careful, in python, unless you explictly say not to, assigment is typically just reference e.g.\n",
    "```python\n",
    "i = [1, 3]\n",
    "j = i\n",
    "i[1] = 5\n",
    "print(j)\n",
    "```\n",
    "outputs:\n",
    "```\n",
    "[1, 5]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "spambase_binary = spambase.copy(deep=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Convert all attributes in `spambase_binary` to Boolean values: 1 if the word or character is present in the email, or 0 otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "spambase_binary[spambase_binary > 0] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** Display the 5 last observations of the transformed dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "spambase_binary.tail(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to get a feeling for how the presence or absence of some specific words could affect the outcome (whether an email is classifed as *ham* or *spam*). We will be focusing on three specific words, namely `make`, `internet` and `edu`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 5 ==========\n",
    "\n",
    "**a)** Using seaborn, produce one figure with three [countplots](https://seaborn.github.io/generated/seaborn.countplot.html?highlight=countplot#seaborn.countplot), one for each of the frequency variables for the words `make`, `internet` and `edu`. For each variable, the count plot should have two bars: the number of emails containing the word (i.e. the variable = 1), and the number not containing that word (i.e. the variable = 0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "words_of_interest = ['word_freq_' + word for word in ['make', 'internet', 'edu']]\n",
    "n_words = len(words_of_interest)\n",
    "plt.subplots(n_words, figsize=(8,10))\n",
    "for i, word in enumerate(words_of_interest):\n",
    "    plt.subplot(n_words,1,i+1)\n",
    "    # Grey colour chosen such that the meaning of 1 and 0 is\n",
    "    # not confused between plots. It is good practice to use\n",
    "    # colour only when it is meaningful, and to not mix \n",
    "    # colour meanings between similar plots\n",
    "    sns.countplot(x=word, data=spambase_binary, color='Gray') \n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Repeat the above but split the bars showing the proportion of emails that are spam/ham. *Hint*: This only requires you to use the `hue` input argument to use different colours for the `is_spam` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "words_of_interest = ['word_freq_' + word for word in ['make', 'internet', 'edu']]\n",
    "n_words = len(words_of_interest)\n",
    "plt.subplots(n_words, figsize=(8,10))\n",
    "for ii, word in enumerate(words_of_interest):\n",
    "    plt.subplot(n_words, 1, ii+1)\n",
    "    sns.countplot(x=word, hue='is_spam', data=spambase_binary)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multinomial Naive Bayes classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the transformed dataset, we now wish to train a Naïve Bayes classifier to distinguish spam from regular email by fitting a distribution of the number of occurrences of each word for all the spam and non-spam e-mails. Read about the [Naive Bayes classifier](https://en.wikipedia.org/wiki/Naive_Bayes_classifier) and the underlying assumption if you are not already familiar with it. In this lab we focus on the [Multinomial Naive Bayes classifier](https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Multinomial_naive_Bayes). \n",
    "\n",
    "We will make use of the `MultinomialNB` class in `sklearn`. **Check out the user guide [description](http://scikit-learn.org/stable/modules/naive_bayes.html#multinomial-naive-bayes) and [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html#sklearn.naive_bayes.MultinomialNB) to familiarise yourself with this class.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All classifiers in `sklearn` implement a `fit()` and `predict()` [method](https://en.wikipedia.org/wiki/Method_%28computer_programming%29). The first learns the parameters of the model and the latter classifies inputs. For a Naive Bayes classifier, the [`fit()`](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html#sklearn.naive_bayes.GaussianNB.fit) method takes at least two input arguments `X` and `y`, where `X` are the input features and `y` are the labels associated with each example in the training dataset (i.e. targets). \n",
    "\n",
    "As a first step we extract the input features and targets from the DataFrame. To do so, we will use the [`values`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.values.html) property. For the input features we want to select all columns except `is_spam` and for this we may use the [`drop`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop.html) method which discards the specified columns along the given axis. In fact, we can combine these two operations in one step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 6 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** Create a Pandas DataFrame object `X` containing only the features (i.e. exclude the label `is_spam`). We need to do this as it is the input Scikit-learn objects expect for fitting. *Hint*: make use of the `drop` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "X = spambase_binary.drop('is_spam', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Create a Pandas Series object `y` that contains only the label from `spambase_binary`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "y = spambase_binary['is_spam']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** Display the dimensionality (i.e. `shape`) of each of the two arrays. *Hint:* The shape of `X` and `y` should be `(4601, 54)` and `(4601,)` respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "print('Input features shape: {}'.format(X.shape))\n",
    "print('Targets shape: {}'.format(y.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 7 ==========\n",
    "\n",
    "Now we want to train a Multinomial Naive Bayes classifier. Initialise a `MultinomialNB` object and [`fit`](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html#sklearn.naive_bayes.MultinomialNB.fit) the classifier using the `X` and `y` arrays extracted in the cell above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "mnb = MultinomialNB()\n",
    "mnb.fit(X=X, y=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can evaluate the classifier by looking at the classification accuracy, and the [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix). \n",
    "\n",
    "Scikit-learn model objects have built in scoring methods. The default [`score` method for `MultinomialNB`](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html#sklearn.naive_bayes.MultinomialNB.score) estimates the classification accuracy score. Alternatively, you can compute the prediction for the training data and make use of the [`accuracy_score`](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html) function (that is in fact what the classifier's `score()` method does under the hood).\n",
    "\n",
    "Scikit-learn also has a [`confusion_matrix`](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html#sklearn.metrics.confusion_matrix) implementation which returns a numpy array (square matrix) of dimensionality `K`, where `K` is the number of classes (2 in our case)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 8 ========== "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** Display the log-prior probabilities for each class. *Hint:* use tab-completion to figure out which attribute of the `MultinomialNB` structure you are interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here \n",
    "'Class log-priors: {}'.format(mnb.class_log_prior_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Predict the output of the classifier by using the training data as input. *Hint*: make use of the `predict` method of the `MultinomialNB` classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here \n",
    "tr_pred = mnb.predict(X=X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** Compute the classification accuracy on the training data by either using the `accuracy_score` metric or the `score` method of the `MultinomialNB`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here \n",
    "ca = accuracy_score(y, mnb.predict(X)) # or ca = gnb.score(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d)** Compute the resulting confusion_matrix by using the builtin scikit-learn class and display the result. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here \n",
    "cm = confusion_matrix(y, tr_pred)\n",
    "cm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e)** Normalise the produced confusion matrix by the true class and display the result. In other words, the matrix should show you what proportion of `Ham` emails were predicted as `Ham`/`Spam` and vice versa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here \n",
    "cm_norm = cm/cm.sum(axis=1)[:, np.newaxis]\n",
    "cm_norm\n",
    "## Confusion matrix has values c_ij such that true label i \n",
    "## is predicted as label j, i.e. rows should sum to 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f)** By making use of the `plot_confusion_matrix` provided below, visualise the normalised confusion matrix. Plot the appropriate labels on both axes by making use of the `classes` input argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(cm, classes=None, title='Confusion matrix'):\n",
    "    \"\"\"Plots a confusion matrix.\"\"\"\n",
    "    if classes is not None:\n",
    "        sns.heatmap(cm, xticklabels=classes, yticklabels=classes, vmin=0., vmax=1., annot=True)\n",
    "    else:\n",
    "        sns.heatmap(cm, vmin=0., vmax=1.)\n",
    "    plt.title(title)\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm_norm, classes=['ham', 'spam'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 9 ==========\n",
    "\n",
    "Study the output produced, most importantly the percentages of correctly and incorrectly classified instances. You probably will notice that your classifer does rather well despite making a very strong assumption on the form of the data. If we didn't make this assumption, what would be the main practical problems? *Hint*: If you've forgotten the assumption of the Naive Bayes model, check wikipedia and/or sklearn documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here:*\n",
    "\n",
    "The classifier is doing well, but it's only been tested so far on the same data we used for training it, so we can't be sure that it can generalise to new examples. \n",
    "\n",
    "The main practical problem if we didn't make the Naive Bayes assumption (conditional independence given the label) is that we would have to estimate a full covariance matrix of size 55 X 55 (i.e. ~1500 parameters) and we only have 4000 samples, so the covariance estimate might be dominated by noise. Assuming conditional independence allows us to esimate a diagonal covariance matrix i.e. estimate a variance for each variable independently and assume all covariances between distinct variables are 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 10 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The empirical log probability of input features given a class $P\\left(x_i  |  y\\right)$ is given by the attribute `feature_log_prob` of the classifier. For each feature there are two such conditional probabilities, one for each class. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** What dimensionality do you expect the `feature_log_prob_` array to have? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here:*\n",
    "\n",
    "There is a probability for each feature(/variable) conditional on each of the two outcome values, so the dimensionality should logically be (54,2) or (2,54)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Inspect the log probabilities of the features. Verify that it has the expected dimensionality (i.e. `shape`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "mnb.feature_log_prob_.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** Create a list of the names of the features that have higher log probability when the email is `Ham` than `Spam` i.e. what features imply an email is more likely to be `Ham`? *Hint:* There are a many ways to do this. Try it on your own then, if you get stuck, you can do it using index numbers (look up [`np.argwhere`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.argwhere.html)), or using a boolean mask (look up [pandas indexing](http://pandas.pydata.org/pandas-docs/stable/indexing.html)). The column names of a Pandas DataFrame are contained in the `columns` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "\n",
    "feature_names = X.columns\n",
    "hammy_bool = mnb.feature_log_prob_[0] > mnb.feature_log_prob_[1]\n",
    "\n",
    "## Simple boolean selection\n",
    "ham_features = feature_names[hammy_bool].tolist()\n",
    "print(ham_features)\n",
    "\n",
    "## With np.argwhere\n",
    "idx1 = np.argwhere(hammy_bool).squeeze().tolist()\n",
    "print(feature_names[idx1].tolist())\n",
    "\n",
    "## with list comprehension \n",
    "idx2 = [ii for ii, x in enumerate(hammy_bool) if x]\n",
    "print(feature_names[idx2].tolist())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 11 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the final part of this section we will now pretend we are spammers wishing to fool a spam checking system based on Naïve Bayes into classifying a spam e-mail as ham (i.e. a valid e-mail). For this we will use a test set consisting of just one data point (i.e. e-mail). This tiny dataset is called `spambase_test` and has already been pre-processed for you which means that the redundant attributes have been removed and word frequencies have been replaced by word presence/absence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** Load `./datasets/spambase_test.csv` dataset into a new pandas structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "data_path_test = os.path.join(os.getcwd(), 'datasets', 'spambase_test.csv')\n",
    "spambase_test = pd.read_csv(data_path_test, delimiter = ',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Use `spambase_test` to create a pandas DataFrame object X_test, contatining the test features, and pandas Series object y_test, containing the test outcome"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "X_test = spambase_test.drop('is_spam', axis=1)\n",
    "y_test = spambase_test['is_spam'] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** Feed the input features into the classifier and compare the outcome to the true label. Make sure you don't feed the target into the classifier as you will receive an error (why?). Does the classifer classify the spam e-mail correctly?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "prediction = mnb.predict(X_test)\n",
    "print('Actual label: {}\\t Prediction: {}'.format(int(y_test), int(prediction)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d)** Pick one (perhaps random) attribute that has higher probability for the ham class (using your feature names in Question 10c) and set the corresponding value in `X_test` to 1. Now predict the new outcome. Has it changed? If not, keep modifying more attributes until you have achieved the desired outcome (i.e. model classifies the e-mail as ham)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "#3. Loop until prediction is ham\n",
    "for feat_name in ham_features:\n",
    "    # Multiple indexing methods are available for Pandas\n",
    "#     X_test.iloc[0][feat_name] = 1\n",
    "#     X_test.loc[0][feat_name] = 1\n",
    "    X_test.ix[0, feat_name] = 1\n",
    "    prediction = mnb.predict(X_test)\n",
    "    if prediction != y_test.iloc[0]:\n",
    "        print('The word \"', feat_name[10:], '\" did the trick!')\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 12 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**This is an extension for people keen to learn more advanced plotting.** We'll be happy to discuss your conclusions in the lab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** Create a plot of the spam/ham log probabilities for all of the features. This will help you find the spammiest/hammiest words to use in your emails! *Hint*: you can do this however you like, but try 'adapting' [this matplotlib demo](http://matplotlib.org/examples/api/barchart_demo.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "log_probs = mnb.feature_log_prob_\n",
    "ticklabs = feature_names\n",
    "\n",
    "plt.figure(figsize=(12,6))\n",
    "plt.plot(log_probs.T)\n",
    "plt.xticks(range(len(feature_names)), \n",
    "           [f[10:] for f in ticklabs],\n",
    "           rotation='vertical')\n",
    "plt.show()\n",
    "\n",
    "# or...more fancily\n",
    "import matplotlib as mpl\n",
    "colour_palette = mpl.rcParams['axes.prop_cycle'].by_key()['color']\n",
    "N = len(ticklabs)\n",
    "ind = np.arange(N)  # the x locations for the groups\n",
    "width = 0.35        # the width of the bars\n",
    "fig, ax = plt.subplots(figsize=(12,6))\n",
    "ham_rects = ax.bar(ind, log_probs[0], width, color=colour_palette[0])\n",
    "spam_rects = ax.bar(ind + width, log_probs[1], width, color=colour_palette[1])\n",
    "ax.set_ylabel('Log Probability')\n",
    "ax.set_xticks(ind + width)\n",
    "ax.set_xticklabels([f[10:] for f in ticklabs], rotation='vertical')\n",
    "ax.legend((ham_rects[0], spam_rects[0]), ('Ham', 'Spam'), loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** The features are in the order they appear in the dataset. Can you order them by probability of being `Ham`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "sort_idx = np.argsort(mnb.feature_log_prob_[0])\n",
    "log_probs = mnb.feature_log_prob_[:, sort_idx]\n",
    "ticklabs = feature_names[sort_idx]\n",
    "\n",
    "\n",
    "plt.figure(figsize=(12,6))\n",
    "plt.plot(log_probs.T)\n",
    "plt.xticks(range(len(feature_names)), \n",
    "           [f[10:] for f in ticklabs],\n",
    "           rotation='vertical')\n",
    "plt.show()\n",
    "\n",
    "# or...more fancily\n",
    "import matplotlib as mpl\n",
    "colour_palette = mpl.rcParams['axes.prop_cycle'].by_key()['color']\n",
    "N = len(ticklabs)\n",
    "ind = np.arange(N)  # the x locations for the groups\n",
    "width = 0.35        # the width of the bars\n",
    "fig, ax = plt.subplots(figsize=(12,6))\n",
    "ham_rects = ax.bar(ind, log_probs[0], width, color=colour_palette[0])\n",
    "spam_rects = ax.bar(ind + width, log_probs[1], width, color=colour_palette[1])\n",
    "ax.set_ylabel('Log Probability')\n",
    "ax.set_xticks(ind + width)\n",
    "ax.set_xticklabels([f[10:] for f in ticklabs], rotation='vertical')\n",
    "ax.legend((ham_rects[0], spam_rects[0]), ('Ham', 'Spam'), loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** What about ordering by the absolute difference between `Ham` and `Spam` log probability?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "sort_idx = np.argsort(abs(mnb.feature_log_prob_[0]-mnb.feature_log_prob_[1]))\n",
    "log_probs = mnb.feature_log_prob_[:, sort_idx]\n",
    "ticklabs = feature_names[sort_idx]\n",
    "\n",
    "\n",
    "plt.figure(figsize=(12,6))\n",
    "plt.plot(log_probs.T)\n",
    "plt.xticks(range(len(feature_names)), \n",
    "           [f[10:] for f in ticklabs],\n",
    "           rotation='vertical')\n",
    "plt.show()\n",
    "\n",
    "# or...more fancily\n",
    "import matplotlib as mpl\n",
    "colour_palette = mpl.rcParams['axes.prop_cycle'].by_key()['color']\n",
    "N = len(ticklabs)\n",
    "ind = np.arange(N)  # the x locations for the groups\n",
    "width = 0.35        # the width of the bars\n",
    "fig, ax = plt.subplots(figsize=(12,6))\n",
    "ham_rects = ax.bar(ind, log_probs[0], width, color=colour_palette[0])\n",
    "spam_rects = ax.bar(ind + width, log_probs[1], width, color=colour_palette[1])\n",
    "ax.set_ylabel('Log Probability')\n",
    "ax.set_xticks(ind + width)\n",
    "ax.set_xticklabels([f[10:] for f in ticklabs], rotation='vertical')\n",
    "ax.legend((ham_rects[0], spam_rects[0]), ('Ham', 'Spam'), loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}