protein_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "65KuEOQYX0oI"
   },
   "source": [
    "# Human Protein Atlas Image Classification\n",
    "\n",
    "\n",
    "Image classification of microscope slides based on mixed protein patterns.\n",
    "\n",
    "### Project Description\n",
    "\n",
    "The objective for this project is to determine the locations of protein organelles present in the microscope images. This breaks down into two parts: first, identifying the protein location (in general) from the image, and second, labelling each organelle within the protein.\n",
    "\n",
    "In particular we aim to build a model that can reliably make predictions even when the images contain a mixture of different cell types with different morphologies. \n",
    "\n",
    "### Task \n",
    "\n",
    "The problem is a multi-label image classification task. Each image will contain a mixture of different cell types there are 28 in total. Therefore, the CNN must be able to predict for each image one or many of the 28 labels. \n",
    "\n",
    "The training images that was provided by the kaggle competition includes a train_csv file that contains a list of image ids with the identified protein labels. \n",
    "\n",
    "\n",
    "### Requirements\n",
    "\n",
    "- keras\n",
    "- tensorflow\n",
    "- numpy\n",
    "- pydot\n",
    "- pandas\n",
    "- OpenCV (opencv-python)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 83
    },
    "colab_type": "code",
    "id": "RjD80xLQX0oT",
    "outputId": "ce65eeed-4dc2-495b-aedc-87bc428f6b1c"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "# libraries\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "from keras.layers import Dense, Activation\n",
    "from keras.layers import Input, Flatten, Dropout\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "from keras.layers.convolutional import Conv2D\n",
    "from keras.layers.convolutional import MaxPooling2D\n",
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "from keras.preprocessing.image import img_to_array\n",
    "from keras.preprocessing.image import save_img\n",
    "from keras.callbacks import ModelCheckpoint\n",
    "from keras.optimizers import Adam\n",
    "from keras.optimizers import SGD\n",
    "from keras.models import load_model\n",
    "from keras import backend\n",
    "\n",
    "from sklearn.preprocessing import MultiLabelBinarizer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.utils import class_weight\n",
    "\n",
    "from PIL import Image as pil_image # installed by \"pip install Pillow\"\n",
    "\n",
    "import cv2                      # openCV image processing\n",
    "import numpy as np              # linear algebra\n",
    "import pandas as pd             # data processing\n",
    "import matplotlib.pyplot as plt # graphing/visualization\n",
    "import h5py                     # object serialization\n",
    "\n",
    "import os\n",
    "import gc\n",
    "import random\n",
    "import csv\n",
    "import sys\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g8IT9AOJX0os"
   },
   "source": [
    "### Labels\n",
    "All of the 28 labels for identifying the protien. We are predicting protein organelle localization labels for each sample. The dataset was acquired in a standardized way using one imaging modality (confocal microscopy). All image samples are represented by four filters (stored as individual files), the protein of interest (green) plus three cellular landmarks: nucleus (blue), microtubules (red), endoplasmic reticulum (yellow). The green filter should hence be used to predict the label, and the other filters are used as references."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3xCNdgSFX0ow"
   },
   "outputs": [],
   "source": [
    "NUCLEOPLASM                   = 0\n",
    "NUCLEAR_MEMBRANE              = 1\n",
    "NUCLEOLI                      = 2\n",
    "NUCLEOLI_FIBRILLAR_CENTER     = 3\n",
    "NUCLEAR_SPECKLES              = 4\n",
    "NUCLEAR_BODIES                = 5\n",
    "ENDOPLASMIC_RETICULUM         = 6\n",
    "GOLGI_APPARATUS               = 7\n",
    "PEROXISOMES                   = 8\n",
    "ENDOSOMES                     = 9\n",
    "LYSOSOMES                     = 10\n",
    "INTERMEDIATE_FILAMENTS        = 11\n",
    "ACTIN_FILAMENTS               = 12\n",
    "FOCAL_ADHESION_SITES          = 13\n",
    "MICROTUBULES                  = 14\n",
    "MICROTUBULE_ENDS              = 15\n",
    "CYTOKINETIC_BRIDGE            = 16\n",
    "MITOTIC_SPINDLE               = 17\n",
    "MICROTUBULE_ORGANIZING_CENTER = 18\n",
    "CENTROSOME                    = 19\n",
    "LIPID_DROPLETS                = 20\n",
    "PLASMA_MEMBRANE               = 21\n",
    "CELL_JUNCTIONS                = 22\n",
    "MITOCHONDRIA                  = 23\n",
    "AGGRESOME                     = 24\n",
    "CYTOSOL                       = 25\n",
    "CYTOPLASMIC_BODIES            = 26\n",
    "RODS_AND_RINGS                = 27"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cy6hSk4LX0o7"
   },
   "source": [
    "### Some Constants\n",
    "Here we define some constants that will define: the local datapath of the training, testing, and label data. We will also define some fundamental inputs to our model, such as: the number of epochs, learning rate, batch size, and the size of the subset we wish to train our data on. As we are constrained by our hardware, we will use a subset of the training data. Finally, we define a random.seed so we maintain the random numbers chosen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "o8C04NDFX0pA"
   },
   "outputs": [],
   "source": [
    "LABELS_DIR = 'dataset'\n",
    "TRAIN_DIR = 'dataset/train'\n",
    "TEST_DIR = 'dataset/test'\n",
    "TRAIN_SUBSET_SIZE = 1000 # Selection of images to train from (chosen at random, max size is 31072)\n",
    "LEN_LABELS = 28\n",
    "\n",
    "EPOCHS = 15 # epochs to train for (the more the better)\n",
    "INIT_LR = 1e-3 # initial learning rate for the solver\n",
    "BS = 32 # batch size\n",
    "IMG_SIZE = (512, 512, 1) # 512 by 512 pixels and only one channel (black and white)\n",
    "\n",
    "DOWNSAMPLING=True\n",
    "\n",
    "# random seed\n",
    "random.seed(56732)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AvfKbFHBX0pK"
   },
   "source": [
    "### Image Loading\n",
    "\n",
    "The first step is to take our dataset and prepare it so that it's ready for our model.\n",
    "\n",
    "Images are split into four filters/layers:\n",
    "- **green**: the protein of interest\n",
    "- **blue**: the nucleus\n",
    "- **red**: the microtubules\n",
    "- **yellow**: the endoplasmic reticulum\n",
    "\n",
    "For this project we will mostly be interested in the **green** filter, which will be used to predict the label, while the other filters will be used as references."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 243
    },
    "colab_type": "code",
    "id": "cq-iRTUlX0pN",
    "outputId": "cdc15b8e-dcea-44e7-b952-b4120c8a1439"
   },
   "outputs": [],
   "source": [
    "# Get images by layer\n",
    "\n",
    "# training set images\n",
    "train_green  = [(TRAIN_DIR+'/{}').format(i) for i in os.listdir(TRAIN_DIR) if 'green'  in i]\n",
    "train_green_all = train_green.copy() # for class_weights later\n",
    "\n",
    "# Select TRAIN_SUBSET_SIZE images from the training set at random\n",
    "random.shuffle(train_green)\n",
    "train_green = train_green[:TRAIN_SUBSET_SIZE]\n",
    "train_green_ids = [i[:-10].replace((TRAIN_DIR+'/'),'') \n",
    "                       for i in train_green] # remove '_green.png' and '_green.tif'\n",
    "\n",
    "# Retrieve the other three layers for our subset (NB: unsorted)\n",
    "train_blue =   [(TRAIN_DIR+'/{}').format(i) for i in os.listdir(TRAIN_DIR)\n",
    "                  if ((i[:-9] in train_green_ids) and ('blue' in i))]\n",
    "train_red =    [(TRAIN_DIR+'/{}').format(i) for i in os.listdir(TRAIN_DIR)\n",
    "                  if ((i[:-8] in train_green_ids) and ('red' in i))]\n",
    "train_yellow = [(TRAIN_DIR+'/{}').format(i) for i in os.listdir(TRAIN_DIR)\n",
    "                  if ((i[:-11] in train_green_ids) and ('yellow' in i))]\n",
    "\n",
    "# force garbage collection to make sure memory isn't wasted\n",
    "gc.collect()\n",
    "\n",
    "# test set images\n",
    "test_green  = [(TEST_DIR+'/{}').format(i) for i in os.listdir(TEST_DIR) if 'green'  in i]\n",
    "test_blue   = [(TEST_DIR+'/{}').format(i) for i in os.listdir(TEST_DIR) if 'blue'   in i]\n",
    "test_red    = [(TEST_DIR+'/{}').format(i) for i in os.listdir(TEST_DIR) if 'red'    in i]\n",
    "test_yellow = [(TEST_DIR+'/{}').format(i) for i in os.listdir(TEST_DIR) if 'yellow' in i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YhGwPb54X0pV"
   },
   "outputs": [],
   "source": [
    "# Testing the above by seeing if the IDs are the same\n",
    "\n",
    "train_green.sort()\n",
    "train_blue.sort()\n",
    "train_red.sort()\n",
    "train_yellow.sort()\n",
    "#print(train_green, end='\\n\\n')\n",
    "#print(train_blue, end='\\n\\n')\n",
    "#print(train_red, end='\\n\\n')\n",
    "#print(train_yellow, end='\\n\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LXf6fgfVjvH0"
   },
   "source": [
    "### Label Loading\n",
    "\n",
    "Loading the labels from *train.csv*. Here we seek to obtain all the labels for the train and test data. These labels could be multi-label, as in indicating more then one protein may be present in any data sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dtNBfjMfjxV2"
   },
   "outputs": [],
   "source": [
    "train_labels = {} # dictionary with key = photo ID and entry = list of labels\n",
    "train_labels_all = {} # this will for calculating sample_weights later\n",
    "\n",
    "# Get all the labels\n",
    "with open('dataset/train.csv') as label_file:\n",
    "    csvreader = csv.reader(label_file, delimiter=',', quotechar='|')\n",
    "    for row in csvreader:\n",
    "        if \"Id\" not in row[0]:\n",
    "            # only take the ones that were taken in the Image Loading part\n",
    "            if any(row[0] in x for x in train_green):\n",
    "                train_labels[row[0]] = row[1:][0].split(' ') # labels are separated by spaces\n",
    "            if DOWNSAMPLING:\n",
    "                if any(row[0] in x for x in train_green_all):\n",
    "                    train_labels_all[row[0]] = row[1:][0].split(' ') # labels are separated by spaces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8TTKKnnkewLO"
   },
   "source": [
    "### Image Preprocessing\n",
    "We now move onto some image preprocessing after reading the data image .csv files into lists. We will now define a function to produce the input labels and output labels in a unique vector style called one hot encoding. We will also load all the training data into a pandas.dataframe with their labels. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Nhr-zmArr_Ag"
   },
   "source": [
    "#### One Hot Encoding\n",
    "One hot encoding is a process by which categorical variables are converted into a form that could be provided to ML algorithms to do a better job in prediction.The categorical value represents the numerical value of the entry in the dataset. For example, if a training example is labelled as having NUCLEOLI_FIBRILLAR_CENTER (=3) and ENDOPLASMIC_RETICULUM  (=6), the one_hot_encoded label would be: [0 0 0 1 0 0 1 0....(28 places in total)]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kV7Weri7sRWK"
   },
   "outputs": [],
   "source": [
    "def one_hot_encode(train_tags):\n",
    "    ''' Creates a one hot encoded dictionary of all labels for each pic'''\n",
    "    encoded = dict()\n",
    "  \n",
    "    for key in train_tags.keys():\n",
    "        # create empty vector\n",
    "        encoding = np.zeros(LEN_LABELS, dtype='uint8')\n",
    "        # mark 1 for each tag in the vector\n",
    "        for tag in train_tags[key]:\n",
    "            encoding[int(tag)] = 1\n",
    "            \n",
    "        encoded[key] = tuple(encoding)\n",
    "  \n",
    "    return encoded"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load into Pandas\n",
    "We now load every data example, and its corresponding one-hot encoded label into a pandas.dataframe for easier reference later when recalling the label as an answer. We model the training set in this way when providing answers for training the model and testing the validation of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Photo ID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>003feb6e-bbca-11e8-b2bc-ac1f6b6435d0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>09599842-bbc7-11e8-b2bc-ac1f6b6435d0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1263dcf2-bba1-11e8-b2b9-ac1f6b6435d0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17c12330-bbc9-11e8-b2bc-ac1f6b6435d0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1a7de584-bbad-11e8-b2ba-ac1f6b6435d0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1ec3408c-bbb5-11e8-b2ba-ac1f6b6435d0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1edd9a1e-bbb4-11e8-b2ba-ac1f6b6435d0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>233a0498-bbc9-11e8-b2bc-ac1f6b6435d0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2664d9c8-bba5-11e8-b2ba-ac1f6b6435d0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27e40a9e-bba0-11e8-b2b9-ac1f6b6435d0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      0   1   2   3   4   5   6   7   8   9   \\\n",
       "Photo ID                                                                       \n",
       "003feb6e-bbca-11e8-b2bc-ac1f6b6435d0   1   0   0   0   0   0   0   0   0   0   \n",
       "09599842-bbc7-11e8-b2bc-ac1f6b6435d0   0   0   1   0   0   0   0   0   0   0   \n",
       "1263dcf2-bba1-11e8-b2b9-ac1f6b6435d0   1   1   0   0   0   0   0   0   0   0   \n",
       "17c12330-bbc9-11e8-b2bc-ac1f6b6435d0   0   0   1   0   0   0   0   0   0   0   \n",
       "1a7de584-bbad-11e8-b2ba-ac1f6b6435d0   0   0   0   0   0   0   0   0   0   0   \n",
       "1ec3408c-bbb5-11e8-b2ba-ac1f6b6435d0   1   0   1   0   0   0   0   0   0   0   \n",
       "1edd9a1e-bbb4-11e8-b2ba-ac1f6b6435d0   0   0   0   0   0   0   0   1   0   0   \n",
       "233a0498-bbc9-11e8-b2bc-ac1f6b6435d0   0   0   0   0   0   0   0   0   0   0   \n",
       "2664d9c8-bba5-11e8-b2ba-ac1f6b6435d0   1   0   1   0   0   0   0   0   0   0   \n",
       "27e40a9e-bba0-11e8-b2b9-ac1f6b6435d0   1   0   0   0   0   0   0   0   0   0   \n",
       "\n",
       "                                      ...  18  19  20  21  22  23  24  25  26  \\\n",
       "Photo ID                              ...                                       \n",
       "003feb6e-bbca-11e8-b2bc-ac1f6b6435d0  ...   0   0   0   0   0   0   0   0   0   \n",
       "09599842-bbc7-11e8-b2bc-ac1f6b6435d0  ...   0   0   0   0   0   0   0   1   0   \n",
       "1263dcf2-bba1-11e8-b2b9-ac1f6b6435d0  ...   0   0   0   0   0   0   0   0   0   \n",
       "17c12330-bbc9-11e8-b2bc-ac1f6b6435d0  ...   0   0   0   0   0   1   0   0   0   \n",
       "1a7de584-bbad-11e8-b2ba-ac1f6b6435d0  ...   0   0   0   0   0   0   0   1   0   \n",
       "1ec3408c-bbb5-11e8-b2ba-ac1f6b6435d0  ...   0   0   0   0   0   0   0   0   0   \n",
       "1edd9a1e-bbb4-11e8-b2ba-ac1f6b6435d0  ...   0   0   0   0   0   0   0   0   0   \n",
       "233a0498-bbc9-11e8-b2bc-ac1f6b6435d0  ...   0   0   0   0   0   0   0   0   0   \n",
       "2664d9c8-bba5-11e8-b2ba-ac1f6b6435d0  ...   0   0   0   0   0   0   0   1   0   \n",
       "27e40a9e-bba0-11e8-b2b9-ac1f6b6435d0  ...   0   0   0   0   0   0   0   1   0   \n",
       "\n",
       "                                      27  \n",
       "Photo ID                                  \n",
       "003feb6e-bbca-11e8-b2bc-ac1f6b6435d0   0  \n",
       "09599842-bbc7-11e8-b2bc-ac1f6b6435d0   0  \n",
       "1263dcf2-bba1-11e8-b2b9-ac1f6b6435d0   0  \n",
       "17c12330-bbc9-11e8-b2bc-ac1f6b6435d0   0  \n",
       "1a7de584-bbad-11e8-b2ba-ac1f6b6435d0   0  \n",
       "1ec3408c-bbb5-11e8-b2ba-ac1f6b6435d0   0  \n",
       "1edd9a1e-bbb4-11e8-b2ba-ac1f6b6435d0   0  \n",
       "233a0498-bbc9-11e8-b2bc-ac1f6b6435d0   0  \n",
       "2664d9c8-bba5-11e8-b2ba-ac1f6b6435d0   0  \n",
       "27e40a9e-bba0-11e8-b2b9-ac1f6b6435d0   0  \n",
       "\n",
       "[10 rows x 28 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ohe_labels = one_hot_encode(train_labels)\n",
    "\n",
    "if DOWNSAMPLING:\n",
    "    ohe_all_labels = pd.DataFrame.from_dict(one_hot_encode(train_labels_all), orient='index')\n",
    "    # print(ohe_all_labels.astype(bool).sum(axis=0))\n",
    "\n",
    "training_set = pd.DataFrame.from_dict(ohe_labels, orient='index')\n",
    "training_set.index.name = \"Photo ID\"\n",
    "training_set.reset_index()\n",
    "training_set.head(n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Features Visualization (using matplotlib)\n",
    "\n",
    "As we can see the training set is highly imbalanced.\n",
    "This may be indicative of the rarity of certain protein types.\n",
    "\n",
    "Data imbalance is a well-known problem in Machine Learning. Where some classes in the dataset are more frequent than others, and the neural net just learns to predict the frequent classes.\n",
    "\n",
    "This is a big issue, to mitigate in this case, we can easily balance the data using sampling techniques.\n",
    "By removing some frequent examples(downsampling)\n",
    "By creating more in-frequent examples(upsampling) using image augmentation or any other method.\n",
    "\n",
    "Perhaps a larger data set will show more balanced results!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1ffeb39e1d0>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de9xlc93/8dfbYMZpCIMxg6HbKKcGFymFkDRC6YBUE2pKSrpTiftXOt23VDrpUQ3DILlTJIeKuR0rhWs0zGhQGBkzmSlmyDjN+Pz+WN+Lbc8+rGtfe+21x34/H4/9uPZee63v9zOrfK91fddnfb6KCMzMrHesUnYAZmbWWR74zcx6jAd+M7Me44HfzKzHeOA3M+sxHvjNzHqMB36zIZB0pKRr2tjeXZL2Tu9PlfSTNrZ9sqSz29Werbw88FvbSZor6SlJ/654bTrENveWNK9dMebsc5qkZyU9kV6zJf2PpHUH9omICyNi/5xtfbXZfhGxXUTcMMTQa56viPjviPjQUNu2lZ8HfivKQRGxdsVrfpnBSFq1xUNPj4h1gFHAUcDuwB8krdW24BhSfGaD5oHfOkrS7pJulrRY0h0D0xrpu6MkzUlX1/dL+kjavhbwG2DTyr8gqq+iq69y018en5N0J/CkpFXTcZdIWiTpAUnH54k7Ip6OiNuAg4ENyH4JIOmDkn6f3kvStyUtlLRE0p2Stpc0GTgS+GyK/YoG8c2VtF9F1yMk/Sydk9slvabi3xeS/qPi8zRJX21wvl4ydSTp4DS1tFjSDZJeXXXuTkz/hiUphhF5zpV1Pw/81jGSxgBXAV8F1gdOBC6RNCrtshB4GzCSbGD9tqSdI+JJ4K3A/Bb+gjgCOBBYD3geuAK4AxgD7AucIOktef8NEfEEMB14Y42v9wf2BMan/g4D/hURU4ALyf56WDsiDqoVX0Qsq9HmIcDPyc7XT4HLJK3WJMam50vSeOAi4ASyv2Z+DVwhafWK3d4DHABsCewIfLBRv7by8MBvRbksXUkulnRZ2vY+4NcR8euIeD4ipgP9wESAiLgqIu6LzI3ANdQeYAfjexHxUEQ8BewKjIqIL0fEsxFxP3AWcPgg25xPNhBXew5YB3gVoIiYExELBhFfLTMi4hcR8RxwBjCCbLppqA4DroqI6antbwJrAK+vim1+RDxK9gtzQhv6tS7geUUrytsj4v+qtm0BvFtS5RXvasD1AJLeCnyR7Ip5FWBNYNYQ43ioqv9NJS2u2DYM+N0g2xwDPFq9MSKuk3Qm8ANgc0m/BE6MiMdzxtfw+4h4Pk1lDelGebIp8GBV2w+R/dsG/KPi/dI29WtdwFf81kkPARdExHoVr7Ui4jRJw4FLyK48N46I9cimH5SOrVVG9kmyXw4DNqmxT+VxDwEPVPW/TkRMzPsPkLQ2sB91fllExPciYhdgO7JfYJ9pEH+j7QM2q+h7FWAs2V8ckA3G9f79zdqdT/aLcKBtpb4ebnKcvQx44LdO+glwkKS3SBomaUS6ITsWWB0YDiwClqWr/8o0yUeADSpTKYGZwERJ60vahGy+upFbgcfTDdU1UgzbS9q1WeCShkvaBbgMeAw4t8Y+u0p6bZqDfxJ4GlheEf9WzfqpYRdJh6asnxOAZ4A/pe9mAu9N/44DgL0qjqt1vipdDBwoad8U76dT2ze3EKOtZDzwW8dExENkNytPJhvgHyK7Il4l3TQ9nmxAegx4L3B5xbF3k92MvD/dN9gUuIDsRu1csvsBP2vS/3LgILK56geAfwJnA/UGR8gycZ4gm9o5H5gBvD7dQK02kuyewWNk0yj/IvsLBmAqsG3VPY88fkU2H/8Y8H7g0DQnD/DJ9O9ZTJY19EK7dc4XFd/fQ3bP5ftk5+EgshTcZwcRm62k5IVYzMx6i6/4zcx6jAd+M7Me44HfzKzHeOA3M+sxK8UDXBtuuGGMGzeu7DDMzFYqM2bM+GdEjKreXtjAL2kzsvS3TchqpEyJiO9K+hmwTdptPWBxRDR8FHzcuHH09/cXFaqZ2cuSpAdrbS/yin8Z8OmIuF3SOsAMSdMj4rCKoL4FLCkwBjMzq1LYwJ+KUy1I75+QNIesDshf4IVHxN8D7FNUDGZmtqKO3NyVNA7YCbilYvMbgUci4q91jpksqV9S/6JFi4oP0sysRxQ+8KeiVpcAJ1RVKTyC7JHymiJiSkT0RUTfqFEr3JswM7MWFZrVk4o/XQJcGBGXVmxfFTgU2KXI/s3MbEWFXfGnOfypwJyIOKPq6/2AuyOio4tnm5lZsVM9e5BVE9xH0sz0Gqh7fjgNpnnMzKw4RQ78DwI3kK2wtBpwbkT8On03A/hUWuj59AJjMDOzKh3P4wc2JqvJvmNEPCNpowJjMDOzKmXk8X8YOC0inknfLSwqBjMzW1EZefzjgTdKukXSjfWWvXMev5lZMcrI418VeAWwO9myexenDKCXcB6/mVkxCh346+TxzwMujcytZAXcNiwyDjMze1EZefyXkerzSBoPrE622LOZmXVAkVk9A3n8syTNTNtOBs4BzpE0G3gWmBRe8d3MrGPKyOM/GXgTWbrnKsCIAmMwM7MqZeTxA3w7Ir6Zt6FZDy9h3ElXFRKkmTU397QDyw7B2qiMPH4zMytRWfX4Py7pTknnSHpFnWNeyONfvtSLdJmZtUsZefw/BF4JTCD7i+BbtY6rzOMftua6RYdpZtYzOl6PPyIeqfj+LODKZu3sMGZd+j3HaGbWFh3P45c0umK3dwCzi4rBzMxWVOQV/zvI8vifkfQR4F/AZOAISROADYDRwPYFxmBmZlWKnOP/I7BLRIwANgKWAnMj4v3ARGAW8HfgkfpNmJlZu5WRzvkX4NvAZ4Ff5WmrWR6/c4zNzPLreDqnpIOBhyPijk70bWZmL1VoVg+8NJ2T7GneU4D9cxw3meyeAMNGuiyzmVm7qMj6aCmd80rg6og4Q9IOwLVk8/0AY4H5wG4R8Y967fT19UV/f39hcZqZvRxJmhERfdXbC7vir5XOGRGzyG70DuwzF+iLCJdlNjPrkCLn+AfKMu8jaWZ6TSywPzMzy6HjZZklfSXV6ZkJ3Eu2EIuZmXVIYXP86Qnd0ZVlmYG3A/NSzR4kHQ9sGxEfbdTW8NFbx+hJ3ykkTqvNKbJmK796c/yFXfFHxIKIuD29fwKYA4wZGPSTtQCvvmVm1kGFp3PCimWZJX0N+ACwhGw1rlrHOJ3TzKwAZZRlJiJOiYjNgAuBj9c6zmWZzcyK0dE8/hrfbwFcFRENC7U5j9/MbPA6PsffoCzz1hW7HQzcXVQMZma2oiLn+Afy+Gel1E2Ak4FjJG0DPE+W8tkwo8fMzNqr43n8wP28+AtnGfBkgTGYmVmVMvL4xwLXRcQySV8HiIjPNWqrm/P4ne9uZt2qm/L4r4mIZWm3P5H9IjAzsw7peD3+qq+OBn5T55jJkvol9S9fuqTYAM3Mekgpefxp+ylkc/wX1jrOefxmZsUo9MndlMd/CXBhRFxasX0S8DZg38hxk2GHMevS77l0M7O26Gg9/rT9AOBzwF4RsbTe8WZmVowy8vi/BwwHpme/G/hTs+qcZmbWPkUO/AN5/JuQPaw1JdXjXws4FdiRbMlF12IwM+ugIgf+ZcCnK/P4JU0HZgOHAj/O29Csh5cw7qSrCgqzN/n5A7PeVdjAHxELgAXp/ROSBvL4pwOkaR4zM+uwsvP4Gx3jPH4zswKUlsffjPP4zcyKUUoe/2A5j9/MrH06Xo/fzMzKVeRUzzvI8vg/KukpSfMkTZT0fklPA3sCf5B0bYExmJlZlSIH/j8Cu0TECGAjYCkwF9gBODUiVgG+CDiP38ysgzpelhk4BDgv7XYeWY1+MzPrkDLSOTdOOf4Duf4bdSIGMzPLdG06Z2Ue/6JFi4oL0MysxxQ68NdJ53wkLcs4sDzjwlrHVubxjxo1qsgwzcx6ShnpnJcDk9L7ScCviorBzMxWVEZZ5tOAiyUdA/wdeHeBMZiZWZUiB/6jgUXAKhGxI4Ck1wBXAmsDdwNHDmbe38zMhk45Vj5srWFpT+DfwPkRsX3adhtwYkTcKOloYMuI+H/N2ho+eusYPek7LcfiEsRm1oskzYiIvurtRebx3wQ8WrV5G+Cm9H468M6i+jczs9o6ksdfYTZwcHr/bmCzeju6LLOZWTE6PfAfDRwnaQawDvBsvR1dltnMrBi5bu5KWgPYPCLuGUpnEXE3sH9qczyQa/LdZZnNzNqn6RW/pIOAmcBv0+cJki5vpTNJG6WfqwD/BfyolXbMzKx1eaZ6TgV2AxYDRMRMYFyzgyRdRFahc5tUkvkY4AhJ95Klcs4Hzm0tbDMza1WegX9ZRLRyd/UpYBhwT0SMjYipwI1kmT5Lgf2AXVto18zMhiDPHP9sSe8FhknaGjgeuDnHcdOAM4HzK7adDnwpIn4jaWL6vHezhmY9vIRxJ11V93vn6ZuZ5Zfniv8TwHbAM8BFwOPACc0OqpPHH8DI9H5dsukeMzProKZX/BGxFDglvYbqBOBqSd8k+6Xz+no7SpoMTAYYNtLVOc3M2qXuwC/pCrIr9Joi4uB63zVwLPCpiLhE0nvIqnfuV6f9KcAUyEo2tNCXmZnVULdWj6S9Gh0YETc2bTxbeevKilo9S4D1IiJS2eYlETGyQRMA9PX1RX+/l+Y1MxuMerV66l7x5xnYWzAf2Au4AdgH+GsBfZiZWQONpnpmUXuqR0AMlFpucPxFZBk7G0qaB3wR+DDwXUmrAk+T5vDNzKxzGt3cfdsQ267M4x+Y6vlZ2hbAxmRz/BOG2I+ZmQ1Co6meBwfeS9oC2Doi/i/V7cmT/z+Nqjz+iDisos1vAS67aWbWYXlq9XwY+AXw47RpLHBZs+Pq5PEPtCngPWTPBZiZWQfleYDrOLL1cx8HiIi/AhsNsd83Ao+ktmqqrMe/aNGiIXZnZmYD8gz8z0TEC3Xz043ZoebVH0GTq/3KevyjRvkBLjOzdskzV3+jpJOBNSS9GfgYcEWrHaZfHIcCu7TahpmZtS7PFf9JwCJgFvAR4NdktfRbtR9wd0TMG0IbZmbWojy1ep4Hzkqv3Grl8afSzIfjm7pmZqXJk9Wzh6Tpku6VdL+kByTdn6PtWvX4AWYAn5J0l6TThxC7mZm1IM8c/1TgU2QD9vJBtD2Nqjx+SW8CDgF2jIhnBpZiNDOzzskz8C+JiN8MtuGIuCkVaat0LHBaRDyT9lk42HbNzGxo8tzcvV7SNyS9TtLOA68W+xsPvFHSLZJulFR36UXn8ZuZFSPPFf9r08/K0p5BVl2zlf5eAexOtt7uxZK2ihq1oSvr8ff19bkev5lZm+TJ6nlTG/ubB1yaBvpbJT0PbEiWLmpmZh2Q54ofSQeSrbs7YmBbRHy5hf4uI/tL4QZJ44HVgX+20I6ZmbUoTzrnj4DDyBZdF/BuYIscx90H3AdsJ2mepGOAMcCJkp4CZgI/qDXNY2Zmxclzc/f1EfEB4LGI+BLwOmCzHMcdRTaPf1dFHv9y4AsRsUZErBkRp7UcuZmZtSTPwP9U+rlU0qbAc8CWzQ5qVJbZzMzKk2fgv1LSesA3gNuBucD/DqHPj0u6U9I5kl5Rbyenc5qZFUODmWKXNBwYERG5Vs5KD3BdWbH04sZkN3MD+AowOiKObtZOX19f9Pf3547TzMxA0oyI6Kve3jSrR9KhNbYtAWYN9snbiHikoo2zgCsHc7yZmQ1dnnTOY8hu6F6fPu8N/AkYL+nLEXFB3s4kjY6IBenjO4DZg4jVzMzaIM/A/zzw6oGr9TRd80OyJ3pvAmoO/LXKMgN7S5pANtUzl6y+v5mZdVCem7vjKqdogIXA+Ih4lCzDp54VyjJHxPsjYgeyip0HNTnezMwKkOeK/3eSrgR+nj6/E7hJ0lrA4gbHTaOqLDOApM2ANwN/zxvkrIeXMO6kq/Lu3lZzTzuwlH7NzIqS54r/OOBcYAKwE9lAflxEPNmojk+DPP5vA59l6Au2m5lZCxpe8UsaBlwdEfsBlwy1M0kHAw9HxB2Smu07GZgMMGzkqKF2bWZmScOBPyKWS1oqad28ufv1SFoTOAXYP8/+lWWZh4/e2n8dmJm1SZ45/qeBWZKmA08ObIyI4wfZ1yvJSj0MXO2PBW6XtFtE/KPRgTuMWZd+z7WbmbVFnoH/qvQakoiYBbywxq6kuUBfRLgss5lZB+VZiOU8SauTLZsIWXpm0zTMWnn8qUKnmZmVKE/Jhr2B88geuBKwmaRJKWunkco8/oFaPV8BDiF7KOxesoVYzMysg/Kkc34L2D8i9oqIPYG3kKVkNjMNOKBq2zciYseImEBWp+cLgwnWzMyGLs/Av1pE3DPwISLuBVZrdlCtPP6IeLzi41o4l9/MrOPy3NztlzSVF2vyHAnMaLVDSV8DPgAsAeo+AFaZx7/55pu32p2ZmVXJc8V/LHAXcDzwSeAvwEdb7TAiTomIzYALgY832G9KRPRFRN+oUX6Ay8ysXfJk9TwDnJFe7fRTsjTRL7a5XTMzayDPFX/bSNq64uPBwN2d7N/MzPLN8bekTj3+iZK2IUvnfJAhTBmZmVlrChv4qZ3H/ypgO+BZYBkVJSDMzKwzmi62Lmk88BlgCyp+UUTEPk2O2xP4N3B+xcC/P3BdRCyT9PXUzueaBTl89NYxetJ3mu1Wl2vqm1kvanmxdbIFWH4EnAUsz9thRNwkaVzVtmsqPv4JeFfe9szMrD3yDPzLIuKHBfR9NPCzel+6Hr+ZWTHyZPVcIeljkkZLWn/gNZROJZ1CNsd/Yb19KvP4h6257lC6MzOzCnmu+Celn5+p2BbAVq10KGkS8DZg32h2gyFxPX4zs/bJ8wDXlu3qTNIBwOeAvSJiabvaNTOz/OoO/JL2iYjrJB1a6/uIuLRRw5LuA8YBq1Tk8X8V2BBYKOle4PqIcC6/mVkHNbri3wu4DjioxncBNBz4gaNYMZ3zZrKHt34MnBgR/YOO2MzMhqTuwB8RX0w/j2ql4TrpnHMA0pq7uc16eAnjThry6o8t8TMAZvZy0zSrR9LGkqZK+k36vK2kY4oOTNJkSf2S+pcvXVJ0d2ZmPSNPOuc04Gpg0/T5XuCEogIa4HROM7Ni5Enn3DAiLpb0eYBUbiH3E7zt4HROM7P2yXPF/6SkDUjLJEranWz1LDMzWwnlueL/T+By4JWS/gCMAt7d7KA6ZZkfBb6f2rhK0syIeEuLsZuZWQvyDPx3kaV2bgMIuId8fynUKsu8PjAnfTcXOHzwIZuZ2VDkGcD/GBHLIuKuiJgdEc8Bf8xx3DTggKptJwHXRsTWwLXps5mZdVCjJ3c3AcYAa0jaiexqH2AksGazhmvl8QOHkE3/AJwH3EBWwsHMzDqk0VTPW4APAmN56ULrTwAnt9jfxhGxACAiFkjaqN6OlWWZN9988xa7MzOzao2e3D0POE/SOyPikg7GNND/FGAKQF9fX64qnmZm1lyeOf5rJZ0x8BStpG9JavWJqkckjQZIPxe22I6ZmbUoz8A/lWx65z3p9Thwbov9Xc6L9f0nAb9qsR0zM2tRnnTOV0bEOys+f0nSzGYH1cnjPw24ONX6+Ts5ngcwM7P2ypWPL+kNAx8k7UGWh99QRBwREaMjYrWIGBsRUyPiXxGxL3AmsDHwO0mF1/0xM7MX5bni/yhwfsW8/mO8OF0zaJK2Bz4M7AY8C/xW0lUR8ddW2zQzs/waDvySVgG2iYjXSBoJEBGPD7HPVwN/Glh6UdKNwDuA04fYrpmZ5dBwqicingc+nt4/3oZBH2A2sKekDSStCUwENqveqbIe/6JFi9rQrZmZQb45/umSTpS0maT1B16tdphW4fo6MB34LXAHsKzGfi/U4x81alSr3ZmZWZU8c/xHp5/HVWwLYKtWO42IqWRpokj6b2Beq22ZmdngNB34I2LLdncqaaOIWChpc+BQ4HXt7sPMzGprOvBLGgF8DHgD2ZX+74AfRcTTQ+j3krS4y3PAcRHx2BDaMjOzQcgzx38+sB3ZAipnAtsCFwyx30vJfokMAz6UfrmYmVkH5Jnj3yYiXlPx+XpJd7TaoaQxwPHAthHxlKSLyRZkmdZqm2Zmll+eK/4/p3V2AZD0WuAPQ+x3VbI6/6uS1fafP8T2zMwspzwD/2uBmyXNlTSXbPWtvSTNknTnYDuMiIeBb5LV6lkALImIa6r3cx6/mVkx8kz1VC+fOCSSXkG2EteWwGLg55LeFxE/qdzP9fjNzIqRJ53zwTb3uR/wQEQsApB0KfB64CcNjzIzs7bIM9XTbn8Hdpe0piQB+wJzSojDzKwn5ZnqabfFwAbAo7yY0jnoewVmZtaajg/8EXEPsAmApGHAw8AvOh2HmVmvKuOKv9K+wH3N7iPMengJ4066qkMhrTzmnnZg2SGY2UqojDn+SocDF5Ucg5lZTylt4Je0OnAw8PM637+Qx7986ZLOBmdm9jKmiHJS5CUdQlagbf9m+/b19UV/f38HojIze/mQNCMi+qq3lznVcwSe5jEz67hSBv605OKbyap0mplZB5V1xb86cD1wi6Q5krwQi5lZh5SVzvld4LcR8a50k3fNRjsPNZ3TaY9mZi/q+MAvaSSwJ/BBgIh4Fni203GYmfWqMqZ6tgIWAedK+rOksyWtVb2T0znNzIpRxsC/KrAz8MOI2Al4EjipeqeImBIRfRHRN2zNdTsdo5nZy1YZc/zzgHkRcUv6/AtqDPyVdhizLv2epzcza4uOX/FHxD+AhyRtkzbtC/yl03GYmfWqsrJ6PgFcmDJ67geOKikOM7OeU1Ye/2XAcOB5YGxEPFZSHGZmPafMssxvioh/5tnx5VqW2c8XmFkZyi7LbGZmHVbWwB/ANZJmSJpcawfn8ZuZFaOsqZ49ImK+pI2A6ZLujoibKneIiCnAFIDho7cup3a0mdnLUCkDf0TMTz8XSvolsBtwU739ncdvZtY+HZ/qkbSWpHUG3gP7A7M7HYeZWa8q44p/Y+CXkgb6/2lE/LaEOMzMelLHB/6IuB94jaRhQD/gWvxmZh1UZh7/J4E5wMhmO5aZx+9cezN7uSlr6cWxwIHA2WX0b2bWy8rK4/8O8Fmykg01OY/fzKwYZWT1vA1YGBEzGu3nevxmZsUoY45/D+BgSROBEcBIST+JiPfVO8B5/GZm7VNGPf7PR8TYiBgHHA5c12jQNzOz9ipjsfURZE/pDifL6Hmu0zGYmfWyMm7uPgPsExGvAcYDj0navYQ4zMx6UhkPcAXw7/RxtfRqWITNefxmZu1TVh7/MEkzgYXA9IqF183MrGClDPwRsTwiJgBjgd0kbV+9j/P4zcyKUeoKXBGxGLgBOKDGd87jNzMrQBlZPaOA5yJisaQ1gP2Arzc6xnn8ZmbtU8YDXKOB81J1zlWAiyPiyhLiMDPrSWVM9TwGLCbL5hHgCXwzsw4q44p/GfDpiLg9rcQ1Q9L0iPhLvQPKTOc0q+T0Xns5KKNkw4KIuD29f4KsJv+YTsdhZtarSs3qkTQO2AlYIY/f6ZxmZsUobeCXtDZwCXBCRDxe/b3TOc3MilHK0ouSViMb9C+MiEub7e90TjOz9iljIRYBU4E5EXFGp/s3M+t1ZUz17AG8H9hH0sz0mlhCHGZmPamM6py/l3QuMLAE4wp1eszMrDilzPED04AzgfPz7NzNefzO6zazlU1Z1TlvAh4to28zs15Xah5/I87jNzMrRtcO/M7jNzMrRllz/IPiPH4zs/bp2it+MzMrRllr7l4E/BHYRtI8SceUEYeZWS8q64r/POBx4EHgzIiYWlIcZmY9p4ylF4cBPwDeDMwDbpN0uevxm5m9VFHPCZVxxb8b8LeIuD8ingX+FzikhDjMzHpSGQP/GOChis/zqLEQi/P4zcyKUcbArxrbYoUNzuM3MytEGXn884DNKj6PBeY3OsB5/GZm7VPGFf9twNaStpS0OnA4cHkJcZiZ9SRFrDDLUnynWf397wDDgHMi4mtN9n8CuKcTsbVgQ+CfZQdRh2MbvG6NCxxbq3o5ti0iYlT1xlIG/sGS1B8RfWXHUYtja023xtatcYFja5VjW5FLNpiZ9RgP/GZmPWZlGfinlB1AA46tNd0aW7fGBY6tVY6tykoxx29mZu2zslzxm5lZm3jgNzPrMV098Es6QNI9kv4m6aSy46kkaa6kWZJmSurvgnjOkbRQ0uyKbetLmi7pr+nnK7okrlMlPZzO3cz0XEfHSdpM0vWS5ki6S9In0/ZuOG/1Yiv93EkaIelWSXek2L6Utm8p6ZZ03n6WHtDsltimSXqg4rxN6HRsKY5hkv4s6cr0uZxzFhFd+SJ7uOs+YCtgdeAOYNuy46qIby6wYdlxVMSzJ7AzMLti2+nASen9ScDXuySuU4ETu+CcjQZ2Tu/XAe4Ftu2S81YvttLPHVm9rbXT+9WAW4DdgYuBw9P2HwHHdlFs04B3dcH/5/4T+ClwZfpcyjnr5it+l28ehIi4CXi0avMhZIvekH6+vaNBUTeurhARCyLi9vT+CWAOWaXYbjhv9WIrXWT+nT6ull4B7AP8Im0v67zVi610ksYCBwJnp8+ipHPWzQN/rvLNJQrgGkkzJE0uO5g6No6IBZANJMBGJcdT6eOS7kxTQR2fSqkmaRywE9kVYledt6rYoAvOXZqymAksBKaT/XW+OCKWpV1K+++1OraIGDhvX0vn7duShpcQ2neAzwLPp88bUNI56+aBP1f55hLtERE7A28FjpO0Z9kBrUR+CLwSmAAsAL5VZjCS1gYuAU6IiMfLjKVajdi64txFxPKImEBWXXc34NW1dutsVKnTqtgkbQ98HngVsCuwPvC5TsYk6W3AwoiYUbm5xq4dOWfdPPAPunxzJ0XE/PRzIfBLsv/zd5tHJI0GSD8XlhwPABHxSPqP83ngLEo8d5JWIxtYL4yIS9PmrjhvtWLrpnOX4lkM3EA2j76epIFS76X/91oR2wFp6iwi4hngXDp/3vYADpY0l2zaeh+yvwBKOWfdPPB3bflmSWtJWmfgPbA/MLvxUaW4HJiU3k8CflViLC8YGFSTd1DSuUtzrFOBORFxRsVXpZ+3erF1w7mTNErSeun9GsB+ZPcgrgfelXYr6/ZrAvEAAAPkSURBVLzViu3uil/kIptH7+h5i4jPR8TYiBhHNpZdFxFHUtY5K/sud5M74BPJshnuA04pO56KuLYiyzK6A7irG2IDLiL70/85sr+WjiGbQ7wW+Gv6uX6XxHUBMAu4k2yQHV3SOXsD2Z/WdwIz02til5y3erGVfu6AHYE/pxhmA19I27cCbgX+BvwcGN5FsV2Xztts4CekzJ+S/n+3Ny9m9ZRyzlyywcysx3TzVI+ZmRXAA7+ZWY/xwG9m1mM88JuZ9RgP/GZmPcYDv600JP2PpL0lvV2DrNaa8rtvSZUR31hUjGYrAw/8tjJ5LVm9mr2A3w3y2H2BuyNip4gY7LFtUfGEplmpPPBb15P0DUl3ktVZ+SPwIeCHkr5QY98tJF2binFdK2nzVHv9dGBiqsW+RtUx+6a/BGalwmfD0/ZdJd2carvfKmmdVADsm2nfOyV9Iu07V9KG6X2fpBvS+1MlTZF0DXB+Ov4bkm5Lx38k7be3pBsk/ULS3ZIuTE+ZNoqjVjujJd2U/p2z/deN1VTW02t++TWYF1ltle+Tldn9Q4P9rgAmpfdHA5el9x8Ezqyx/wiyKrDj0+fzgRPI1oC4H9g1bR8JrAocS1Y/Z9W0ff30cy5pfQagD7ghvT8VmAGskT5PBv4rvR8O9ANbkj3NuYSsXssqZL/g3tAgjnrtfJr0JDnZmhbrlP2/nV/d9/Kfnray2ImsbMGrgL802O91wKHp/QVkV/qNbAM8EBH3ps/nAceRlWpYEBG3AUSq2ilpP+BHkUrpRkSetQYuj4in0vv9gR0lDdRnWRfYGngWuDUi5qV+ZgLjyH4Z1IqjXju3AeekAm+XRcTMHPFZj/HAb10tTdNMI7sS/iewZrZZM4HXVQyo9TSrSVKrNO7A9lrH1tu+jBenTkdUffdk1fGfiIirX9KotDfwTMWm5WT/fTaKY4V2Ult7ki34cYGkb0TE+TWOtx7mOX7rahExM7La6gNLD14HvCUiJtQZ9G8mq34IcCTw+yZd3A2Mk/Qf6fP7gRvT9k0l7QqQ5tVXBa4BPjpwo1bS+um4ucAu6f07G/R3NXBsuiJH0vhU4bVRfLXiqNmOpC3I6r6fRVbdc+cm/37rQb7it64naRTwWEQ8L+lVEdFoqud4sqmOzwCLgKMatR0RT0s6Cvh5GlBvI5vKeVbSYcD3083gp8hK/J4NjAfulPQcWU38M4EvAVMlncyLK2XVcjbZFM7t6ebtIhost9ckjlrt7A18JsX2b+ADjf791ptcndPMrMd4qsfMrMd44Dcz6zEe+M3MeowHfjOzHuOB38ysx3jgNzPrMR74zcx6zP8HqHvYRz9EJuAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure()\n",
    "plt.title(\"Feature Distribution\")\n",
    "plt.xlabel(\"# of occurences\")\n",
    "plt.ylabel(\"protein organelle\")\n",
    "# If this gives a boring graph, increase the subset size to look at more features.\n",
    "training_set.astype(bool).sum(axis=0).plot(kind='barh')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Preprocessing\n",
    "\n",
    "(see https://keras.io/preprocessing/image/)\n",
    "\n",
    "This will provide real-time data augmentation later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# construct the image generator for data augmentation\n",
    "datagen = ImageDataGenerator(rotation_range=25, width_shift_range=0.1,\n",
    "              height_shift_range=0.1, shear_range=0.2, zoom_range=0.2,\n",
    "              horizontal_flip=True, fill_mode=\"nearest\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read and Process (into Keras-ready train and test sets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Based on Keras and OpenCV docs\n",
    "\n",
    "def read_and_process(images_paths, ohe_dataframe):\n",
    "    '''\n",
    "    takes a panda dataframe with rows as picture IDs and columns as labels\n",
    "    returns lists for keras (to be used as x and y inputs to the CNN)\n",
    "    '''\n",
    "\n",
    "    images = [] # will be used by keras as x\n",
    "    labels = [] # will be used by keras as y\n",
    "\n",
    "    for img in images_paths:\n",
    "        # build image list using img_to_array() and OpenCV\n",
    "        images.append(img_to_array(cv2.imread(img, cv2.IMREAD_GRAYSCALE))) # load images using cv2.imread()\n",
    "\n",
    "        # build labels/target/tag list\n",
    "        # Check labels at each step and build labels list accordingly\n",
    "        # --> order is important, the lists are meant to be zip()ed\n",
    "        for idx, row in ohe_dataframe.iterrows():\n",
    "            if idx in img:\n",
    "                labels.append(tuple(row))\n",
    "    \n",
    "    return np.array(images), np.array(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test and Train splitting\n",
    "\n",
    "Split the data from the train set into 80% training and 20% validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(80, 512, 512, 1) (80, 28) (20, 512, 512, 1) (20, 28)\n"
     ]
    }
   ],
   "source": [
    "# WIP\n",
    "# USE sklearn.model_selection import train_test_split FOR THIS PART\n",
    "X, Y = read_and_process(train_green, training_set)\n",
    "trainX, testX, trainY, testY = train_test_split(X, Y, test_size=0.2, random_state=1)\n",
    "print(trainX.shape, trainY.shape, testX.shape, testY.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train_x, train_y = read_and_process(train_green, training_set)\n",
    "# test_x,  test_y  = read_and_process(validation_green, training_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CNN Setup\n",
    "\n",
    "Now that the preprocessing is out of the way, we can build our CNN structure.\n",
    "\n",
    "This will be a smaller implementation of *VGGNet*, as described here: https://arxiv.org/pdf/1409.1556/\n",
    "\n",
    "#### Input Layer Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "height, width, depth = IMG_SIZE\n",
    "input_shape = (height, width, depth) # derived from image size\n",
    "chanDim = -1\n",
    "classes = LEN_LABELS # number of classification classes\n",
    "\n",
    "# if we are using \"channels first\", update the input shape\n",
    "# and channels dimension\n",
    "if backend.image_data_format() == \"channels_first\":\n",
    "    input_shape = (depth, height, width)\n",
    "    chanDim = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Conv + ReLU + Pool Blocks\n",
    "\n",
    "Increasing filter sizes help increase depth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First CONV => RELU => POOL block\n",
    "model.add(Conv2D(16, (3, 3), padding=\"valid\", input_shape=input_shape)) # 32 filters and a 3x3 kernel\n",
    "model.add(Activation(\"relu\")) # standard activation layer\n",
    "model.add(BatchNormalization(axis=chanDim)) # normalize all inputs to the [0, 1] range\n",
    "model.add(MaxPooling2D(pool_size=(3, 3)))\n",
    "model.add(Dropout(0.25)) # dropout will reduce overfitting by randomly dropping 25% of node connections\n",
    "\n",
    "# (CONV => RELU) * 2 => POOL (first double convolution before pooling)\n",
    "model.add(Conv2D(32, (3, 3), padding=\"valid\")) # 64 filters and 3x3 kernel\n",
    "model.add(Activation(\"relu\"))\n",
    "model.add(BatchNormalization(axis=chanDim))\n",
    "model.add(Conv2D(32, (3, 3), padding=\"valid\"))\n",
    "model.add(Activation(\"relu\"))\n",
    "model.add(BatchNormalization(axis=chanDim))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2))) # smaller pool size this time to reduce spatial size\n",
    "model.add(Dropout(0.25))\n",
    " \n",
    "# (CONV => RELU) * 2 => POOL (second double convolution before pooling)\n",
    "model.add(Conv2D(64, (3, 3), padding=\"valid\")) # 128 filters and 3x3 kernel\n",
    "model.add(Activation(\"relu\"))\n",
    "model.add(BatchNormalization(axis=chanDim))\n",
    "model.add(Conv2D(64, (3, 3), padding=\"valid\"))\n",
    "model.add(Activation(\"relu\"))\n",
    "model.add(BatchNormalization(axis=chanDim))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "# (CONV => RELU) * 2 => POOL (third double convolution before pooling)\n",
    "model.add(Conv2D(128, (3, 3), padding=\"valid\")) # 256 filters and 3x3 kernel\n",
    "model.add(Activation(\"relu\"))\n",
    "model.add(BatchNormalization(axis=chanDim))\n",
    "model.add(Conv2D(128, (3, 3), padding=\"valid\"))\n",
    "model.add(Activation(\"relu\"))\n",
    "model.add(BatchNormalization(axis=chanDim))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "# (CONV => RELU) * 2 => POOL (fourth double convolution before pooling)\n",
    "model.add(Conv2D(256, (3, 3), padding=\"valid\")) # 512 filters and 3x3 kernel\n",
    "model.add(Activation(\"relu\"))\n",
    "model.add(BatchNormalization(axis=chanDim))\n",
    "model.add(Conv2D(256, (3, 3), padding=\"valid\"))\n",
    "model.add(Activation(\"relu\"))\n",
    "model.add(BatchNormalization(axis=chanDim))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fully Connected + ReLU block"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first (and only) set of FC => RELU layers\n",
    "model.add(Flatten()) # Collapses the spatial dimensions of the input into the channel dimensions\n",
    "model.add(Dense(1024)) # dense layer of neurons with 1024 being the dimensionality of the output space\n",
    "model.add(Activation(\"relu\"))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Dropout(0.5)) # Dropout of 50%\n",
    "\n",
    "# Since this is multi-label classification we use the sigmoid activation function\n",
    "model.add(Dense(classes)) # output dimensionality equal to the number of output classes\n",
    "model.add(Activation(\"sigmoid\"))\n",
    "\n",
    "##I think we want \"softmax\" not sigmoid, any picture may be multiple 1's"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# opt = SGD(lr=0.01, momentum=0.9) # Stochastic Gradient Descent with 0.01 learning rate\n",
    "opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS) # Adam will provide results faster than SGD with similar quality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### FBeta Score , Precision, and Recall Functions\n",
    "\n",
    "We must define our performance metrics before compiling the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate fbeta score for multi-class/label classification\n",
    "def fbeta(y_true, y_pred, beta=2):\n",
    "    # clip predictions\n",
    "    y_pred = backend.clip(y_pred, 0, 1)\n",
    "    # calculate elements\n",
    "    tp = backend.sum(backend.round(backend.clip(y_true * y_pred, 0, 1)), axis=1)\n",
    "    fp = backend.sum(backend.round(backend.clip(y_pred - y_true, 0, 1)), axis=1)\n",
    "    fn = backend.sum(backend.round(backend.clip(y_true - y_pred, 0, 1)), axis=1)\n",
    "    # calculate precision\n",
    "    p = tp / (tp + fp + backend.epsilon())\n",
    "    # calculate recall\n",
    "    r = tp / (tp + fn + backend.epsilon())\n",
    "    # calculate fbeta, averaged across each class\n",
    "    bb = beta ** 2\n",
    "    fbeta_score = backend.mean((1 + bb) * (p * r) / (bb * p + r + backend.epsilon()))\n",
    "    return fbeta_score\n",
    "\n",
    "# calculate precision\n",
    "def precision(y_true, y_pred):\n",
    "    # clip predictions\n",
    "    y_pred = backend.clip(y_pred, 0, 1)\n",
    "    # calculate elements\n",
    "    tp = backend.sum(backend.round(backend.clip(y_true * y_pred, 0, 1)), axis=1)\n",
    "    fp = backend.sum(backend.round(backend.clip(y_pred - y_true, 0, 1)), axis=1)\n",
    "    # calculate precision\n",
    "    p = tp / (tp + fp + backend.epsilon())\n",
    "    \n",
    "    return p\n",
    "\n",
    "# calculate recall\n",
    "def recall(y_true, y_pred):\n",
    "    # clip predictions\n",
    "    y_pred = backend.clip(y_pred, 0, 1)\n",
    "    # calculate elements\n",
    "    tp = backend.sum(backend.round(backend.clip(y_true * y_pred, 0, 1)), axis=1)\n",
    "    fn = backend.sum(backend.round(backend.clip(y_true - y_pred, 0, 1)), axis=1)\n",
    "    # calculate recall\n",
    "    r = tp / (tp + fn + backend.epsilon())\n",
    "    \n",
    "    return r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Compilation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since we're using checkpoints we need to load the model from disk, if possible\n",
    "try:\n",
    "    model = load_model('model.h5')\n",
    "except OSError:\n",
    "    # Otherwise we compile from scratch\n",
    "\n",
    "    # We use binary cross-entropy as this is multilabel multicategory classification\n",
    "    # For metrics we look at accuracy but also a more advanced metric for multilabel classification called the F-beta score\n",
    "    model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy', precision, recall, fbeta])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1 (Conv2D)            (None, 510, 510, 16)      160       \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 510, 510, 16)      0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_1 (Batch (None, 510, 510, 16)      64        \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 170, 170, 16)      0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 170, 170, 16)      0         \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 168, 168, 32)      4640      \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 168, 168, 32)      0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_2 (Batch (None, 168, 168, 32)      128       \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 166, 166, 32)      9248      \n",
      "_________________________________________________________________\n",
      "activation_3 (Activation)    (None, 166, 166, 32)      0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_3 (Batch (None, 166, 166, 32)      128       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 83, 83, 32)        0         \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 83, 83, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 81, 81, 64)        18496     \n",
      "_________________________________________________________________\n",
      "activation_4 (Activation)    (None, 81, 81, 64)        0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_4 (Batch (None, 81, 81, 64)        256       \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 79, 79, 64)        36928     \n",
      "_________________________________________________________________\n",
      "activation_5 (Activation)    (None, 79, 79, 64)        0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_5 (Batch (None, 79, 79, 64)        256       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 39, 39, 64)        0         \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 39, 39, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 37, 37, 128)       73856     \n",
      "_________________________________________________________________\n",
      "activation_6 (Activation)    (None, 37, 37, 128)       0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_6 (Batch (None, 37, 37, 128)       512       \n",
      "_________________________________________________________________\n",
      "conv2d_7 (Conv2D)            (None, 35, 35, 128)       147584    \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 35, 35, 128)       0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_7 (Batch (None, 35, 35, 128)       512       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_4 (MaxPooling2 (None, 17, 17, 128)       0         \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 17, 17, 128)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_8 (Conv2D)            (None, 15, 15, 256)       295168    \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 15, 15, 256)       0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_8 (Batch (None, 15, 15, 256)       1024      \n",
      "_________________________________________________________________\n",
      "conv2d_9 (Conv2D)            (None, 13, 13, 256)       590080    \n",
      "_________________________________________________________________\n",
      "activation_9 (Activation)    (None, 13, 13, 256)       0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_9 (Batch (None, 13, 13, 256)       1024      \n",
      "_________________________________________________________________\n",
      "max_pooling2d_5 (MaxPooling2 (None, 6, 6, 256)         0         \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (None, 6, 6, 256)         0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 9216)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1024)              9438208   \n",
      "_________________________________________________________________\n",
      "activation_10 (Activation)   (None, 1024)              0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_10 (Batc (None, 1024)              4096      \n",
      "_________________________________________________________________\n",
      "dropout_6 (Dropout)          (None, 1024)              0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 28)                28700     \n",
      "_________________________________________________________________\n",
      "activation_11 (Activation)   (None, 28)                0         \n",
      "=================================================================\n",
      "Total params: 10,651,068\n",
      "Trainable params: 10,647,068\n",
      "Non-trainable params: 4,000\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set up Checkpoints\n",
    "\n",
    "We'll automatically save the best model to disk so that we can resume training on it later.\n",
    "\n",
    "Different metrics can be chosen to determine the \"best\" model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint = ModelCheckpoint('model.h5', monitor='val_fbeta', verbose=1, save_best_only=True, mode='max')\n",
    "callbacks_list = [checkpoint] # callbacks list used by Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Class Weights\n",
    "\n",
    "Using class weights may help us alleviate the problem of dataset imbalance\n",
    "by telling the model to \"pay more attention\" to underrepresented classes.\n",
    "\n",
    "The class weights are derived from the entire dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 0.1407561394755807, 1: 1.4462861699703804, 2: 64.77295918367346, 3: 1.6593255783557705, 4: 2.636108803986711, 5: 3.377361000266028, 6: 1.7013535245242561, 7: 86.3639455782313, 8: 3.4219676549865228, 9: 8.63639455782313, 10: 2.0106905289832118, 11: 1.2237806053595528, 12: 0.5008679528149288, 13: 10.544435215946844, 14: 0.48018079352471726, 15: 2.261400071250445, 16: 0.6116839315827511, 17: 5.632431233362911, 18: 0.22042329328425586, 19: 5.529398954703833, 20: 164.87662337662337, 21: 1.1618468015008694, 22: 0.9761264031985237, 23: 0.721704280598033, 24: 1.7992488662131518, 25: 0.6426799635516858, 26: 34.21967654986523, 27: 40.303174603174604}\n"
     ]
    }
   ],
   "source": [
    "if DOWNSAMPLING:\n",
    "    all_labels = []\n",
    "    \n",
    "    #for idx, row in ohe_all_labels.iterrows():\n",
    "    for idx, entry in train_labels_all.items():\n",
    "        for label in entry:\n",
    "            all_labels.append(label)\n",
    "        \n",
    "    all_labels_array = np.array(all_labels)\n",
    "        \n",
    "    class_weights = class_weight.compute_class_weight('balanced', np.unique(all_labels_array), all_labels_array)\n",
    "    sample_weights = class_weight.compute_sample_weight('balanced', all_labels_array)\n",
    "    class_weights_dict = dict(enumerate(class_weights))\n",
    "    print(class_weights_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 80 samples, validate on 20 samples\n",
      "Epoch 1/15\n",
      "80/80 [==============================] - 60s 756ms/step - loss: 1.0508 - accuracy: 0.5058 - precision: 0.0569 - recall: 0.5021 - fbeta: 0.1812 - val_loss: 2.3180 - val_accuracy: 0.3304 - val_precision: 0.0640 - val_recall: 0.6792 - val_fbeta: 0.2223\n",
      "\n",
      "Epoch 00001: val_fbeta improved from -inf to 0.22233, saving model to model.h5\n",
      "Epoch 2/15\n",
      "80/80 [==============================] - 42s 523ms/step - loss: 0.9092 - accuracy: 0.5210 - precision: 0.0657 - recall: 0.5354 - fbeta: 0.2164 - val_loss: 4.0696 - val_accuracy: 0.3268 - val_precision: 0.0539 - val_recall: 0.5250 - val_fbeta: 0.1858\n",
      "\n",
      "Epoch 00002: val_fbeta did not improve from 0.22233\n",
      "Epoch 3/15\n",
      "80/80 [==============================] - 42s 531ms/step - loss: 0.8458 - accuracy: 0.5415 - precision: 0.0830 - recall: 0.6875 - fbeta: 0.2675 - val_loss: 4.7099 - val_accuracy: 0.2571 - val_precision: 0.0472 - val_recall: 0.5000 - val_fbeta: 0.1661\n",
      "\n",
      "Epoch 00003: val_fbeta did not improve from 0.22233\n",
      "Epoch 4/15\n",
      "80/80 [==============================] - 45s 567ms/step - loss: 0.7794 - accuracy: 0.5741 - precision: 0.0946 - recall: 0.7729 - fbeta: 0.2998 - val_loss: 5.1812 - val_accuracy: 0.3143 - val_precision: 0.0570 - val_recall: 0.6167 - val_fbeta: 0.2001\n",
      "\n",
      "Epoch 00004: val_fbeta did not improve from 0.22233\n",
      "Epoch 5/15\n",
      "80/80 [==============================] - 44s 547ms/step - loss: 0.7780 - accuracy: 0.5625 - precision: 0.0923 - recall: 0.7458 - fbeta: 0.2933 - val_loss: 4.9546 - val_accuracy: 0.3571 - val_precision: 0.0734 - val_recall: 0.7625 - val_fbeta: 0.2554\n",
      "\n",
      "Epoch 00005: val_fbeta improved from 0.22233 to 0.25536, saving model to model.h5\n",
      "Epoch 6/15\n",
      "80/80 [==============================] - 45s 563ms/step - loss: 0.7593 - accuracy: 0.5705 - precision: 0.0939 - recall: 0.7417 - fbeta: 0.2954 - val_loss: 5.9521 - val_accuracy: 0.3446 - val_precision: 0.0665 - val_recall: 0.6542 - val_fbeta: 0.2283\n",
      "\n",
      "Epoch 00006: val_fbeta did not improve from 0.25536\n",
      "Epoch 7/15\n",
      "80/80 [==============================] - 43s 543ms/step - loss: 0.7513 - accuracy: 0.5781 - precision: 0.0907 - recall: 0.7104 - fbeta: 0.2761 - val_loss: 5.0750 - val_accuracy: 0.3857 - val_precision: 0.0678 - val_recall: 0.6042 - val_fbeta: 0.2278\n",
      "\n",
      "Epoch 00007: val_fbeta did not improve from 0.25536\n",
      "Epoch 8/15\n",
      "32/80 [===========>..................] - ETA: 25s - loss: 0.7449 - accuracy: 0.6138 - precision: 0.1140 - recall: 0.8906 - fbeta: 0.3663"
     ]
    }
   ],
   "source": [
    "# train the network\n",
    "\n",
    "# With data augmentation\n",
    "#hist = model.fit_generator(datagen.flow(trainX, trainY, batch_size=BS),\n",
    "#            validation_data=(testX, testY),\n",
    "#            steps_per_epoch=(len(trainX) // BS),\n",
    "#            epochs=EPOCHS, verbose=1)\n",
    "\n",
    "# Without data augmentation\n",
    "if DOWNSAMPLING:\n",
    "    hist = model.fit(trainX, trainY,\n",
    "                class_weight=class_weights,\n",
    "                batch_size=BS,\n",
    "                validation_data=(testX, testY),\n",
    "                epochs=EPOCHS, callbacks=callbacks_list,\n",
    "                verbose=1)\n",
    "else:\n",
    "    hist = model.fit(trainX, trainY,\n",
    "                batch_size=BS,\n",
    "                validation_data=(testX, testY),\n",
    "                epochs=EPOCHS, callbacks=callbacks_list,\n",
    "                verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Diagnostics Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# WIP\n",
    "def summarize_diagnostics(history):\n",
    "    plt.figure(figsize=(20,10))\n",
    "    # plot accuracy\n",
    "    plt.subplot(511)\n",
    "    plt.title('Accuracy')\n",
    "    plt.plot(history.history['accuracy'], color='blue', label='train')\n",
    "    plt.plot(history.history['val_accuracy'], color='orange', label='test')\n",
    "    # plot loss \n",
    "    plt.subplot(512)\n",
    "    plt.title('Cross Entropy Loss')\n",
    "    plt.plot(history.history['loss'], color='blue', label='train')\n",
    "    plt.plot(history.history['val_loss'], color='orange', label='test')\n",
    "    # plot accuracy\n",
    "    plt.subplot(513)\n",
    "    plt.title('Fbeta')\n",
    "    plt.plot(history.history['fbeta'], color='blue', label='train')\n",
    "    plt.plot(history.history['val_fbeta'], color='orange', label='test')\n",
    "    #plot precision\n",
    "    plt.subplot(514)\n",
    "    plt.title('Precision')\n",
    "    plt.plot(history.history['precision'], color='blue', label='train')\n",
    "    plt.plot(history.history['val_precision'], color='orange', label='train')\n",
    "    #plot recall\n",
    "    plt.subplot(515)\n",
    "    plt.title('Recall')\n",
    "    plt.plot(history.history['recall'], color='blue', label='train')\n",
    "    plt.plot(history.history['val_recall'], color='orange', label='train')\n",
    "    # save plot to file\n",
    "    filename = sys.argv[0].split('/')[-1]\n",
    "    plt.savefig(filename + '_plot.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summarize_diagnostics(hist)\n",
    "#precision = true positives / (true positives + false positives)\n",
    "#recall = true positives / (true positives + false negatives)\n",
    "#fbeta ==> generalized f1 score derived from both precision and recall averaged over all classes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutional Filter Visualization\n",
    "\n",
    "based on https://keras.io/examples/conv_filter_visualization/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Utility Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalize(x):\n",
    "    \"\"\"utility function to normalize a tensor.\n",
    "\n",
    "    # Arguments\n",
    "        x: An input tensor.\n",
    "\n",
    "    # Returns\n",
    "        The normalized input tensor.\n",
    "    \"\"\"\n",
    "    return x / (backend.sqrt(backend.mean(backend.square(x))) + backend.epsilon())\n",
    "\n",
    "\n",
    "def deprocess_image(x):\n",
    "    \"\"\"utility function to convert a float array into a valid uint8 image.\n",
    "\n",
    "    # Arguments\n",
    "        x: A numpy-array representing the generated image.\n",
    "\n",
    "    # Returns\n",
    "        A processed numpy-array, which could be used in e.g. imshow.\n",
    "    \"\"\"\n",
    "    # normalize tensor: center on 0., ensure std is 0.25\n",
    "    x -= x.mean()\n",
    "    x /= (x.std() + backend.epsilon())\n",
    "    x *= 0.25\n",
    "\n",
    "    # clip to [0, 1]\n",
    "    x += 0.5\n",
    "    x = np.clip(x, 0, 1)\n",
    "\n",
    "    # convert to RGB array\n",
    "    x *= 255\n",
    "    if backend.image_data_format() == 'channels_first':\n",
    "        x = x.transpose((1, 2, 0))\n",
    "    x = np.clip(x, 0, 255).astype('uint8')\n",
    "    return x\n",
    "\n",
    "\n",
    "def process_image(x, former):\n",
    "    \"\"\"utility function to convert a valid uint8 image back into a float array.\n",
    "       Reverses `deprocess_image`.\n",
    "\n",
    "    # Arguments\n",
    "        x: A numpy-array, which could be used in e.g. imshow.\n",
    "        former: The former numpy-array.\n",
    "                Need to determine the former mean and variance.\n",
    "\n",
    "    # Returns\n",
    "        A processed numpy-array representing the generated image.\n",
    "    \"\"\"\n",
    "    if backend.image_data_format() == 'channels_first':\n",
    "        x = x.transpose((2, 0, 1))\n",
    "    return (x / 255 - 0.5) * 4 * former.std() + former.mean()\n",
    "\n",
    "\n",
    "def visualize_layer(model,\n",
    "                    layer_name,\n",
    "                    step=1.,\n",
    "                    epochs=15,\n",
    "                    upscaling_steps=9,\n",
    "                    upscaling_factor=1.2,\n",
    "                    output_dim=(412, 412),\n",
    "                    filter_range=(0, None)):\n",
    "    \"\"\"Visualizes the most relevant filters of one conv-layer in a certain model.\n",
    "\n",
    "    # Arguments\n",
    "        model: The model containing layer_name.\n",
    "        layer_name: The name of the layer to be visualized.\n",
    "                    Has to be a part of model.\n",
    "        step: step size for gradient ascent.\n",
    "        epochs: Number of iterations for gradient ascent.\n",
    "        upscaling_steps: Number of upscaling steps.\n",
    "                         Starting image is in this case (80, 80).\n",
    "        upscaling_factor: Factor to which to slowly upgrade\n",
    "                          the image towards output_dim.\n",
    "        output_dim: [img_width, img_height] The output image dimensions.\n",
    "        filter_range: Tupel[lower, upper]\n",
    "                      Determines the to be computed filter numbers.\n",
    "                      If the second value is `None`,\n",
    "                      the last filter will be inferred as the upper boundary.\n",
    "    \"\"\"\n",
    "\n",
    "    def _generate_filter_image(input_img,\n",
    "                               layer_output,\n",
    "                               filter_index):\n",
    "        \"\"\"Generates image for one particular filter.\n",
    "\n",
    "        # Arguments\n",
    "            input_img: The input-image Tensor.\n",
    "            layer_output: The output-image Tensor.\n",
    "            filter_index: The to be processed filter number.\n",
    "                          Assumed to be valid.\n",
    "\n",
    "        #Returns\n",
    "            Either None if no image could be generated.\n",
    "            or a tuple of the image (array) itself and the last loss.\n",
    "        \"\"\"\n",
    "        s_time = time.time()\n",
    "\n",
    "        # we build a loss function that maximizes the activation\n",
    "        # of the nth filter of the layer considered\n",
    "        if backend.image_data_format() == 'channels_first':\n",
    "            loss = backend.mean(layer_output[:, filter_index, :, :])\n",
    "        else:\n",
    "            loss = backend.mean(layer_output[:, :, :, filter_index])\n",
    "\n",
    "        # we compute the gradient of the input picture wrt this loss\n",
    "        grads = backend.gradients(loss, input_img)[0]\n",
    "\n",
    "        # normalization trick: we normalize the gradient\n",
    "        grads = normalize(grads)\n",
    "\n",
    "        # this function returns the loss and grads given the input picture\n",
    "        iterate = backend.function([input_img], [loss, grads])\n",
    "\n",
    "        # we start from a gray image with some random noise\n",
    "        intermediate_dim = tuple(\n",
    "            int(x / (upscaling_factor ** upscaling_steps)) for x in output_dim)\n",
    "        if backend.image_data_format() == 'channels_first':\n",
    "            input_img_data = np.random.random(\n",
    "                (1, 3, intermediate_dim[0], intermediate_dim[1]))\n",
    "        else:\n",
    "            input_img_data = np.random.random(\n",
    "                (1, intermediate_dim[0], intermediate_dim[1], 3))\n",
    "        input_img_data = (input_img_data - 0.5) * 20 + 128\n",
    "\n",
    "        # Slowly upscaling towards the original size prevents\n",
    "        # a dominating high-frequency of the to visualized structure\n",
    "        # as it would occur if we directly compute the 412d-image.\n",
    "        # Behaves as a better starting point for each following dimension\n",
    "        # and therefore avoids poor local minima\n",
    "        for up in reversed(range(upscaling_steps)):\n",
    "            # we run gradient ascent for e.g. 20 steps\n",
    "            for _ in range(epochs):\n",
    "                loss_value, grads_value = iterate([input_img_data])\n",
    "                input_img_data += grads_value * step\n",
    "\n",
    "                # some filters get stuck to 0, we can skip them\n",
    "                if loss_value <= backend.epsilon():\n",
    "                    return None\n",
    "\n",
    "            # Calculate upscaled dimension\n",
    "            intermediate_dim = tuple(\n",
    "                int(x / (upscaling_factor ** up)) for x in output_dim)\n",
    "            # Upscale\n",
    "            img = deprocess_image(input_img_data[0])\n",
    "            img = np.array(pil_image.fromarray(img).resize(intermediate_dim,\n",
    "                                                           pil_image.BICUBIC))\n",
    "            input_img_data = np.expand_dims(\n",
    "                process_image(img, input_img_data[0]), 0)\n",
    "\n",
    "        # decode the resulting input image\n",
    "        img = deprocess_image(input_img_data[0])\n",
    "        e_time = time.time()\n",
    "        print('Costs of filter {:3}: {:5.0f} ( {:4.2f}s )'.format(filter_index,\n",
    "                                                                  loss_value,\n",
    "                                                                  e_time - s_time))\n",
    "        return img, loss_value\n",
    "\n",
    "    def _draw_filters(filters, n=None):\n",
    "        \"\"\"Draw the best filters in a nxn grid.\n",
    "\n",
    "        # Arguments\n",
    "            filters: A List of generated images and their corresponding losses\n",
    "                     for each processed filter.\n",
    "            n: dimension of the grid.\n",
    "               If none, the largest possible square will be used\n",
    "        \"\"\"\n",
    "        if n is None:\n",
    "            n = int(np.floor(np.sqrt(len(filters))))\n",
    "\n",
    "        # the filters that have the highest loss are assumed to be better-looking.\n",
    "        # we will only keep the top n*n filters.\n",
    "        filters.sort(key=lambda x: x[1], reverse=True)\n",
    "        filters = filters[:n * n]\n",
    "\n",
    "        # build a black picture with enough space for\n",
    "        # e.g. our 8 x 8 filters of size 412 x 412, with a 5px margin in between\n",
    "        MARGIN = 5\n",
    "        width = n * output_dim[0] + (n - 1) * MARGIN\n",
    "        height = n * output_dim[1] + (n - 1) * MARGIN\n",
    "        stitched_filters = np.zeros((width, height, 3), dtype='uint8')\n",
    "\n",
    "        # fill the picture with our saved filters\n",
    "        for i in range(n):\n",
    "            for j in range(n):\n",
    "                img, _ = filters[i * n + j]\n",
    "                width_margin = (output_dim[0] + MARGIN) * i\n",
    "                height_margin = (output_dim[1] + MARGIN) * j\n",
    "                stitched_filters[\n",
    "                    width_margin: width_margin + output_dim[0],\n",
    "                    height_margin: height_margin + output_dim[1], :] = img\n",
    "\n",
    "        # save the result to disk\n",
    "        save_img('vgg_{0:}_{1:}x{1:}.png'.format(layer_name, n), stitched_filters)\n",
    "\n",
    "    # this is the placeholder for the input images\n",
    "    assert len(model.inputs) == 1\n",
    "    input_img = model.inputs[0]\n",
    "\n",
    "    # get the symbolic outputs of each \"key\" layer (we gave them unique names).\n",
    "    layer_dict = dict([(layer.name, layer) for layer in model.layers[1:]])\n",
    "\n",
    "    output_layer = layer_dict[layer_name]\n",
    "    assert isinstance(output_layer, layers.Conv2D)\n",
    "\n",
    "    # Compute to be processed filter range\n",
    "    filter_lower = filter_range[0]\n",
    "    filter_upper = (filter_range[1]\n",
    "                    if filter_range[1] is not None\n",
    "                    else len(output_layer.get_weights()[1]))\n",
    "    assert(filter_lower >= 0\n",
    "           and filter_upper <= len(output_layer.get_weights()[1])\n",
    "           and filter_upper > filter_lower)\n",
    "    print('Compute filters {:} to {:}'.format(filter_lower, filter_upper))\n",
    "\n",
    "    # iterate through each filter and generate its corresponding image\n",
    "    processed_filters = []\n",
    "    for f in range(filter_lower, filter_upper):\n",
    "        img_loss = _generate_filter_image(input_img, output_layer.output, f)\n",
    "\n",
    "        if img_loss is not None:\n",
    "            processed_filters.append(img_loss)\n",
    "\n",
    "    print('{} filter processed.'.format(len(processed_filters)))\n",
    "    # Finally draw and store the best filters to disk\n",
    "    _draw_filters(processed_filters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running the Script\n",
    "\n",
    "This script will help us visualize what the convolutional layers are learning.\n",
    "\n",
    "Unfortunately because of a TensorFlow issue it only works if images are read\n",
    "with cv2.IMREAD_COLOR and an RGB channel depth in image_size. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conv_layers = [l.name for l in model.layers if 'conv2d' in l.name]\n",
    "print(conv_layers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for layer_name in conv_layers[2:]:\n",
    "    try:\n",
    "        visualize_layer(model, layer_name)\n",
    "    except Exception as e:\n",
    "        print(\"This model isn't compatible with the script. Sorry!\")\n",
    "        print(str(e))\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "protein_classification.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}