NLP-Cheat-Sheet.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iim81vxey7xT"
      },
      "source": [
        "# NLP Cheat Sheet - Introduction - Overview - Python - Starter Kit\n",
        "\n",
        "Introduction to Natural Language Processing (NLP) tools, frameworks, concepts, resources for Python\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/janlukasschroeder/nlp-cheat-sheet-python/blob/master/NLP-Cheat-Sheet.ipynb)\n",
        "\n",
        "\n",
        "\n",
        "# NLP Python Libraries\n",
        "\n",
        "- [🤗 Models & Datasets](https://huggingface.co/) - includes all state-of-the models like BERT and datasets like CNN news\n",
        "- [spacy](https://spacy.io/) - NLP library with out-of-the box Named Entity Recognition, POS tagging, tokenizer and more\n",
        "- [NLTK](https://www.nltk.org/) - similar to spacy, simple GUI model download `nltk.download()`\n",
        "- [gensim](https://radimrehurek.com/gensim) - topic modelling, accessing corpus, similarity calculations between query and indexed docs, SparseMatrixSimilarity, Latent Semantic Analysis\n",
        "- [lexnlp](https://github.com/LexPredict/lexpredict-lexnlp) - information retrieval and extraction for real, unstructured legal text\n",
        "- [Holmes](https://github.com/msg-systems/holmes-extractor#the-basic-idea) - information extraction, document classification, search in documents\n",
        "- [fastText](https://fasttext.cc/) - library for efficient text classification and representation learning\n",
        "- [Stanford's Open IE](https://nlp.stanford.edu/software/openie.html) - information extraction of relation tuples from plain text, such as (Mark Zuckerberg; founded; Facebook). \"Barack Obama was born in Hawaii\" would create a triple (Barack Obama; was born in; Hawaii). [Open IE in Python](https://github.com/philipperemy/stanford-openie-python).\n",
        "\n",
        "# NLP Models\n",
        "\n",
        "## BERT & Transformer Models\n",
        "- [BERT](https://github.com/google-research/bert)\n",
        "- [Alibaba’s StructBERT](https://github.com/alibaba/AliceMind)\n",
        "- [FinBERT](https://github.com/ProsusAI/finBERT) - analyze sentiment of financial text\n",
        "- [DistilBERT](https://arxiv.org/abs/1910.01108)\n",
        "- [RoBERTa](https://arxiv.org/abs/1907.11692)\n",
        "- [VideoBERT](https://arxiv.org/pdf/1904.01766.pdf)\n",
        "- [XLnet](https://github.com/zihangdai/xlnet)\n",
        "\n",
        "## GPT Variants\n",
        "- [GPT-2](https://openai.com/blog/better-language-models/)\n",
        "- [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) - open source version of GPT\n",
        "\n",
        "## Other Models\n",
        "- [Flan-T5](https://arxiv.org/pdf/2210.11416.pdf) (most powerful state-of-the art model as of writing)\n",
        "- [Vega v2](https://arxiv.org/abs/2212.01853)\n",
        "- [ERNIE](https://github.com/PaddlePaddle/ERNIE/)\n",
        "- [Microsoft's Turing URL v6](https://arxiv.org/abs/2210.14867)\n",
        "- [Google's T5](https://github.com/google-research/text-to-text-transfer-transformer)\n",
        "- [Flair](https://github.com/flairNLP/flair)\n",
        "- [Whisper](https://github.com/openai/whisper) - speech-to-text transcription \n",
        "- [Exhaustive list of models (text, vision, audio) and systems](https://docs.google.com/spreadsheets/d/1AAIebjNsnJj_uKALHbXNfn3_YsT6sHXtCU0q7OIPuc4/edit#gid=0)\n",
        "\n",
        "Uncased model is better unless you know that case information is important for your task (e.g., Named Entity Recognition or Part-of-Speech tagging)\n",
        "\n",
        "# NLP Tasks\n",
        "\n",
        "## Text Generation\n",
        "- Text summarization, e.g. summarize an earnings call\n",
        "- Question answering, e.g. a chatbot answering simple customer questions\n",
        "- Google Ads copy generator, e.g. provide a text to the model for it to generate a Google Ad copy\n",
        "- Translation\n",
        "- Synonym finder\n",
        "\n",
        "## Text Classifcation\n",
        "- Sentiment analysis, e.g. assign sentiment (`positive`, `neutral`, `negative`) to a product review\n",
        "- Support ticket classification, e.g. assign classes (`bug`, `feature request`) to a customer support ticket\n",
        "- Document classification, e.g. find articles matching a search query\n",
        "- Fact checking\n",
        "\n",
        "## In-Text Analysis\n",
        "- Spell checking\n",
        "- Named entity recognition\n",
        "- Part-of speech tagging\n",
        "\n",
        "#  Framworks\n",
        "\n",
        "- PyTorch\n",
        "- TensorFlow\n",
        "- Keras\n",
        "\n",
        "## Training & ML Frameworks\n",
        "\n",
        "- [Uber's Horovod](https://horovod.readthedocs.io/en/stable/): distributed deep learning training framework for TensorFlow, Keras, and PyTorch \n",
        "- [Amazon's SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/distributed-training.html): distributed training libraries\n",
        "- [DeepSpeed](https://www.deepspeed.ai/)\n",
        "- [Uber's Michelangelo](https://www.uber.com/en-DE/blog/michelangelo-machine-learning-platform/)\n",
        "- [Uber's Ludwig](https://ludwig.ai/latest/)\n",
        "- [Google's TFX](https://www.tensorflow.org/tfx)\n",
        "- [H2O AutoML](https://docs.h2o.ai/): you don’t need to specify the model structure or hyperparameters. It experiments with multiple model architectures and picks out the best model given the features and the task.\n",
        "- [Google's AutoML](https://cloud.google.com/automl): for developers with limited machine learning expertise to train high-quality models.\n",
        "\n",
        "\n",
        "\n",
        "# Datasets\n",
        "\n",
        "- [Gutenberg Corpus](https://block.pglaf.org/germany.shtml) - contains 25,000 free electronic books. `from nltk.corpus import gutenberg`\n",
        "- [OntoNotes 5](https://github.com/ontonotes/conll-formatted-ontonotes-5.0) - corpus comprising various genres of text (news, conversational telephone speech, weblogs, usenet newsgroups, broadcast, talk shows) in three languages (English, Chinese, and Arabic) with structural information (syntax and predicate argument structure) and shallow semantics (word sense linked to an ontology and coreference).\n",
        "- [wiki_en_tfidf.mm in gensim](https://radimrehurek.com/gensim/wiki.html#latent-semantic-analysis) 3.9M documents, 100K features (distinct tokens) and 0.76G non-zero entries in the sparse TF-IDF matrix. The Wikipedia corpus contains about 2.24 billion tokens in total.\n",
        "- [GPT-2 Dataset](https://github.com/openai/gpt-2-output-dataset)\n",
        "- [Brown corpus](http://icame.uib.no/brown/bcm-los.html) - contains text from 500 sources, and the sources have been categorized by genre, such as news, editorial, and so on.\n",
        "- [Reuters Corpus - 10,788 news documents totaling 1.3 million words](https://www.nltk.org/book/ch02.html)\n",
        "- [Newsfilter.io stock market news corpus](https://developers.newsfilter.io/) - contains over 4 million press releases, earnings reports, FDA drug approvals, analyst ratings, merger agreements and many more covering all US companies listed on NASDAQ, NYSE, AMEX\n",
        "- [Kaggle - All the news, 143K articles](https://www.kaggle.com/snapcrack/all-the-news)\n",
        "- [Kaggle - Daily news for stock market prediction](https://www.kaggle.com/aaron7sun/stocknews)\n",
        "- [CNN News](https://drive.google.com/uc?export=download&id=0BwmD_VLjROrfTHk4NFg2SndKcjQ)\n",
        "- [AG News - PyTorch integrated](https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html)\n",
        "- [CommonCrawl News Dataset](https://commoncrawl.org/2016/10/news-dataset-available/)\n",
        "\n",
        "# Benchmarks\n",
        "\n",
        "Evaluate models performance on language understanding tasks, such as question answering and text summarization.\n",
        "\n",
        "- [Glue](https://gluebenchmark.com/leaderboard)\n",
        "- [SuperGlue](https://super.gluebenchmark.com/leaderboard)\n",
        "\n",
        "# Other Resources\n",
        "\n",
        "[PapersWithCode](https://paperswithcode.com/task/text-classification)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YrcS0p5Ry7xZ"
      },
      "source": [
        "# Starting with Spacy\n",
        "\n",
        "spacy (good for beginners; use NLTK for bigger projects)\n",
        "\n",
        "```shell\n",
        "pip install spacy\n",
        "python -m spacy download en \n",
        "# python -m spacy download en_core_web_lg\n",
        "```\n",
        "LexNLP (good for dealing with legal and financial documents; [installation guide here](https://github.com/LexPredict/lexpredict-contraxsuite/blob/master/documentation/Installation%20and%20Configuration/Installation%20and%20Configuration%20Guide.pdf))\n",
        "```shell\n",
        "pip install https://github.com/LexPredict/lexpredict-lexnlp/archive/master.zip\n",
        "python # to open REPL console\n",
        ">>> import nltk\n",
        ">>> nltk.download() # download all packages\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WiTCV9C9y7xZ"
      },
      "source": [
        "# Concepts\n",
        "---\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vR4ZUkIoy7xa"
      },
      "source": [
        "# Word embeddings (=word vectors)\n",
        "\n",
        "Visualizing word vectors using PCA ([link to paper](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)).\n",
        "\n",
        "![word-embs](https://www.tensorflow.org/images/linear-relationships.png)\n",
        "\n",
        "- Word embeddings are vector representation of words and learned from your data.\n",
        "- Example sentence: word embeddings are words converted into numbers.\n",
        "- A word in this sentence may be “Embeddings” or “numbers ” etc.\n",
        "- A dictionary may be the list of all unique words in the sentence, eg [‘Word’,’Embeddings’,’are’,’Converted’,’into’,’numbers’]\n",
        "- It’s common to see word embeddings that are 256-dimensional, 512-dimensional, or 1,024-dimensional\n",
        "- Another vector representation of a word may be a one-hot encoded vector where 1 stands for the position where the word exists and 0 everywhere else. \n",
        "- The geometric relationships between word vectors should reflect the semantic relationships between these words.\n",
        "- Words meaning different things are embedded at points far away from each other, whereas related words are closer.\n",
        "- For instance, by adding a “female” vector to the vector “king,” we obtain the vector “queen.” By adding a “plural” vector, we obtain “kings.”\n",
        "- The is a \"perfect\" word-embedding space for each task, for example the perfect word-embedding space for a movie-review sentiment-analysis model may look different from the perfect space for a legal-document-classification model.\n",
        "- Learn a new embedding space with every new task.\n",
        "\n",
        "\n",
        "**Example: one-hot encoded vector**\n",
        "\n",
        "- `numbers` word is represented as one-hot encoded vector = [0,0,0,0,0,1] \n",
        "- `converted` = [0,0,0,1,0,0]\n",
        "\n",
        "**Example: word-embeddings**\n",
        "\n",
        "![word-emb](https://i.imgur.com/6dNjnO7_d.webp?maxwidth=760&fidelity=grand)\n",
        "\n",
        "## One-Hot Vectors vs Word Embeddings\n",
        "\n",
        "![img](https://i.imgur.com/qJzBY8N_d.webp?maxwidth=760&fidelity=grand)\n",
        "\n",
        "Credits: Deep Learning with Python\n",
        "\n",
        "\n",
        "## Pre-trained word embeddings\n",
        "\n",
        "There are two ways to obtain word embeddings: \n",
        "\n",
        "1. Learn word embeddings jointly with the main task you care about (such as document classification or sentiment prediction). In this setup, you start with random word vectors and then learn word vectors in the same way you learn the weights of a neural network.\n",
        "2. Load into your model word embeddings that were precomputed using a different machine-learning task than the one you’re trying to solve. These are called pretrained word embeddings.\n",
        "\n",
        "**Pretrained word embeddings**\n",
        "\n",
        "- [Word2Vec (Google, 2013)](https://code.google.com/archive/p/word2vec/), uses Skip Gram and CBOW\n",
        "- [Vectors trained on Google News (1.5GB)](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit) - vocabulary of 3 million words trained on around 100 billion words from the google news dataset\n",
        "- [GloVe (Stanford)](https://nlp.stanford.edu/projects/glove), contains 100-dimensional embedding vectors for 400,000 words\n",
        "- [Stanford Named Entity Recognizer (NER)](https://nlp.stanford.edu/software/CRF-NER.shtml)\n",
        "- [LexPredict: pre-trained word embedding models for legal or regulatory text](https://github.com/LexPredict/lexpredict-lexnlp)\n",
        "- [LexNLP legal models](https://github.com/LexPredict/lexpredict-legal-dictionary) - US GAAP, finaical common terms, US federal regulators, common law\n",
        "\n",
        "> When parts of a model are pretrained (like your Embedding layer) and parts are randomly initialized (like your classifier), the pretrained parts shouldn’t be updated during training, to avoid forgetting what they already know."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Comparison of Embedding Models\n",
        "\n",
        "How OpenAI GPT-3 embeddings compare to Google and Sentence-Transformer embeddings.\n"
      ],
      "metadata": {
        "id": "K9QrRoHrbpN_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![image.png](https://pbs.twimg.com/media/FKLqFkuXEAY5Ehq?format=png)\n",
        "\n",
        "[Credit](https://twitter.com/Nils_Reimers/status/1487014195568775173?s=20&t=cXD8dnwWSbK_2fgw15yz5w)"
      ],
      "metadata": {
        "id": "erg3I2Rqb2Ss"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Embedding with SentenceTransformers\n",
        "\n",
        "[SentenceTransformers](https://www.sbert.net/) is a Python framework for state-of-the-art sentence, text and image embeddings. The initial work is described in the paper [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084).\n",
        "\n",
        "\n",
        "Install it with:\n"
      ],
      "metadata": {
        "id": "fx71hvkTGt6l"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install -U sentence-transformers"
      ],
      "metadata": {
        "id": "zpLdY1myHcQ9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sentence_transformers import SentenceTransformer\n",
        "model = SentenceTransformer('all-MiniLM-L6-v2')\n",
        "\n",
        "# Sentences to encode\n",
        "sentences = [\n",
        "  \"This framework generates embeddings for each input sentence.\",\n",
        "  \"Sentences are passed as a list of string.\",\n",
        "  \"The quick brown fox jumps over the lazy dog.\"\n",
        "]\n",
        "\n",
        "embeddings = model.encode(sentences)\n",
        "\n",
        "for sentence, embedding in zip(sentences, embeddings):\n",
        "    print(\"Sentence:\", sentence)\n",
        "    print(\"Embedding:\", embedding[:15])\n",
        "    print(\"Embedding dimension\", embedding.shape)\n",
        "    print(\"\")"
      ],
      "metadata": {
        "id": "X68rW7ttGtjO",
        "outputId": "09c9ec00-be08-420e-d840-70b0adae6248",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sentence: This framework generates embeddings for each input sentence.\n",
            "Embedding: [-0.01195314 -0.05562933 -0.00824256  0.00889048  0.02768425  0.1139881\n",
            "  0.01469875 -0.03189586  0.04145184 -0.08188552  0.01413268 -0.0203336\n",
            "  0.04077511  0.02262853 -0.04784386]\n",
            "Embedding dimension (384,)\n",
            "\n",
            "Sentence: Sentences are passed as a list of string.\n",
            "Embedding: [ 0.0564525   0.05500239  0.03137959  0.03394853 -0.03542475  0.08346675\n",
            "  0.09888012  0.00727544 -0.00668658 -0.0076581   0.07937384  0.00073965\n",
            "  0.01492921 -0.01510471  0.03676743]\n",
            "Embedding dimension (384,)\n",
            "\n",
            "Sentence: The quick brown fox jumps over the lazy dog.\n",
            "Embedding: [ 0.04393354  0.05893442  0.04817837  0.07754811  0.02674442 -0.03762956\n",
            " -0.0026051  -0.05994309 -0.002496    0.02207284  0.04802594  0.05575529\n",
            " -0.03894543 -0.0266168   0.0076934 ]\n",
            "Embedding dimension (384,)\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### GloVe Embeddings with Keras\n",
        "\n",
        "Fine-tuning pre-trained GloVe embeddings with Keras.\n"
      ],
      "metadata": {
        "id": "yFnssqXjMt9W"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from keras.models import Sequential\n",
        "from keras.layers import Embedding, Flatten, Dense\n",
        "\n",
        "# download from https://nlp.stanford.edu/projects/glove\n",
        "f = open('./glove.6B.100d.txt')\n",
        "\n",
        "embeddings_index = {}\n",
        "\n",
        "for line in f:\n",
        "  values = line.split()\n",
        "  word = values[0]\n",
        "  coefs = np.asarray(values[1:], dtype='float32')\n",
        "  embeddings_index[word] = coefs\n",
        "\n",
        "f.close()\n",
        "\n",
        "print('Found %s word vectors.' % len(embeddings_index))\n",
        "\n",
        "embedding_dim = 100\n",
        "maxlen = 100\n",
        "training_samples = 200\n",
        "validation_samples = 10000\n",
        "max_words = 10000\n",
        "\n",
        "embedding_matrix = np.zeros((max_words, embedding_dim))\n",
        "\n",
        "for word, i in word_index.items():\n",
        "  if i < max_words:\n",
        "    embedding_vector = embeddings_index.get(word)\n",
        "      if embedding_vector is not None:\n",
        "        embedding_matrix[i] = embedding_vector\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Embedding(max_words, embedding_dim, input_length=maxlen))\n",
        "# option A)\n",
        "model.add(Flatten())\n",
        "model.add(Dense(32, activation='relu'))\n",
        "# option B) LSTM\n",
        "#model.add(LSTM(32))\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "model.summary()\n",
        "\n",
        "model.compile(optimizer='rmsprop',\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['acc'])\n",
        "\n",
        "history = model.fit(x_train, y_train,\n",
        "                    epochs=10,\n",
        "                    batch_size=32,\n",
        "                    validation_data=(x_val, y_val))\n",
        "\n",
        "model.save_weights('pre_trained_glove_model.h5')"
      ],
      "metadata": {
        "id": "0iNTVllHLyTt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can also train the same model without loading the pretrained word embeddings and without freezing the embedding layer. In that case, you’ll learn a task-specific embedding of the input tokens, which is generally more powerful than pretrained word embeddings when lots of data is available."
      ],
      "metadata": {
        "id": "QJ7E8Jl-NFt8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Universal Sentence Encoder in TensorFlow (by Google)\n",
        "\n",
        "The Universal Sentence Encoder ([Cer et al., 2018](https://arxiv.org/pdf/1803.11175.pdf)) (USE) is a model that encodes text into 512-dimensional embeddings.\n",
        "\n",
        "[Source](https://github.com/tensorflow/tfjs-models/tree/master/universal-sentence-encoder)"
      ],
      "metadata": {
        "id": "gUYPf0DQmoR3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "import numpy as np\n",
        "\n",
        "module_url = \"https://tfhub.dev/google/universal-sentence-encoder/4\"\n",
        "model = hub.load(module_url)\n",
        "\n",
        "def embed(input):\n",
        "  return model(input)\n",
        "\n",
        "word = \"Elephant\"\n",
        "sentence = \"I am a sentence for which I would like to get its embedding.\"\n",
        "paragraph = (\n",
        "    \"Universal Sentence Encoder embeddings also support short paragraphs. \"\n",
        "    \"There is no hard limit on how long the paragraph is. Roughly, the longer \"\n",
        "    \"the more 'diluted' the embedding will be.\")\n",
        "\n",
        "messages = [word, sentence, paragraph]\n",
        "\n",
        "message_embeddings = embed(messages)\n",
        "\n",
        "for i, message_embedding in enumerate(np.array(message_embeddings).tolist()):\n",
        "  print(\"Message: {}\".format(messages[i]))\n",
        "  print(\"Embedding size: {}\".format(len(message_embedding)))\n",
        "  message_embedding_snippet = \", \".join(\n",
        "      (str(x) for x in message_embedding[:3]))\n",
        "  print(\"Embedding: [{}, ...]\\n\".format(message_embedding_snippet))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PD33Ryl3mn7O",
        "outputId": "2df28565-fc2b-40c5-98af-fcfcf76b0e27"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Message: Elephant\n",
            "Embedding size: 512\n",
            "Embedding: [0.008344486355781555, 0.00048085825983434916, 0.06595248728990555, ...]\n",
            "\n",
            "Message: I am a sentence for which I would like to get its embedding.\n",
            "Embedding size: 512\n",
            "Embedding: [0.050808604806661606, -0.016524329781532288, 0.01573779620230198, ...]\n",
            "\n",
            "Message: Universal Sentence Encoder embeddings also support short paragraphs. There is no hard limit on how long the paragraph is. Roughly, the longer the more 'diluted' the embedding will be.\n",
            "Embedding size: 512\n",
            "Embedding: [-0.028332693502306938, -0.0558621808886528, -0.012941480614244938, ...]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create word vectors yourself\n",
        "\n",
        "```python\n",
        "import gensim\n",
        "word2vev_model = gensim.models.word2vec.Word2Vec(sentence_list)\n",
        "```\n",
        "\n",
        "[Source](https://www.analyticsvidhya.com/blog/2017/06/word-embeddings-count-word2veec/)"
      ],
      "metadata": {
        "id": "Jf0u0kCPL2sC"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ukDygw-ay7xb"
      },
      "source": [
        "## How to create word vectors?\n",
        "- Count-based methods compute the statistics of how often some word co-occurs with its neighbor words in a large text corpus, and then map these count-statistics down to a small, dense vector for each word. \n",
        "- Predictive models directly try to predict a word from its neighbors in terms of learned small, dense embedding vectors (considered parameters of the model).\n",
        "  - Example: Word2vec (Google)\n",
        "\n",
        "### 1. Count based word embeddings\n",
        "\n",
        "#### Count Vector (= Document Term Matrix)\n",
        "\n",
        "![img](https://s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2017/06/04164920/count-vector.png)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### TF-IDF\n",
        "\n",
        "Term Frequency - Inverse Document Frequency. The goal is to determine a TF-IDF vector for each document. The vectors are then used to calculate the similarity between documents.\n",
        "\n",
        "- Term frequency (TF) is the number of times a word appears in a document divided by the total number of words in the document. \n",
        "- Inverse document frequency (IDF) = importance of the term across a corpus. It calculates the weight of rare words in all documents in the corpus, with rare words having a high IDF score, and words that are present in all documents (e.g. `a`, `the`, `is`) having IDF close to zero.\n",
        "\n",
        "The TF is calculated for a term `t` in a document `d`. Hence, every term in every document has a TF and we need to calculate TFs for every term in every document."
      ],
      "metadata": {
        "id": "6pGQVLLlb4uT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![img](https://latex.codecogs.com/png.image?\\small&space;\\dpi{150}TF(t,&space;d)&space;=&space;\\frac{\\text{Number&space;of&space;occurences&space;of&space;term&space;\\textit{t}&space;in&space;doc&space;\\textit{d}}}{\\text{Total&space;number&space;of&space;terms&space;in&space;doc&space;\\textit{d}}})\n",
        "\n"
      ],
      "metadata": {
        "id": "YEU2VZ96gDOu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The IDF score is calculated once for each term `t` occuring in the  corpus."
      ],
      "metadata": {
        "id": "K_uvgeBUgK0g"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![img](https://i.imgur.com/Vy0ocKG_d.webp?maxwidth=760&fidelity=grand)"
      ],
      "metadata": {
        "id": "njyFyObqjxVU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Combining these two, we get the TF-IDF score `w` for a term `t` in a document `d`:\n"
      ],
      "metadata": {
        "id": "pAFhSHOj0_jA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![img](https://latex.codecogs.com/png.image?\\small&space;\\dpi{150}w(t,&space;d)&space;=&space;TF(t,&space;d)&space;*&space;IDF(t))"
      ],
      "metadata": {
        "id": "_PXsDfE2k3Ah"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "(sklearn) in Python has a function `TfidfVectorizer()` that will compute the TF-IDF values for you."
      ],
      "metadata": {
        "id": "p8UO_W_mk7HZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Example 1\n",
        "\n",
        "The document size of our corpus is N=4."
      ],
      "metadata": {
        "id": "Kw-ejDE11M_i"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAJYAAACUCAYAAABxydDpAAABRGlDQ1BJQ0MgUHJvZmlsZQAAKJFjYGASSCwoyGFhYGDIzSspCnJ3UoiIjFJgf8bAzSDEwMcgwyCbmFxc4BgQ4ANUwgCjUcG3awyMIPqyLsgs1619QSyeWhKnsuv+te/QccNUjwK4UlKLk4H0HyBOSS4oKmFgYEwAspXLSwpA7BYgW6QI6CggewaInQ5hrwGxkyDsA2A1IUHOQPYVIFsgOSMxBch+AmTrJCGJpyOxofaC3RDhZGRu6BGq4EjAsaSCktSKEhDtnF9QWZSZnlGi4AgMoVQFz7xkPR0FIwMjIwYGUHhDVH8OAocjo9g+hFj+EgYGi28MDMwTEWJJUxgYtrcxMEjcQoipzGNg4AeG1bZDBYlFiXAHMH5jKU4zNoKweewZGFjv/v//WYOBgX0iA8Pfif///178///fxUDzbzMwHKgEAF/JXuMCB+4PAAAAVmVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAADkoYABwAAABIAAABEoAIABAAAAAEAAACWoAMABAAAAAEAAACUAAAAAEFTQ0lJAAAAU2NyZWVuc2hvdOJXkfoAAAHWaVRYdFhNTDpjb20uYWRvYmUueG1wAAAAAAA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4OnhtcHRrPSJYTVAgQ29yZSA2LjAuMCI+CiAgIDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+CiAgICAgIDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiCiAgICAgICAgICAgIHhtbG5zOmV4aWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20vZXhpZi8xLjAvIj4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjE0ODwvZXhpZjpQaXhlbFlEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlBpeGVsWERpbWVuc2lvbj4xNTA8L2V4aWY6UGl4ZWxYRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpVc2VyQ29tbWVudD5TY3JlZW5zaG90PC9leGlmOlVzZXJDb21tZW50PgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KAu6RbQAAHp5JREFUeAHtXQl8jVcWP51SrU5rqX3fqSUEkSCRpPbYgkq1tddemVLGUp0apah1UNRuUEtp7ISIfYtdYieKIrGFoMW0deack/m+ee8lb/+eLff+fu+9+93l3HPP/X/nnvvlvX9eQUqgkrKAwRb4i8HylDhlAbGAApYCgkcsoIDlEbMqoQpYCgMesYAClkfMqoQqYCkMeMQCClgeMasSqoClMOARCyhgecSsSqgCloMYUH+gcNBQ/2vmMrDY0Ldu3YILFy7AvXv3nBvVwNae1mPgwIGQM2dOGDFiRJpab968GUqWLAnJyclp1qfXQpeAtXbtWihWrJgYtEOHDlCpUiXInTs3DB8+HB4+fKjb8sGDB7Bs2TJo2LAh8AIZnRzVw51xeU5169a1KqJ8+fIQFBQEmTNnljZPnjwB1ivdJ7rjXUozZ87EgIAAve/Ro0fR29sb27Ztq5fFxMRgjx49sHTp0ti3b1+93MiMI3q4Ox7dPEgAc0jMzp07sXLlyg61fZkbueSx0robK1asCEuXLoUFCxbA+fPnpUm1atVg6tSp4Ofnl1YXj5SlpceOHTtEh3fffRdCQ0Nh48aN+tirV68GX19febHnrVWrFnTp0kWv1zKXL1+G+vXrQ548eaSerxMTE6Usf/78cP/+fdi6dSt89tlncOrUKfDy8oKWLVtK9zNnzkCdOnWgSpUqEBISAkeOHJHy69evQ7t27aSuRIkSZnpxg27dukHx4sVh/vz5wLbMli0btGnTBrZv3y765sqVC7p27Sqy+I3DkhYtWkD16tWl/Z49e6SO14D1GTVqFDRv3ly2dtbn3Llzel/DM67eNZaeQpOTN29eXLFihXYpn+3bt39qHksbWNPjl19+wezZs+Px48eRtikk8OPbb7+NtGXjo0eP8K233kJuw4niKAwPD8c//vhDEyOf7LGCg4MxISEBr127hrVr18Y+ffpI3c2bN/lrR0gxllyvWbMmlcdq2rQpRkdHS/3YsWOxVatWkh88eDCOGzdO8osWLUK6ASSvvf32229IWy0SGPDEiRN48eJFfP3117FJkyZ47NgxvHLliuh/6NAh6cK7wrBhwyQ/dOhQ7Ny5s+R5PgRArFChAq5fvx5//fVXZJ0oPNGGMvzTMI+lIZ5jjAIFCmiXz+xT02PDhg0SA5YrVw5eeeUVCAsLAwIakIHhL3/5i+QfP34sembMmBFee+01ePXVV1PpTWASb0WABVowPY6ihU7V1rSAPRmBChYuXAidOnUSr3bgwAFpwp58yJAh4nXYI1FoYdoV3njjDciUKRMQsKFs2bJQuHBh8TzstdgDsaek8APoppF+BFoYMGAAxMbGAse3BEQp5/nQDSSekeNdjgfZ+3I7T6UMRgo+e/YskBeQyRop11lZpnrwtmN5YmNw8CGDgUR3vxicF61o0aIwcuRIu8NxXz4pOpJ4u2SQ8+GFQcKJwcupXr16cPDgQWBAUFwGy5cvt3lQ4D6ss2nSZHEZb3lz584Ff39/uH37tmmzVHlLOakauFlgmMfiuKJjx44wY8aMNO94N/V0uLulHhxzxMXFwf79+0VGfHw8nDx5EmhrA9oSYM6cObB3717Ytm2bLArHUJaJ9gm9iD0BbVsSy+iFJpm//vWv8hhGK+JHEfxiz5kvXz7xOm+++aZUN2vWTIBCYYXEU6yXO2nKlCnQv39/mDBhgng41tXRxI9NDE2ubK6bNm3CUqVKyX7PMQAFi0hbDFJQaSaOgkOkuxLJkEjuFxs1amRW7+6Fo3rMmjVL4gsKzrFIkSJIi6wPzWW0JSLd+UigwMDAQIll9AaU+fzzz5FjNtq6kLwa0uJJNcdcjRs3lhiLbcDxFsdu3KZgwYLo4+ODSUlJSIcFpGAZKdhGekyDHFtx6t69u8ilgF7k0PNAKdfeOE5iu9GhAynQl/iQY0K2PceFERERmCVLFpFJNw5OmzYN6bEP0mEEBw0aJH0JvBLzUngi43P8S1ukxG4siwJ6pOeRmCNHDoyMjNSGdvsTXJXw559/ihE5wHyWyRk9aHswU5UBxzeEJoMXljwcTpw40ayddsHA4QOAI4m2QJFr2pYD/P/85z+mRXJtCSizBk5e8HpQzCi9OEh3VF/ybk6OZLu5yzEWB772AldDXasVYc7owUG7aaITHtBCyLGbzAS7d++WLZNOaqbN9Dzd1XreXoYfGFsmOo1aFslWaGS8wwG/lrSHttq1rU9te7bVxpm6Vxh3znR4mdpyjDVmzBiJsfjEyM/A6DGCnP5epnk+i7mka2A9C4OnlzENOxWmF4OpeTpmAQUsx+ykWjlpAQUsJw2mmjtmAQUsx+ykWjlpAQUsJw2mmjtmAQUsx+ykWjlpAQUsJw2mmjtmAQUsx+ykWjlpAQUsJw2mmjtmAZt/K6Q/zjomRbVKVxZI64uQlgawCayYmJTvMFl2Utfp2wI1alS3awCbfyu891B5LLsWTIcN3n4j9Ve3Lc2gYixLi6hrQyyggGWIGZUQSwsoYFlaRF0bYgEFLEPMqIRYWkABy9Ii6toQCyhgGWJGJcTSAgpYlhZR14ZYQAHLEDMqIZYWUMCytIi6NsQCCliGmFEJsbSAApalRexcp+OfYdqxjHm1y8BiA98mDtKff362HKTm0/Hc1ZnTpyCgelXImyP1r5m1UVuGNoIJY7/VLj3yefXqFXivVg3ImTVzKhYdjwzoolCXgBW5fh1ULFsSKnuVgZ5dP4EAvypQokg+GDPqGzMO0utE4dOh7YdQtmQRMcbKFT+5qGba3W7dvAm1A2tClswZoEundmaNmobUhexvZYLwnv9nvDNr4ORF6TLvwozZ/4bff//das9GjZtCKWqnpaiNkUBcDdqlIZ/58xeAJctWGC7XEOVMhLj87YZ/z50NSxYthA1RW0VcXOwx+LR7Z3i3bDmYPmuelAUHVIfGTZrC3/r0g1kzpsGEcaPh5NmLkCGDzW/rmKhnPxuzbw/0Du8J168nwrmfrwqFEt/Vrd8PhQvx5+Hqjbv2hTjY4vLlS+BdvjTcvvfIbg9i0YP8ubLC+UsJQnpmt4MTDZgaIF/OLHA54TYQ24wTPY1p+lS/3VDBqyLMnb9IwMYLysmLuBA+6dpDiC+CgmvDb2QQjT3PmCkCMLFHg5DGtHhvw949u0TssqWLoVuPXjKWxhHFi9G/b2+oXzsQ/KqSrrNnStvdu3ZA3eAA6NWjC3wxoB+UK1UUqvtUAvbKaSUmOpv0r3FQiQDGW+PiHxZIsxnfTwHfKl7wzbB/AgO7Tev3hYSubrA/+JNHv3PnjrQbP2YUBPn7QU3fyjBi+FB9iEH9+8JHYS1kJxg3OjX52/G4WOCttlYNH9LXX+/HGSaCG/j3z0Um68By7yQlSRtmFGRvXifIH2pU85ZPbnP+3FkzGUZfuLQVWlOieImSRKiRl4jNTkiTid99D1mzZpX8ooXzIaz1x2A0q0liwjVgktdWH3wIKyNSttq1q1dBaIv3IUeOnHA9MUHG37plM8THn4ON0dth3L8mCzi4oqZ/LfikSzdYs3ollCpdBvYfOQ6twlrD2DQWl9vzIhYqXAQOHj0BffsPEkDyInbt/inUCiQyNyI74+1q1ryF3Byitu6CXfsOCTHtsaNHYNvWaCrbCZu37YbZM7+H06dOwtYt0fDzhXhY9GMERKxeD6+bMMawjJs3bkBokwbwt979YMeeAzSHHVysp6+HfEnUkPdhy469sGZ9FOyhm2XOrOlS//2USUQN+SaNt0vGvHjxAqxcGwm8Vp5MhgKLFX2CT4Qb01Rp3jK3REfBV0OHmxYbkk9MSCDaxlwQRsBavSoCePGKEwMxM+vlIiqhxP8Bq3GTZvDTynUQf/4cJJCXu3zpoj5+1qzZiGmvCHTo1FmAH0Kx0onjafNz8tdyQ5u3lO2cP4sWKw6bozaKLI0KUhdskVm1MoL+2cJ92rp7QL8+4ZAxQ0Y4fOggAbo0UUbuF4/EHvjT8N5mPXfu2Ea8roUgMChYypm6yTSt+GkZfNy2vVBR8pw/atMeflr+ozTJQZSWjx+nbN2s+2sZX4MMr2YQPlZTGUbnzTV0Uzq718fEQepV0VuXxHflzOnTYNXajbr30isNyFy7dhXeIc/E3qZgwcLQt3c4fPhxSiDPRuUDBCeOxXhL4m3szJnTNkd2hq/qEXkw9oyOJAZzTf8AGPDFP+S1iTxX85atxMvHHIwFHx9fOuy0hm9HDDMTd/fuHbv0m8l3//+fMXjXYM/KqXGTUNiwbo1soR+3bgn/+OcwyEke3tPJMGDxcbxnt87A25/2ZXve65f/uBTmLlgkDMVJRLjKRGdGJt4KNaLZtu07kje6SltSkAyRK1duSKB6Tj8uWQy+fjVEv3r1GwjhrGZ8aeDAm+UzrM2bNsopsWo131S9ORZjgN66dVOva0YeLmpTpHiLQoUKEydpfqmbNGEsrKfFHzj4K5g2fQ7EHjuq9+FM/YaN4OiRQ3CIvBqn43HH5FN7Y3Au/mG+gImJdCN++hFCGjWR6mlTJ0Prj9rCtl0xsHzFWrrp2mrd4MD+fcJPrxcYmSFjWU3Jv/2Bab1WronEEiVTOEjLliuP1XyrI00O12/aordPuv8Yc+fJI/ycpK98Mtfn+Inf6W3Sku1MWfee4cKD6l25Ku6OOYwJt+7hvoPHRP6Y8RORPBbS1oCz5y3E6O27sUCBgljJuzK2adcBy5WvgHSCxbWR0cj9mT+9/6AvpW/DkMZIRGxIW52ZrodjT8m8ixQthhUreZOsKrj3wFFp86/JU4lPNJ+MOXL0OClr2eoDzEYc83SwEdtcT3qArDPrUax4CSQvi3Gn43HajDlIXgSD3quNwbXr4LZd+8zGZZsMHzkaKXZDAiO2eD8MaevGqj6+uGf/EZFBgEECq8gNa/0RXruZLDIobhOOVbrZkbZq4mAtiv/8+hupC6gViH36DUg1lr01sAoYkwqXHzfwncHPaEiW8JEbCXZPyWKd+XTInOesO7P4ObPtaXrxyZZfaVE/am20T/bSb1K8Zxp/sc3u0imRQKc1EzvevXtXgny90CLDjzCY7pzjR27LVJ2mdJ08N36Uo43Fc+QT7r8XLgF+Dsf9T56IgxbNGsHFKzdEFre1jNkshk116cjjhgypejlY4Az3p4MiPd6MdWZQcTLlR3d2YF4MbfHs9c3+zjupmjCgTUHFDaSM/qWJrcSgYVBx0k7bpu0tT9y81d+5kyR/HWE9ztK/XuFTYrv2nWQ8U75SUzlG5F32WEYMrmR43gL8KCNi+VI4c/o0FKP/y1O/QYgcGNwZ2RGPpYDljoXTaV9HgGXYqTCd2lhN24oFFLCsGEYVu2cBBSz37Kd6W7GAApYVw6hi9yyggOWe/VRvKxZQwLJiGFXsngUUsNyzn+ptxQIKWFYMo4rds4DNP+k8eaKI19wz78va2z7xmk1gnYo99LJaRs3LDQvQf5O129vmn3Ts9lYNlAWsWEDFWFYMo4rds4AClnv2U72tWEABy4phVLF7FlDAcs9+qrcVCyhgWTGMKnbPAgpY7tlP9bZiAQUsK4ZRxe5ZQAHLPfup3lYsoIBlxTCq2D0LKGC5Zz/V24oFFLCsGEYVu2cBm3+EtiWaf817m37le+/ePSLFyOHQr4JtyXO17nnRw1X9X9Z+LnmstWvXQrFixaBkyZLQoUMHqFSpEuQm+pzhw4frLCdssMOHD0P9+vWhUKFCUK1aNdi6NYX9zyhjOqqHUeM5I+f48ePExxXvTBePt2WKAbbZU0l0x7uUZs6ciQEBAXrfo0ePore3N7Zt21Yv8/f3x4iICKQJ4bhx4zAwMFCvMyrjiB5GjeWMnNq1a+Pq1aud6eLxtjt37sTKlSt7fBwegMkoXEqWC8pCzp49K6wy586dE5lEYKHL7tevH7Zu3Vq/NirjiB7bt29HX19fLFOmDDZr1gwjIyP14VetWoXkTeVVsWJFuVk6d+6s13Pm9OnTyEDhRWnYsCGSJ5Z68kjYvHlz9PPzQx8fH9y9e7eUd+3aFTNnzkyMMIWxQoUKGBUVJeW9e/eW8YsWLYrffPONlGlvU6ZMkbYjRozAJk2aYHZiqWG9Dhw4gE2bNkUKN2TsK1euaF2QvA9WrVoVWe/27dtjUlKS1E2ePBnpO1NYvnx57N69u5Rt2bJF9Ce+BhmnRYsWuhxPZAwFFiuYN29eXLFiha4rLyIbJl++fHjz5k293KhMWsBi2Zoev/zyiywSbU3iOZcuXYrEEoNEmIEMfCIJQW7DiRc1PDwciZVFrrU31j86Oloux44di61atZJ83759cdiwYZIfOnQomgKySpUqZh6LwcWA4cQ34Pjx4yWvvfGYDEgG4saNGzE5OVmAVbZsWVy3bh1SLCugZ8/PidibkcIRpDhX5sU6MjgTEhLE1sQ8I/NjIGnzW7NmzVPzWC4H79b2ad7HCxQooFf/+eef0LJlS6HKCQkJgf37n84/MNf02LBhg8SA5cqVE53CwsJgwIABsH79eqCFBvIMOuEuUxoxC41GHMcdmByWQAXvEFvLwoUL4QbxgZ44kcKxSiAT4rXY2FjiAH0AFy9elDHSeiNvCTExMUAeT8bv06ePWTMek5lw6tSpA/Xq1ZO6oKAgGZ/txik4OBg4duPE8SpTKdFOINfkyYC8G/Ts2ZPIda8S4VwC0M4hrMqXLl0yWxPp4OE3Q4FFd6JwLlGspautGYW2C2DjMntwNjt0PXpnFzOmehw5ciQV0T55MzlkMJAYXLyYtG0BbVEwcqQ5Y3EiUU0ySAcOHKhTF2kUSFOnToW5c+cCxZJyQralLo/JgCSvAgzuXr16wVdffWWrSyruLm1c7sQgZp2HDBmiy2C+LgY+bXMCJC8vL+Ab+1kkl06FaSl66tQp6NixI8yYMcPsjue2TAg2evRoWUBPg8pSDzZyXFyc7in5pHby5Em5+1mvOXPmwN69e2Hbtm0Ckjx58phNj0++/GLPR9u5LKbGQ8Ug6d+/P0yYMAFoyxKvpXVmHiva+rVLGDNmDFAwL0CYN28eMODdSaGhoXDo0CHxTAwwfjF/FsWTcvMuWbJEwMu8WxolOet0i/6biGnat89DdJHaHu/M56ZNm7BUqRSqSA4QOVCkuxA5SDZNgwYNkqCZ4x16LIE0QdNqt/OO6jFr1iyJXeixCFElFkECiT42lzGFJXkDJMPLyZW8gV7PGY55yKsh0X5LXDN48GCpnzZtGtJjFpkjz5UDdj4ccOIAmuXRFowEPCQwSX+WU7duXQnKpeH/3jgupRBC2nA8RjeDjEX8qhJj8QGCAC7x4vTp06XXpEmTkPUvWLAg0UQWQo6haOuWMj6o0JYq60LbuBwgOK7kgwO358MGB/u03SJ5Y1NVDMm7HLyTi5UAmMhqrSrCASafYvhxg6eSI3poY3Oga5oYcHxDaDI4QObT0sSJE02b6XkOqIl+Ub/mDM+fYh0p44DZdK4sj19a4jrt5KaVGfGpBfCmsijk0C9ZL9NE27vMmctYf56/0cnlGMsRqkh2zfnzpzADm7pfI/OO6KGNx4G6aWJOdWZx5iCXDAv0uEC2TDp5mTbT82lxjprSLZLH0ttyRqOl1AodoYPU2jrzaTkv7mtKJWmpFz/M1pKp/lqZEZ/p+udfHGNx7MMxFi86PQ8CPq1ZxllGGDq9yUjXwEpvi/0052vYqfBpKq3Gev4toID1/K/RC6mhAtYLuWzPv9IKWM//Gr2QGipgvZDL9vwrrYD1/K/RC6mhAtYLuWzPv9IKWM//Gr2QGtr8kw79DemFnJRS2rMWMP2+mrWRbAIrJubpfCnPmnKq/Pm0QI0ablJF3nuoPNbzubTPViv137+erf3T9egqeE/Xy++5yStgec626VqyAla6Xn7PTV4By3O2TdeSFbDS9fJ7bvIKWJ6zbbqWrICVrpffc5NXwPKcbdO1ZAWsdL38npu8ApbnbGuI5Lt37z4z/gV3JuAysPgHnreJB+Dnny8IXaQ7Sqi+qS2QSGwxtQNrQmANH6hQpngqYpPUPWyXXL16Bd6rVQNyZs3stizbI6XUugSsyPXroGLZklDZqwz07PoJBPhVgRJF8sGYUd+YUUVqCjD4ypYsAnNnz9CKDPm8RaQbbPwsmTNAl07tzGQ2DakL2d/KBOE9u5qVP+0Lvvli9u1xetgZ06dCs+Yt4NjJczBmwiRgQg93Uv78BWDJshVAFAHuiHG4r82vzViT0iCkEVy/nghLFi2EDVEpvKJxscfg0+6d4fz5czB91jy9K/E3QJeO7YSSh7gL9HIjMjly5oQR346B3uE9IXrzJtky+LtCfHcyXRL/fHzyVGPB7Kzec2ZNB97OfP1qONU1YvmPsHzFGunTqHFTp/paa/ymm+C0Jjetcpc8VlqCKnhVhLnzFwnYLsSf15t8PeRLCAx+DypVrqKXGZlh/oUGIY2JJ+Ft2Ltnl4hetnQxdOvRS4jJNAof/jl9/769oX7tQPCrSrrOniltd+/aAXWDA6BXjy7wxYB+UK5UUajuUwnYK6eVxo8ZBUH+flDTtzKMGD5UmliTPWvGNJgzawYsXrQAalTzhgljv5X2rGen9h/L1sRj8banpZ8vxENokwbAn20+bAX+tBvwTcJ6snf28S4PH4W1gOioTVoXm3XH42KhZWgjqEVbat1gf72PxzO2WEaSf/sDrb0mTZmONWoGpKrPkycv/rD0Jyn/aeU6pDsVb997hKEt3sfxE79L1d6afEfLR40Zj/z6+8DB2KVbT5HvU80Pr964S3SR+fBw7CkpY53q1Ksv+fWbtmCx4iV0XcjDYtZs2XDid9/jtZvJOGTocGQZljrs2HMAA4OC8VbyQ0y8fR/JY2LMoViZrzXZ4Z99jr3+1sdMll/1mrh9934p+/zvA/HStVtm9aw7LTz+kpgk5SfPXcRsxEm67+AxvPvr7zhvwWKhu7ye9ABt1Z2/eA1zEvXS6vVRIufK9Tsi93LCbbPxLOdp79oWZrQ6wzyWdgc8wSfCMMN3Yf9+vWHm3PlCCKbVG/3J4+TMmQvCPvgQVq+KgGNHj0DxEiUkJslFrCqJiSneoHGTZkBAh3jaqhPIy12+dFFXJWvWbERcVgQ6dOoMTKoWQlvPieOxer2WWbUygg4q92nr7QH9+oRDxgwZ4fChg2BLttbX9LNmQC3xIt+OGAZ9+vY3Y4YxbaflozZGghftCO+WLSfkJc1btiJWxOywaeMGsFW3c8c2YvYrBHQziChm5nlayaUYy5py58+dhcePHoFXRW9Y/MMCuHXzBoS1SIkPrlHcs2/Pbti2JRoWLF5mTYTT5deuXYWg9+pAqdJloGDBwtC3dzh8OeRrkcMx2HWieuTEATTXVanqA7lym7P2SQOTN6aQTCsxGGv6B0C3nr2kesAX/xBQOyObOw6kfgG1gmDyxHEwm2KwPTFHgHW1lZKT75lV586TFx49fChl1uoePLifil3RTIgHLwyD8JnTp6Bnt85A24lMpk27DhB7Kh4iN2+XFxuy12d9YMr02YZOJzHhGi1uyqK0bd+RvNFVqBUYJGPkypWbqBSvSf7HJYslgGb96tVvILyixHDnlC7NmreEqE2R4jUKFSpM1JEp3F+2ZHPAfPv2/+kZ+TENx2dBFHdGrFpP3r0g8M1hKzVt1hxOnoiDQwdTfoPA8deZ0ychgOZpq65+w0Zw9Mghvd/xuGNmwxzY7yGaSB5F2xPT+rS2165cE4klSqZQRZYtVx6r+VZHcs/IsYu1Pp6Isbr3DMfXX38dvStXxd0xhzHh1j2JQ1iHMeMnSgxE2yHOnrcQo7fvJirGgljJuzIS6LFc+QpIWwuujYyW/iyn/6AvRf+GIY2R+LIwtHlLs/lwTMNjshyO0chLYtzpeKuyOf7hcTl+Y3u1+uBDkcdj83W9Bg2RDhlmY1y8cgObNA2VWIh1XbJ8pdTT6VZ09qpYCWnbRo5fNVvbqhs+cjTSowai6M6PLd4Pk75VfXxxz/4jGFArEPv0G6DL0eTZ+0wLK5ZlNvmxbP2Ygh8d8DMREijHerNbIY0L9hxvvJHZbjyRRlfDilhnPsEx0x7rzmRr1rY9W4PynO8y+7MJQ6At2fzI5V5yMrxD/3NIS3xaZRZkUyZkrc7W552kJLNxTdtaqyMOeWGz5mdh/OiDbiJ5scfOlCmTUKWbyrGXd+THFC4Dy97gqv7ltYAjwDIsxnp5zahm5ooFFLBcsZrqY9cCClh2TaQauGIBBSxXrKb62LWAApZdE6kGrlhAAcsVq6k+di2ggGXXRKqBKxZQwHLFaqqPXQsoYNk1kWrgigUUsFyxmupj1wI2vzYTfzblX9TalaIapCsLeFf0sjtfm38rpP/PZ1eAapD+LJAlSxa7k7YJLLu9VQNlASsWUDGWFcOoYvcsoIDlnv1UbysWUMCyYhhV7J4FFLDcs5/qbcUCClhWDKOK3bOAApZ79lO9rVhAAcuKYVSxexb4L0P9kSMK/mFTAAAAAElFTkSuQmCC)"
      ],
      "metadata": {
        "id": "wrnFEKMJ1lMQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The 6 unique terms in our corpus are `dog`, `bites`, `man`, `eats`, `meat`, `food`.\n",
        "\n",
        "Let's determine the TF-IDF scores `w` for all terms in the first document `Dog bites man.` "
      ],
      "metadata": {
        "id": "fIZ-GwBSlRR7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "Terms|TF |IDF Score|w = TF * IDF\n",
        "--   |--|--|--------|\n",
        "dog| 1/3|log(4/3)|0.138\n",
        "bites| 1/3|log(4/2)|0.33\n",
        "man| 1/3|log(4/3)|0.138\n",
        "eats| 0 |log(4/2)|0\n",
        "meat| 0 |log(4/1)|0\n",
        "food| 0 |log(4/1)|0"
      ],
      "metadata": {
        "id": "F8TtngAPpeQh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The corresponding TF-IDF vector for document `D1` is:\n",
        "\n",
        "`[0.138, 0.33, 0.138, 0, 0, 0]`"
      ],
      "metadata": {
        "id": "3s9nLf0XrT_9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "##### Example 2\n",
        "\n",
        "Consider a document containing 100 words wherein the word cat appears 3 times. The term frequency (TF) for cat is then (3 / 100) = 0.03. \n",
        "\n",
        "Now, assume we have 10 million documents and the word cat appears in one thousand of these. Then, the inverse document frequency (IDF) is calculated as log(10,000,000 / 1,000) = 4. \n",
        "\n",
        "Thus, the Tf-idf weight is the product of these quantities: 0.03 * 4 = 0.12"
      ],
      "metadata": {
        "id": "XaO7X3FC0ZbQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### TD-IDF Implementation in Python "
      ],
      "metadata": {
        "id": "8-c4ukxn2bQ3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "document_corpus = [\n",
        "  \"Dog bites man\", \n",
        "  \"Man bites dog\", \n",
        "  \"Dog eats meat\", \n",
        "  \"Man eats food\"\n",
        "]\n",
        "\n",
        "tfidf = TfidfVectorizer()\n",
        "bow_rep_tfidf = tfidf.fit_transform(document_corpus)\n",
        "\n",
        "print(\"IDF for all words in the vocabulary\")\n",
        "print(tfidf.idf_)\n",
        "print(\"\\nAll words in the vocabulary.\")\n",
        "print(tfidf.get_feature_names_out())\n",
        "\n",
        "temp = tfidf.transform([\"Dog bites man\"])\n",
        "\n",
        "print(\"\\nTF-IDF representation for 'Dog bites man':\\n\", temp.toarray())"
      ],
      "metadata": {
        "id": "_s-zKz_z2miy",
        "outputId": "71c40e38-dfa3-4a71-cfdb-e578499b48e7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "IDF for all words in the vocabulary\n",
            "[1.51082562 1.22314355 1.51082562 1.91629073 1.22314355 1.91629073]\n",
            "\n",
            "All words in the vocabulary.\n",
            "['bites' 'dog' 'eats' 'food' 'man' 'meat']\n",
            "\n",
            "TF-IDF representation for 'Dog bites man':\n",
            " [[0.65782931 0.53256952 0.         0.         0.53256952 0.        ]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Notice that the TF-IDF scores that we calculated for our corpus doesn't match the TF-IDF scores given by scikit-learn. This is because scikit-learn uses a slightly modified version of the IDF formula. This stems from provisions to account for possible zero divisions and to not entirely ignore terms that appear in all documents."
      ],
      "metadata": {
        "id": "a4aFeOWo44Vq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "TD-IDF with N-Gram"
      ],
      "metadata": {
        "id": "fZqOCSlI2szC"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "qdK5jkeNy7xb",
        "outputId": "a74c1a8e-6825-4ae9-fd5d-e0d477ad3b0e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "IDF for all words in the vocabulary\n",
            "[1.51082562 1.51082562 1.91629073 1.91629073 1.91629073 1.91629073\n",
            " 1.91629073 1.91629073 1.91629073 1.91629073 1.91629073 1.91629073\n",
            " 1.51082562 1.51082562 1.91629073 1.91629073 1.91629073 1.91629073\n",
            " 1.51082562 1.91629073 1.91629073 1.91629073 1.91629073 1.91629073\n",
            " 1.91629073 1.91629073 1.91629073 1.91629073 1.91629073 1.91629073\n",
            " 1.91629073]\n",
            "\n",
            "All words in the vocabulary.\n",
            "[' bite' ' eats' ' food' ' meat' 'Dog b' 'Dog e' 'Man b' 'Man e' 'an bi'\n",
            " 'an ea' 'ats f' 'ats m' 'bites' 'eats ' 'es do' 'es ma' 'g bit' 'g eat'\n",
            " 'ites ' 'n bit' 'n eat' 'og bi' 'og ea' 's dog' 's foo' 's man' 's mea'\n",
            " 'tes d' 'tes m' 'ts fo' 'ts me']\n",
            "\n",
            "TF-IDF representation for 'Dog bites man':\n",
            " [[0.28113163 0.         0.         0.         0.35657982 0.\n",
            "  0.         0.         0.         0.         0.         0.\n",
            "  0.28113163 0.         0.         0.35657982 0.35657982 0.\n",
            "  0.28113163 0.         0.         0.35657982 0.         0.\n",
            "  0.         0.35657982 0.         0.         0.35657982 0.\n",
            "  0.        ]]\n"
          ]
        }
      ],
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "import re\n",
        "\n",
        "document_corpus = [\n",
        "  \"Dog bites man\", \n",
        "  \"Man bites dog\", \n",
        "  \"Dog eats meat\", \n",
        "  \"Man eats food\"\n",
        "]\n",
        "\n",
        "# Write a function for cleaning strings and returning an array of ngrams\n",
        "def ngrams_analyzer(string):\n",
        "    string = re.sub(r'[,-./]', r'', string)\n",
        "    ngrams = zip(*[string[i:] for i in range(5)])  # N-Gram length is 5\n",
        "    return [''.join(ngram) for ngram in ngrams]\n",
        "\n",
        "# Construct your vectorizer for building the TF-IDF matrix\n",
        "tfidf = TfidfVectorizer(analyzer=ngrams_analyzer)\n",
        "\n",
        "bow_rep_tfidf = tfidf.fit_transform(document_corpus)\n",
        "\n",
        "print(\"IDF for all words in the vocabulary\")\n",
        "print(tfidf.idf_)\n",
        "print(\"\\nAll words in the vocabulary.\")\n",
        "print(tfidf.get_feature_names_out())\n",
        "\n",
        "temp = tfidf.transform([\"Dog bites man\"])\n",
        "\n",
        "print(\"\\nTF-IDF representation for 'Dog bites man':\\n\", temp.toarray())\n",
        "\n",
        "# Credits: https://towardsdatascience.com/group-thousands-of-similar-spreadsheet-text-cells-in-seconds-2493b3ce6d8d"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2MawDaNry7xd"
      },
      "source": [
        "#### Co-Occurrence Vector/Matrix\n",
        "\n",
        "Words that are similar to each other will tend to co-occur together. \n",
        "\n",
        "Let’s call the context of the word, the two words that surround a specific word by each side. For example, in a sentence `I ate a peach yesterday`, the word `peach` is surrounded by the words: `ate`, `a`, `yesterday`.\n",
        "\n",
        "To build a co-occurrence matrix, one has to start with the full vocabulary of words in a specific corpus.\n",
        "\n",
        "**Example**\n",
        "\n",
        "Let’s imagine some simple sentences:\n",
        "- I’m riding in my car to the beach.\n",
        "- I’m riding in my jeep to the beach.\n",
        "- My car is a jeep.\n",
        "- My jeep is a car.\n",
        "- I ate a banana.\n",
        "- I ate a peach.\n",
        "\n",
        "The vocabulary of our group of sentences is:\n",
        "\n",
        "`a, ate, banana, beach, car, in, is, I’m, jeep, my, riding, to, the`\n",
        "\n",
        "Our co-occurence vector will be of of size 13, where 13 is the number of distinct words in our vocabulary.\n",
        "\n",
        "The initialized co-occurence vector for the word `car` is:\n",
        "\n",
        "`[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]`\n",
        "\n",
        "In our sentences, the word `car` shows up in 3 sentences:\n",
        "\n",
        "- I’m riding **in my** car **to the** beach.\n",
        "- **My** car **is a** jeep.\n",
        "- My jeep **is a** car.\n",
        "\n",
        "The highlighted words co-occur with the word `car`, i.e. the highlights represent the two words before and two words after `car`.\n",
        "\n",
        "The co-occurence vector for `car` is: \n",
        "\n",
        "```\n",
        "# Vocabulary + co-occurence vector below\n",
        " a, ate, banana, beach, car, in, is, I’m, jeep, my, riding, to, the\n",
        "[2,   0,      0,     0,   0,  1,  2,   0,    0,  2,      0,  1,   1]\n",
        "```\n",
        "\n",
        "Each number represents the number of occurences in the context of the word. For example, `a` appears twice, whereas `ate` didn't appear at all.\n",
        "\n",
        "[Credits](https://towardsdatascience.com/word-vectors-intuition-and-co-occurence-matrixes-a7f67cae16cd)\n",
        "\n",
        "### 2. Prediction based word embeddings\n",
        "\n",
        "- Uses Neural Networks\n",
        "- CBOW predicts target words (e.g. 'mat') from source context words ('the cat sits on the')\n",
        "- Skip-gram does the inverse and predicts source context-words from the target words\n",
        "\n",
        "#### CBOW (Continuous Bag of words)\n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/softmax-nplm.png\" width=\"400\">\n",
        "\n",
        "#### Skip Gram\n",
        "\n",
        "Skip – gram follows the same topology as of CBOW. It just flips CBOW’s architecture on its head. The aim of skip-gram is to predict the context given a word\n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/nce-nplm.png\" width=\"400\">\n",
        "\n",
        "#### Outcome\n",
        "\n",
        "![out](https://github.com/sanketg10/deep-learning-repo/raw/6b207e326cc937930a0512a8c599e86e48c297b2/embeddings/assets/skip_gram_net_arch.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9N4KXzO7y7xe"
      },
      "source": [
        "## Bag of Words"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "skdWw-ICy7xf"
      },
      "outputs": [],
      "source": [
        "# John likes to watch movies. Mary likes movies too.\n",
        "BoW1 = {\"John\":1,\"likes\":2,\"to\":1,\"watch\":1,\"movies\":2,\"Mary\":1,\"too\":1};"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3sPKycLwy7xg"
      },
      "source": [
        "# spacy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "QiXAztrmy7xg"
      },
      "outputs": [],
      "source": [
        "import spacy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dYMicDKjy7xg"
      },
      "outputs": [],
      "source": [
        "# Import dataset\n",
        "nlp = spacy.load(\"en\")\n",
        "# Import large dataset. Needs to be downloaded first.\n",
        "# nlp = spacy.load(\"en_core_web_lg\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ClIx_Z-y7xh"
      },
      "source": [
        "# Stop Words\n",
        "\n",
        "Stop words are the very common words like ‘if’, ‘but’, ‘we’, ‘he’, ‘she’, and ‘they’.\n",
        "We can usually remove these words without changing the semantics of a text and doing so often (but not always) improves the performance of a model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iUl6Yrf_y7xh",
        "outputId": "8dacc3a8-6605-41c7-de79-83e9e9e995df"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "spacy: Number of stop words: 326\n"
          ]
        }
      ],
      "source": [
        "# spacy: Removing stop words\n",
        "spacy_stopwords = spacy.lang.en.stop_words.STOP_WORDS\n",
        "\n",
        "print('spacy: Number of stop words: %d' % len(spacy_stopwords))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7TTuOO98y7xi",
        "outputId": "5eebed21-606e-4c3c-9bd4-078f78b61b64"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "ntlk: Number of stop words: 179\n"
          ]
        }
      ],
      "source": [
        "# nltk: Removing stop words \n",
        "from nltk.corpus import stopwords\n",
        "english_stop_words = stopwords.words('english')\n",
        "\n",
        "print('ntlk: Number of stop words: %d' % len(english_stop_words))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jZC_YPbjy7xi",
        "outputId": "9872e3d4-1e0b-45c1-8fd5-1e48432c6db1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Original text: Larry Page founded Google in early 1990.\n",
            "\n",
            "['Larry', 'Page', 'founded', 'Google', 'early', '1990', '.']\n"
          ]
        }
      ],
      "source": [
        "text = 'Larry Page founded Google in early 1990.'\n",
        "doc = nlp(text)\n",
        "tokens = [token.text for token in doc if not token.is_stop]\n",
        "print('Original text: %s' % (text))\n",
        "print()\n",
        "print(tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qPsiViVVy7xi"
      },
      "source": [
        "# Spans\n",
        "Part of a given text. So doc[2:4] is a span starting at token 2, up to – but not including! – token 4.\n",
        "\n",
        "Docs: https://spacy.io/api/span"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ssm4DM2Ty7xj",
        "outputId": "f9b5ab4b-d31e-4e84-b4f3-a50d31d3f69f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'founded Google'"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "doc = nlp(\"Larry Page founded Google in early 1990.\")\n",
        "span = doc[2:4]\n",
        "span.text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q1XNkQ_cy7xj",
        "outputId": "aae60991-a4ef-4b9a-a54a-9adeb60704ec"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[Larry, Page, founded, Google, in, early, 1990, .]"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "[(spans) for spans in doc]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AWcQBNMUy7xj"
      },
      "source": [
        "# Token and Tokenization\n",
        "\n",
        "Segmenting text into words, punctuation etc.\n",
        "- Sentence tokenization\n",
        "- Word tokenization\n",
        "\n",
        "Docs: https://spacy.io/api/token"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9AciYES8y7xj",
        "outputId": "366e1d0a-7bfb-4d5f-8135-2039932573d8"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['Larry', 'Page', 'founded', 'Google', 'in', 'early', '1990', '.']"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "doc = nlp(\"Larry Page founded Google in early 1990.\")\n",
        "[token.text for token in doc]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZD2gLn0Yy7xj"
      },
      "outputs": [],
      "source": [
        "# Load OpenAI GPT-2 using PyTorch Transformers\n",
        "tokenizer = GPT2Tokenizer.from_pretrained('gpt2')\n",
        "model = GPT2Model.from_pretrained('gpt2')\n",
        "# https://huggingface.co/pytorch-transformers/serialization.html"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Tokenizers\n",
        "\n",
        "- Byte-Pair Encoding (used by GPT-2)\n",
        "- WordPiece (used by BERT)\n",
        "- Unigram (used by T5)\n",
        "\n",
        "[SentencePiece](https://github.com/google/sentencepiece) is a tokenization algorithm for the preprocessing of text.\n",
        "\n",
        "[Unicode normalization](http://www.unicode.org/reports/tr15/) (such as NFC or NFKC), can also be applied by tokenizer.\n",
        "\n",
        "HTML tokenizers \n",
        "- by W3: https://www.w3.org/TR/2011/WD-html5-20110113/tokenization.html\n",
        "- by spact: https://pypi.org/project/spacy-html-tokenizer/\n",
        "\n",
        "## Tokenization Process\n",
        "\n",
        "![tokenizer](https://huggingface.co/datasets/huggingface-course/documentation-images/resolve/main/en/chapter6/tokenization_pipeline.svg)\n",
        "\n",
        "[Credits](https://huggingface.co/course/chapter6/4?fw=pt)"
      ],
      "metadata": {
        "id": "VrhXMEcgngsE"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "88hdf-ccy7xk"
      },
      "source": [
        "# Chunks and Chunking\n",
        "\n",
        "Segments and labels multi-token sequences.\n",
        "\n",
        "- Each of these larger boxes is called a chunk. \n",
        "- Like tokenization, which omits whitespace, chunking usually selects a subset of the tokens. \n",
        "- The pieces produced by a chunker do not overlap in the source text.\n",
        "\n",
        "<img src=\"https://www.nltk.org/images/chunk-segmentation.png\" width=\"400\">\n",
        "<center>Segmentation and Labeling at both the Token and Chunk Levels</center>\n",
        "\n",
        "<img src=\"https://www.nltk.org/images/chunk-tagrep.png\" width=\"400\">\n",
        "<center>Tag Representation of Chunk Structures</center>\n",
        "\n",
        "<img src=\"https://www.nltk.org/images/chunk-treerep.png\" width=\"400\">\n",
        "<center>Tree Representation of Chunk Structures</center>\n",
        "\n",
        "Credits: https://www.nltk.org/book/ch07.html"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bq7yy95fy7xk"
      },
      "source": [
        "# Chinks and Chinking\n",
        "\n",
        "Chink is a sequence of tokens that is not included in a chunk.\n",
        "\n",
        "Credits: https://www.nltk.org/book/ch07.html"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OZr8C40Qy7xk"
      },
      "source": [
        "# Part-of-speech (POS) Tagging\n",
        "\n",
        "Assigning word types to tokens like verb or noun.\n",
        "\n",
        "POS tagging should be done straight after tokenization and before any words are removed so that sentence structure is preserved and it is more obvious what part of speech the word belongs to.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "qmJNlo1Jy7xk",
        "outputId": "4d1368b5-bcd0-496f-a8cd-d3dfdd7110c7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[('Asian', 'ADJ', 'adjective'),\n",
              " ('shares', 'NOUN', 'noun'),\n",
              " ('skidded', 'VERB', 'verb'),\n",
              " ('on', 'ADP', 'adposition'),\n",
              " ('Tuesday', 'PROPN', 'proper noun'),\n",
              " ('after', 'ADP', 'adposition'),\n",
              " ('a', 'DET', 'determiner'),\n",
              " ('rout', 'NOUN', 'noun'),\n",
              " ('in', 'ADP', 'adposition'),\n",
              " ('tech', 'NOUN', 'noun'),\n",
              " ('stocks', 'NOUN', 'noun'),\n",
              " ('put', 'VERB', 'verb'),\n",
              " ('Wall', 'PROPN', 'proper noun'),\n",
              " ('Street', 'PROPN', 'proper noun'),\n",
              " ('to', 'ADP', 'adposition'),\n",
              " ('the', 'DET', 'determiner'),\n",
              " ('sword', 'NOUN', 'noun')]"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "text = \"Asian shares skidded on Tuesday after a rout in tech stocks put Wall Street to the sword\"\n",
        "doc = nlp(text)\n",
        "[(x.orth_, x.pos_, spacy.explain(x.pos_)) for x in [token for token in doc]]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "9tQmuq3Uy7xk",
        "outputId": "34ac0ae1-09b4-4369-8bcc-b1e30e8467df"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[('Asian', 'JJ', 'adjective'),\n",
              " ('shares', 'NNS', 'noun, plural'),\n",
              " ('skidded', 'VBD', 'verb, past tense'),\n",
              " ('on', 'IN', 'conjunction, subordinating or preposition'),\n",
              " ('Tuesday', 'NNP', 'noun, proper singular'),\n",
              " ('after', 'IN', 'conjunction, subordinating or preposition'),\n",
              " ('a', 'DT', 'determiner'),\n",
              " ('rout', 'NN', 'noun, singular or mass'),\n",
              " ('in', 'IN', 'conjunction, subordinating or preposition'),\n",
              " ('tech', 'NN', 'noun, singular or mass'),\n",
              " ('stocks', 'NNS', 'noun, plural'),\n",
              " ('put', 'VBD', 'verb, past tense'),\n",
              " ('Wall', 'NNP', 'noun, proper singular'),\n",
              " ('Street', 'NNP', 'noun, proper singular'),\n",
              " ('to', 'IN', 'conjunction, subordinating or preposition'),\n",
              " ('the', 'DT', 'determiner'),\n",
              " ('sword', 'NN', 'noun, singular or mass')]"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "[(x.orth_, x.tag_, spacy.explain(x.tag_)) for x in [token for token in doc]]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "Vb_ZJ7idy7xl",
        "outputId": "e400cd06-42ee-444d-f6f9-032766f8d475"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[('Asian', 'JJ'),\n",
              " ('shares', 'NNS'),\n",
              " ('skidded', 'VBN'),\n",
              " ('on', 'IN'),\n",
              " ('Tuesday', 'NNP'),\n",
              " ('after', 'IN'),\n",
              " ('a', 'DT'),\n",
              " ('rout', 'NN'),\n",
              " ('in', 'IN'),\n",
              " ('tech', 'JJ'),\n",
              " ('stocks', 'NNS'),\n",
              " ('put', 'VBD'),\n",
              " ('Wall', 'NNP'),\n",
              " ('Street', 'NNP'),\n",
              " ('to', 'TO'),\n",
              " ('the', 'DT'),\n",
              " ('sword', 'NN')]"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# using nltk\n",
        "import nltk\n",
        "\n",
        "tokens = nltk.word_tokenize(text)\n",
        "pos_tags = nltk.pos_tag(tokens)\n",
        "pos_tags"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vpmGFzTIy7xl"
      },
      "source": [
        "## BILUO tagging\n",
        "\n",
        "- BEGIN - The first token of a multi-token entity.\n",
        "- IN - An inner token of a multi-token entity.\n",
        "- LAST - The final token of a multi-token entity.\n",
        "- UNIT - A single-token entity.\n",
        "- OUT - A non-entity token."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NS9ly_Ezy7xl",
        "outputId": "6f89a888-8d10-46ac-a4c2-42448e9e6c80"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[(Asian, 'B', 'NORP'),\n",
              " (shares, 'O', ''),\n",
              " (skidded, 'O', ''),\n",
              " (on, 'O', ''),\n",
              " (Tuesday, 'B', 'DATE'),\n",
              " (after, 'O', ''),\n",
              " (a, 'O', ''),\n",
              " (rout, 'O', ''),\n",
              " (in, 'O', ''),\n",
              " (tech, 'O', ''),\n",
              " (stocks, 'O', ''),\n",
              " (put, 'O', ''),\n",
              " (Wall, 'O', ''),\n",
              " (Street, 'O', ''),\n",
              " (to, 'O', ''),\n",
              " (the, 'O', ''),\n",
              " (sword, 'O', '')]"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "[(token, token.ent_iob_, token.ent_type_) for token in doc]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hxenu67vy7xl"
      },
      "source": [
        "# Stemming\n",
        "\n",
        "Stemming is the process of reducing words to their root form.\n",
        "\n",
        "Examples:\n",
        "- cats, catlike, catty → cat\n",
        "- fishing, fished, fisher → fish\n",
        "\n",
        "There are two types of stemmers in NLTK: [Porter Stemmer](https://tartarus.org/martin/PorterStemmer/) and [Snowball stemmers](https://tartarus.org/martin/PorterStemmer/)\n",
        "\n",
        "[Credits](https://stackabuse.com/python-for-nlp-tokenization-stemming-and-lemmatization-with-spacy-library/)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mrZei4HTy7xl",
        "outputId": "e841dc59-6add-4f65-e81c-ba85454e6eb4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "compute --> comput\n",
            "computer --> comput\n",
            "computed --> comput\n",
            "computing --> comput\n"
          ]
        }
      ],
      "source": [
        "import nltk\n",
        "from nltk.stem.porter import *\n",
        "\n",
        "stemmer = PorterStemmer()\n",
        "tokens = ['compute', 'computer', 'computed', 'computing']\n",
        "for token in tokens:\n",
        "    print(token + ' --> ' + stemmer.stem(token))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bZ18upXjy7xl"
      },
      "source": [
        "# Lemmatization\n",
        "\n",
        "Assigning the base form of word, for example: \n",
        "- \"was\" → \"be\"\n",
        "- \"rats\" → \"rat\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "G5diSm4oy7xm",
        "outputId": "d5cd01ce-daf4-4429-fccb-058e1b3baa6d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[('Was', 'be'),\n",
              " ('Google', 'Google'),\n",
              " ('founded', 'found'),\n",
              " ('in', 'in'),\n",
              " ('early', 'early'),\n",
              " ('1990', '1990'),\n",
              " ('?', '?')]"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "doc = nlp(\"Was Google founded in early 1990?\")\n",
        "[(x.orth_, x.lemma_) for x in [token for token in doc]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "miPBIz1zy7xm"
      },
      "source": [
        "# Sentence Detection\n",
        "\n",
        "Finding and segmenting individual sentences."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rcHoBOniy7xm",
        "outputId": "aacc9dcd-dfbe-49b9-fe0b-968178abd12a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['Larry Page founded Google in early 1990.', 'Sergey Brin joined.']"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "doc = nlp(\"Larry Page founded Google in early 1990. Sergey Brin joined.\")\n",
        "[sent.text for sent in doc.sents]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MSazimUBy7xm"
      },
      "source": [
        "# Dependency Parsing\t\n",
        "\n",
        "Assigning syntactic dependency labels, describing the relations between individual tokens, like subject or object."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "t3RgXrbNy7xn",
        "outputId": "920bd28c-4fda-4448-e332-6dd64ed8c0ae"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[('We', 'nsubj', 'nominal subject'),\n",
              " ('are', 'aux', 'auxiliary'),\n",
              " ('reading', 'ROOT', None),\n",
              " ('a', 'det', 'determiner'),\n",
              " ('text', 'dobj', 'direct object'),\n",
              " ('.', 'punct', 'punctuation')]"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "doc = nlp(\"We are reading a text.\")\n",
        "# Dependency labels\n",
        "[(x.orth_, x.dep_, spacy.explain(x.dep_)) for x in [token for token in doc]]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NKH56Gnly7xn",
        "outputId": "36f749e6-1fbd-48bc-f215-ff50aed68b8d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['reading', 'reading', 'reading', 'text', 'reading', 'reading']"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Syntactic head token (governor)\n",
        "[token.head.text for token in doc]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MrHm0DWxy7xn"
      },
      "source": [
        "# Base noun phrases\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FQPiH9EBy7xn",
        "outputId": "d0ed46d2-cd14-489b-f0d8-b72752f58ce7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['I', 'a red car']"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "doc = nlp(\"I have a red car\")\n",
        "[chunk.text for chunk in doc.noun_chunks]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHOiRRBNy7xn"
      },
      "source": [
        "# Named Entity Recognition (NER)\n",
        "\n",
        "What is NER? Labeling \"real-world\" objects, like persons, companies or locations.\n",
        "\n",
        "2 popular approaches:\n",
        "- Rule-based\n",
        "- ML-based:\n",
        "    - Multi-class classification\n",
        "    - Conditional Random Field (probabilistic graphical model)\n",
        "\n",
        "Datasets:\n",
        "- [Kaggle, IOB, POS tags](https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus/home)\n",
        "\n",
        "Credits: https://medium.com/@yingbiao/ner-with-bert-in-action-936ff275bc73\n",
        "\n",
        "\n",
        "Entities supported by spacy:\n",
        "- PERSON\tPeople, including fictional.\n",
        "- NORP\tNationalities or religious or political groups.\n",
        "- FAC\tBuildings, airports, highways, bridges, etc.\n",
        "- ORG\tCompanies, agencies, institutions, etc.\n",
        "- GPE\tCountries, cities, states.\n",
        "- LOC\tNon-GPE locations, mountain ranges, bodies of water.\n",
        "- PRODUCT\tObjects, vehicles, foods, etc. (Not services.)\n",
        "- EVENT\tNamed hurricanes, battles, wars, sports events, etc.\n",
        "- WORK_OF_ART\tTitles of books, songs, etc.\n",
        "- LAW\tNamed documents made into laws.\n",
        "- LANGUAGE\tAny named language.\n",
        "- DATE\tAbsolute or relative dates or periods.\n",
        "- TIME\tTimes smaller than a day.\n",
        "- PERCENT\tPercentage, including ”%“.\n",
        "- MONEY\tMonetary values, including unit.\n",
        "- QUANTITY\tMeasurements, as of weight or distance.\n",
        "- ORDINAL\t“first”, “second”, etc.\n",
        "- CARDINAL\tNumerals that do not fall under another type.\n",
        "\n",
        "## Alternatives to spacy\n",
        "\n",
        "[LexNLP](https://lexpredict-lexnlp.readthedocs.io/en/latest/modules/extract/extract.html#pattern-based-extraction-methods) entities:\n",
        "- acts, e.g., “section 1 of the Advancing Hope Act, 1986”\n",
        "- amounts, e.g., “ten pounds” or “5.8 megawatts”\n",
        "- citations, e.g., “10 U.S. 100” or “1998 S. Ct. 1”\n",
        "- companies, e.g., “Lexpredict LLC”\n",
        "- conditions, e.g., “subject to …” or “unless and until …”\n",
        "- constraints, e.g., “no more than” or “\n",
        "- copyright, e.g., “(C) Copyright 2000 Acme”\n",
        "- courts, e.g., “Supreme Court of New York”\n",
        "- CUSIP, e.g., “392690QT3”\n",
        "- dates, e.g., “June 1, 2017” or “2018-01-01”\n",
        "- definitions, e.g., “Term shall mean …”\n",
        "- distances, e.g., “fifteen miles”\n",
        "- durations, e.g., “ten years” or “thirty days”\n",
        "- geographic and geopolitical entities, e.g., “New York” or “Norway”\n",
        "- money and currency usages, e.g., “$5” or “10 Euro”\n",
        "- percents and rates, e.g., “10%” or “50 bps”\n",
        "- PII, e.g., “212-212-2121” or “999-999-9999”\n",
        "- ratios, e.g.,” 3:1” or “four to three”\n",
        "- regulations, e.g., “32 CFR 170”\n",
        "- trademarks, e.g., “MyApp (TM)”\n",
        "- URLs, e.g., “http://acme.com/”\n",
        "\n",
        "Stanford NER entities:\n",
        "- Location, Person, Organization, Money, Percent, Date, Time\n",
        "\n",
        "NLTK\n",
        "- NLTK maximum entropy classifier\n",
        "\n",
        "Transformer Models (on HuggingFace)\n",
        "- [GPT-NeoX](https://github.com/EleutherAI/gpt-neox)\n",
        "- [BERT](https://huggingface.co/dslim/bert-base-NER?text=My+name+is+Sarah+and+I+live+in+London)\n",
        "- [BERT-large](https://huggingface.co/dslim/bert-large-NER)\n",
        "- [camemBERT](https://huggingface.co/Jean-Baptiste/camembert-ner)\n",
        "- [Electra](https://huggingface.co/dbmdz/electra-large-discriminator-finetuned-conll03-english)\n",
        "- [NER-English-Large](https://huggingface.co/flair/ner-english-large) (best so far for org NER)\n",
        "- [XML-RoBERTa](https://huggingface.co/xlm-roberta-large-finetuned-conll03-english)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_4DKBDBwy7xn",
        "outputId": "1a22fa99-c406-4fbf-f94a-1d69b065a41f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[('Larry Page', 'PERSON'),\n",
              " ('Google', 'ORG'),\n",
              " ('US', 'GPE'),\n",
              " ('early 1990', 'DATE')]"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "doc = nlp(\"Larry Page founded Google in the US in early 1990.\")\n",
        "# Text and label of named entity span\n",
        "[(ent.text, ent.label_) for ent in doc.ents]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k9BsImaVy7xo",
        "outputId": "6620a2a9-cb94-44eb-ac54-4fec28410b6a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[('European', 'NORP'),\n",
              " ('Google', 'ORG'),\n",
              " ('$5.1 billion', 'MONEY'),\n",
              " ('Wednesday', 'DATE')]"
            ]
          },
          "execution_count": 49,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "doc = nlp('European authorities fined Google a record $5.1 billion on Wednesday for abusing its power in the mobile phone market and ordered the company to alter its practices')\n",
        "[(X.text, X.label_) for X in doc.ents]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P9IbNWCry7xo",
        "outputId": "04d6c666-f985-49d4-8e11-5c960195a67d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Counter({'NORP': 1, 'ORG': 1, 'MONEY': 1, 'DATE': 1})"
            ]
          },
          "execution_count": 50,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from collections import Counter\n",
        "\n",
        "labels = [x.label_ for x in doc.ents]\n",
        "Counter(labels)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "Uz7EWowNy7xo",
        "outputId": "fadbe300-558e-42f7-9c9a-fd65f6501378"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[(European, 'B', 'NORP'),\n",
              " (authorities, 'O', ''),\n",
              " (fined, 'O', ''),\n",
              " (Google, 'B', 'ORG'),\n",
              " (a, 'O', ''),\n",
              " (record, 'O', ''),\n",
              " ($, 'B', 'MONEY'),\n",
              " (5.1, 'I', 'MONEY'),\n",
              " (billion, 'I', 'MONEY'),\n",
              " (on, 'O', ''),\n",
              " (Wednesday, 'B', 'DATE'),\n",
              " (for, 'O', ''),\n",
              " (abusing, 'O', ''),\n",
              " (its, 'O', ''),\n",
              " (power, 'O', ''),\n",
              " (in, 'O', ''),\n",
              " (the, 'O', ''),\n",
              " (mobile, 'O', ''),\n",
              " (phone, 'O', ''),\n",
              " (market, 'O', ''),\n",
              " (and, 'O', ''),\n",
              " (ordered, 'O', ''),\n",
              " (the, 'O', ''),\n",
              " (company, 'O', ''),\n",
              " (to, 'O', ''),\n",
              " (alter, 'O', ''),\n",
              " (its, 'O', ''),\n",
              " (practices, 'O', '')]"
            ]
          },
          "execution_count": 51,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "[(X, X.ent_iob_, X.ent_type_) for X in doc]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "esXqsvAry7xp",
        "outputId": "2804c1bf-4dff-4f66-9f19-29f3b0594441"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['European', 'Google', '$5.1 billion', 'Wednesday']\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "[('European', 1), ('Google', 1), ('$5.1 billion', 1)]"
            ]
          },
          "execution_count": 54,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Show Begin and In entities\n",
        "items = [x.text for x in doc.ents]\n",
        "print(items)\n",
        "Counter(items).most_common(3)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "TEmIu_uCy7xp"
      },
      "outputs": [],
      "source": [
        "import lexnlp.extract.en as lexnlp\n",
        "import nltk"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3ldXcrJ7y7xp",
        "outputId": "93c47fb0-4b67-4f23-afdc-a4ccdbe14dc8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[10, 2.0]\n"
          ]
        }
      ],
      "source": [
        "text = \"There are ten cows in the 2 acre pasture.\"\n",
        "print(list(lexnlp.amounts.get_amounts(text)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xWUQs8joy7xp",
        "outputId": "c79c8619-7f6d-452a-eb8e-f730d9b37d52"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'location_start': 5,\n",
              "  'location_end': 49,\n",
              "  'act_name': 'VERY Important Act',\n",
              "  'section': '12',\n",
              "  'year': '1954',\n",
              "  'ambiguous': False,\n",
              "  'value': 'section 12 of the VERY Important Act of 1954'}]"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import lexnlp.extract.en.acts\n",
        "text = \"test section 12 of the VERY Important Act of 1954.\"\n",
        "lexnlp.extract.en.acts.get_act_list(text)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## NER with flair's NER-English-Large\n",
        "\n",
        "Source: https://huggingface.co/flair/ner-english-large\n",
        "\n",
        "Available tags:\n",
        "- PER, person\n",
        "- LOC, location\n",
        "- ORG, organization\n",
        "- MISC, other name"
      ],
      "metadata": {
        "id": "Rb35B4Av3OV0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install flair"
      ],
      "metadata": {
        "id": "Hus7RC2a3m-A"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from flair.data import Sentence\n",
        "from flair.models import SequenceTagger\n",
        "\n",
        "# load tagger\n",
        "tagger = SequenceTagger.load(\"flair/ner-english-large\")"
      ],
      "metadata": {
        "id": "LtHFYhXI3NNL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# make example sentence\n",
        "sentence = Sentence(\"George Washington went to Washington\")\n",
        "\n",
        "# predict NER tags\n",
        "tagger.predict(sentence)\n",
        "\n",
        "# print sentence\n",
        "print(sentence)\n",
        "\n",
        "# print predicted NER spans\n",
        "print('The following NER tags are found:')\n",
        "\n",
        "# iterate over entities and print\n",
        "for entity in sentence.get_spans('ner'):\n",
        "    print(entity)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mA4Qo1m-4XfY",
        "outputId": "bdad53b2-7eee-4ea3-9b18-c35c8648183e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sentence: \"George Washington went to Washington\" → [\"George Washington\"/PER, \"Washington\"/LOC]\n",
            "The following NER tags are found:\n",
            "Span[0:2]: \"George Washington\" → PER (1.0)\n",
            "Span[4:5]: \"Washington\" → LOC (1.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"We are the platform of choice for customers' SAP workloads in the cloud, companies like Thabani, Munich Re's, Sodexo, Volvo Cars, all run SAP on Azure. We are the only cloud provider with direct and secure access to Oracle databases running an Oracle Cloud infrastructure, making it possible for companies like FedEx, GE, and Marriott to use capabilities from both companies. And with Azure Confidential Computing, we're enabling companies in highly regulated industries, including RBC, to bring their most sensitive applications to the cloud. Just last week, UBS said it will move more than 50% of its applications to Azure.\"\n",
        "\n",
        "# make example sentence\n",
        "sentence = Sentence(text)\n",
        "\n",
        "# predict NER tags\n",
        "tagger.predict(sentence)\n",
        "\n",
        "# print sentence\n",
        "print(sentence)\n",
        "\n",
        "# print predicted NER spans\n",
        "print('\\nThe following NER tags are found:\\n')\n",
        "\n",
        "# iterate over entities and print\n",
        "for entity in sentence.get_spans('ner'):\n",
        "    print(entity)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O8R7ngGR5MQH",
        "outputId": "f1c89f54-d6f3-422c-95b1-dadab297a712"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sentence: \"We are the platform of choice for customers' SAP workloads in the cloud , companies like Thabani , Munich Re 's , Sodexo , Volvo Cars , all run SAP on Azure . We are the only cloud provider with direct and secure access to Oracle databases running an Oracle Cloud infrastructure , making it possible for companies like FedEx , GE , and Marriott to use capabilities from both companies . And with Azure Confidential Computing , we 're enabling companies in highly regulated industries , including RBC , to bring their most sensitive applications to the cloud . Just last week , UBS said it will move more than 50 % of its applications to Azure .\" → [\"SAP\"/ORG, \"Thabani\"/ORG, \"Munich Re\"/ORG, \"Sodexo\"/ORG, \"Volvo Cars\"/ORG, \"SAP\"/ORG, \"Azure\"/MISC, \"Oracle\"/ORG, \"Oracle Cloud\"/MISC, \"FedEx\"/ORG, \"GE\"/ORG, \"Marriott\"/ORG, \"Azure Confidential Computing\"/MISC, \"RBC\"/ORG, \"UBS\"/ORG, \"Azure\"/MISC]\n",
            "\n",
            "The following NER tags are found:\n",
            "\n",
            "Span[8:9]: \"SAP\" → ORG (0.9945)\n",
            "Span[16:17]: \"Thabani\" → ORG (1.0)\n",
            "Span[18:20]: \"Munich Re\" → ORG (0.9604)\n",
            "Span[22:23]: \"Sodexo\" → ORG (1.0)\n",
            "Span[24:26]: \"Volvo Cars\" → ORG (1.0)\n",
            "Span[29:30]: \"SAP\" → ORG (0.9995)\n",
            "Span[31:32]: \"Azure\" → MISC (0.9974)\n",
            "Span[45:46]: \"Oracle\" → ORG (0.9997)\n",
            "Span[49:51]: \"Oracle Cloud\" → MISC (1.0)\n",
            "Span[59:60]: \"FedEx\" → ORG (1.0)\n",
            "Span[61:62]: \"GE\" → ORG (1.0)\n",
            "Span[64:65]: \"Marriott\" → ORG (1.0)\n",
            "Span[74:77]: \"Azure Confidential Computing\" → MISC (0.999)\n",
            "Span[88:89]: \"RBC\" → ORG (1.0)\n",
            "Span[104:105]: \"UBS\" → ORG (1.0)\n",
            "Span[117:118]: \"Azure\" → MISC (0.9993)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OTEQ8SmZy7xp"
      },
      "source": [
        "# Text Classification\n",
        "\n",
        "Two types:\n",
        "- binary classification (text only belongs to one class)\n",
        "- multi-class classification (text can belong to multiple classes)\n",
        "\n",
        "Assigning categories or labels to a whole document, or parts of a document.\n",
        "\n",
        "Approach:\n",
        "- calculate document vectors for each document\n",
        "- use kNN to calculate clusters based on document vectors\n",
        "- each cluster represents a class of documents that are similar to each other"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "KPWPb4Ydy7xp"
      },
      "outputs": [],
      "source": [
        "# Credits: https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html\n",
        "import re\n",
        "from torchtext.data.utils import ngrams_iterator\n",
        "from torchtext.data.utils import get_tokenizer\n",
        "\n",
        "ag_news_label = {1 : \"World\",\n",
        "                 2 : \"Sports\",\n",
        "                 3 : \"Business\",\n",
        "                 4 : \"Sci/Tec\"}\n",
        "\n",
        "def predict(text, model, vocab, ngrams):\n",
        "    tokenizer = get_tokenizer(\"basic_english\")\n",
        "    with torch.no_grad():\n",
        "        text = torch.tensor([vocab[token]\n",
        "                            for token in ngrams_iterator(tokenizer(text), ngrams)])\n",
        "        output = model(text, torch.tensor([0]))\n",
        "        return output.argmax(1).item() + 1\n",
        "\n",
        "ex_text_str = \"MEMPHIS, Tenn. – Four days ago, Jon Rahm was \\\n",
        "    enduring the season’s worst weather conditions on Sunday at The \\\n",
        "    Open on his way to a closing 75 at Royal Portrush, which \\\n",
        "    considering the wind and the rain was a respectable showing. \\\n",
        "    Thursday’s first round at the WGC-FedEx St. Jude Invitational \\\n",
        "    was another story. With temperatures in the mid-80s and hardly any \\\n",
        "    wind, the Spaniard was 13 strokes better in a flawless round. \\\n",
        "    Thanks to his best putting performance on the PGA Tour, Rahm \\\n",
        "    finished with an 8-under 62 for a three-stroke lead, which \\\n",
        "    was even more impressive considering he’d never played the \\\n",
        "    front nine at TPC Southwind.\"\n",
        "\n",
        "vocab = train_dataset.get_vocab()\n",
        "model = model.to(\"cpu\")\n",
        "\n",
        "print(\"This is a %s news\" %ag_news_label[predict(ex_text_str, model, vocab, 2)])\n",
        "\n",
        "# Output: This is a Sports news"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4jwqE8_hy7xq"
      },
      "source": [
        "## CountVectorizer \n",
        "- Convert a collection of text documents to a matrix of token counts\n",
        "- [skikitLearn](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0XDaR8eqy7xq"
      },
      "source": [
        "# Text Similarity Metrics\n",
        "\n",
        "We can calculate the similarity between two (and more):\n",
        "- Characters: `A` vs `a`\n",
        "- Words: `cat` vs `cats`\n",
        "- Tokens: `co-chief` vs `-chief`\n",
        "- Sentences: `I love cats` vs `I love dogs`\n",
        "- Documents: a set of sentences\n",
        "\n",
        "Two types of measures exist returing one of two outputs:\n",
        "1. `True` or `False` (binary), i.e. two inputs are exactly the same or they are not - nothing in between.\n",
        "2. Floating, i.e. 95% of the two given inputs are the same, 5% are not the same. Let's take `cat` vs `cats`: 3 out of 4 (75%) characters are the same, 1 out of 4 (25%) is different. \n",
        "\n",
        "It makes only sense to use type `1.` metrics when the inputs are always of equal length. For example, using any type `1.` metric to compare `cat` and `cats` always results in \"not equal\". On the other hand, type `2.` metrics are applicable to both, equal and non-equal length inputs.\n",
        "\n",
        "[> Overview of semantic similarity](https://en.wikipedia.org/wiki/Semantic_similarity)\n",
        "\n",
        "## Popular Similarity Metrics\n",
        "\n",
        "Similarity metrics are mostly calculated with vector representations of words, sentences or documents. For example,\n",
        "- One-hot encoded vector of two words \n",
        "- Bag of words vector of two sentences\n",
        "- TF-IDF vector of two sentences or documents\n",
        "\n",
        "### Metrics for equal length inputs\n",
        "\n",
        "The following metrics assume two inputs having equal length. If two inputs don't have the same length, they can be normalizable by using padding characters.\n",
        "\n",
        "**Hamming distance** is measured between two strings of equal length and defined as the number of positions that have different characters.\n",
        "\n",
        "**Manhatten distance** (L1 norm, city block distance, taxicab) counts the number of mismatches by subtracting the difference between each pair of characters at each position of two strings.\n",
        "\n",
        "**Euclidean distance** (L2 norm) is defined as the shortest straight-line distance between two points.\n",
        "\n",
        "### Metrics for (non-)equal length inputs\n",
        "\n",
        "The following metrics work with inputs of equal and non-equal length.\n",
        "\n",
        "**[Cosine distance](https://studymachinelearning.com/cosine-similarity-text-similarity-metric/)** (L2-normalized dot product of vectors) measures the similarity by using the normalized length of two input vectors. Order of characters/words are not taken into account.\n",
        "\n",
        "**[Jaccard](https://studymachinelearning.com/jaccard-similarity-text-similarity-metric-in-nlp/) similarity** indicates how many words two documents share by using the intersection and unions of the words.\n",
        "\n",
        "**Levenshtein distance** measures the minimum number of edits needed to transform one input into the other. Considers order of characters or words in input.\n",
        "\n",
        "**Jaro Winkler distance** minimum edit distance, considers prefixes.\n",
        "\n",
        "**Okapi BM25(F) ranking** takes token distributions across corpus into account.\n",
        "\n",
        "```\n",
        "q_idf * dot(q_tf, d_tf[i]) * 1.5 (dot(q_tf, d_tf[i]) + .25 + .75 * d_num_words[i] / d_num_words.mean()))\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RR6-VxPcy7xq"
      },
      "source": [
        "## Cosine Similarity\n",
        "\n",
        "A document, sentence or word is represented as a vector and the Cosine sim calculates the angle (=\"similarity\") between two vectors. \n",
        "\n",
        "The resulting similarity ranges from:\n",
        "\n",
        "- 1 if the vectors are the same\n",
        "- 0 if the vectors don’t have any relationship (orthogonal vectors)\n",
        "\n",
        "![cos-sim](https://studymachinelearning.com/wp-content/uploads/2019/09/Cosine-similarity-Wikipedia.png)\n",
        "\n",
        "Similarity measurements for:\n",
        "- Term frequency vectors (bag of words)\n",
        "- TF-IDF vectors\n",
        "- Oc-occurence vectors\n",
        "\n",
        "\n",
        "### Example 1\n",
        "\n",
        "Figure below shows three word vectors and Cosine distance (=similarity) between \n",
        "- \"I hate cats\" and \"I love dogs\" (result: not very similar)\n",
        "- \"I love dogs\" and \"I love, love, love, .. dogs\" (result: similar)\n",
        "\n",
        "<img src=\"https://miro.medium.com/max/700/1*QNp4TNCNwo1HMqjFV4jq1g.png\" width=\"600\">\n",
        "\n",
        "[Credits](https://towardsdatascience.com/group-thousands-of-similar-spreadsheet-text-cells-in-seconds-2493b3ce6d8d)\n",
        "\n",
        "### Example 2\n",
        "\n",
        "```\n",
        "doc_1 = \"Data is the oil of the digital economy\" \n",
        "doc_2 = \"Data is a new oil\" \n",
        "\n",
        "# Vector representation of the document\n",
        "doc_1_vector = [1, 1, 1, 1, 0, 1, 1, 2]\n",
        "doc_2_vector = [1, 0, 0, 1, 1, 0, 1, 0]\n",
        "```\n",
        "\n",
        "![vec](https://studymachinelearning.com/wp-content/uploads/2019/09/word_cnt_vect_cosine_similarity.png)\n",
        "\n",
        "![vec-calc](https://studymachinelearning.com/wp-content/uploads/2019/09/cosine_similarity_example_1.png)\n",
        "\n",
        "[Credits](https://studymachinelearning.com/cosine-similarity-text-similarity-metric/)\n",
        "\n",
        "---\n",
        "\n",
        "Cosine sim with scikit: [sklearn.metrics.pairwise.cosine_similarity](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.html#sklearn.metrics.pairwise.cosine_similarity)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Jaccard Similarity\n",
        "\n",
        "Jaccard sim is calculated by dividing the number of words occuring in both documents/sentences (intersection) with the number of all words in both docs/sents (union).\n",
        "\n",
        "### Example \n",
        "```\n",
        "doc_1 = \"Data is the new oil of the digital economy\"\n",
        "doc_2 = \"Data is a new oil\"\n",
        "```\n",
        "\n",
        "Each sentence is tokenized into words.\n",
        "\n",
        "```\n",
        "words_doc1 = {'data', 'is', 'the', 'new', 'oil', 'of', 'digital', 'economy'}\n",
        "words_doc2 = {'data', 'is', 'a', 'new', 'oil'}\n",
        "```\n",
        "\n",
        "Four words occur in both sentences (intersection): data, is, new, oil.\n",
        "\n",
        "Across both sentences, nine unique words exist (union): data, a, of, is, economy, the, new, digital, oil.\n",
        "\n",
        "![jac-sim1](https://studymachinelearning.com/wp-content/uploads/2020/04/jaccard_example.png)\n",
        "\n",
        "Visualized:\n",
        "\n",
        "![jac-sim2](https://studymachinelearning.com/wp-content/uploads/2020/04/intersection_example.png)"
      ],
      "metadata": {
        "id": "NTdUVxD79Cse"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Levenshtein Distance\n",
        "\n",
        "Levenshtein distance between two words is the minimum number of single-character edits (i.e. insertions, deletions or substitutions) required to change one word into the other. \n",
        "\n",
        "The distance can also be calculated for sentences, i.e. the minimum number of single token edits required to change one sentence into the other.\n",
        "\n",
        "Levenshtein distance for two strings `a`, `b` of length `|a|` and `|b|` is given by `lev`:\n",
        "\n",
        "![lev-dic](https://wikimedia.org/api/rest_v1/media/math/render/svg/70962a722b0b682e398f0ee77d60c714a441c54e)\n",
        "\n",
        "### Example 1\n",
        "\n",
        "Levenshtein distance between `HONDA` and `HYUNDAI` is 3 because it takes three transformations to change `HONDA` to `HYUNDAI`:\n",
        "\n",
        "1. Add `Y` to `HONDA` => `HYONDA`\n",
        "2. Substitue `O` with `U` in `HYONDA` => `HYUNDA`\n",
        "3. Add `I` to `HYUNDA` => `HYUNDAI`\n",
        "\n",
        "### Example 2\n",
        "\n",
        "Lev distance between two sentences `I love cats` and `I love dogs` is 1.\n",
        "\n",
        "Step 1: tokenize both sentences to `[\"I\", \"love\", \"cats\"]`, `[\"I\", \"love\", \"docs\"]`\n",
        "\n",
        "Step 2: perform one transformation, i.e. replace `cats` with `dogs`.\n"
      ],
      "metadata": {
        "id": "YFaYjZuTktek"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Jaro-Winkler Distance\n",
        "\n",
        "![img](https://miro.medium.com/max/720/1*0efk-HIrKHqv_uimWWwhDQ.jpeg)"
      ],
      "metadata": {
        "id": "cNZ3RL41Ve_i"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6yTF26dqy7xq"
      },
      "source": [
        "## L2 Norm\n",
        "\n",
        "Length of a word vector. Also known as **Euclidean norm**.\n",
        "\n",
        "Example:\n",
        "- length of \"I like cats\" is 4.7"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Pazy3CbQy7xq",
        "outputId": "16da15cc-d0fa-4a65-8b76-941e4ff3b766"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.957709143352323"
            ]
          },
          "execution_count": 54,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "doc1 = nlp(\"I like cats\")\n",
        "doc2 = nlp(\"I like dogs\")\n",
        "# Compare 2 documents\n",
        "doc1.similarity(doc2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fPGAIVZYy7xq",
        "outputId": "d2ea9dfe-51b7-48b4-8ee9-9ad2a82bb11f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.83117634"
            ]
          },
          "execution_count": 59,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# \"cats\" vs \"dogs\"\n",
        "doc1[2].similarity(doc2[2])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "15h3aPuYy7xq",
        "outputId": "c71f0c07-e80e-44ad-ac7d-b7f54acdcfba"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.46475163"
            ]
          },
          "execution_count": 70,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# \"I\" vs \"like dogs\"\n",
        "doc1[0].similarity(doc2[1:3])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "xVLmUsUdy7xr",
        "outputId": "6c3cec59-b74f-4606-b1a3-1da7e177785b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "4.706799587675896"
            ]
          },
          "execution_count": 73,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "doc = nlp(\"I like cats\")\n",
        "# L2 norm of \"I like cats\"\n",
        "doc.vector_norm"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qMJVf-Apy7xr",
        "outputId": "eb475f79-26ce-4e38-dae3-bc87defd29bd"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "6.933004"
            ]
          },
          "execution_count": 76,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# L2 norm of \"cats\"\n",
        "doc[2].vector_norm"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "W9f8Q3tcy7xr",
        "outputId": "e77ee675-8c09-4d9c-8da6-5eda2ff03c74"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([-0.26763  ,  0.029846 , -0.3437   , -0.54409  , -0.49919  ,\n",
              "        0.15928  , -0.35278  , -0.2036   ,  0.23482  ,  1.5671   ,\n",
              "       -0.36458  , -0.028713 , -0.27053  ,  0.2504   , -0.18126  ,\n",
              "        0.13453  ,  0.25795  ,  0.93213  , -0.12841  , -0.18505  ,\n",
              "       -0.57597  ,  0.18538  , -0.19147  , -0.38465  ,  0.21656  ,\n",
              "       -0.4387   , -0.27846  , -0.41339  ,  0.37859  , -0.2199   ,\n",
              "       -0.25907  , -0.019796 , -0.31885  ,  0.12921  ,  0.22168  ,\n",
              "        0.32671  ,  0.46943  , -0.81922  , -0.20031  ,  0.013561 ,\n",
              "       -0.14663  ,  0.14438  ,  0.0098044, -0.15439  ,  0.21146  ,\n",
              "       -0.28409  , -0.4036   ,  0.45355  ,  0.12173  , -0.11516  ,\n",
              "       -0.12235  , -0.096467 , -0.26991  ,  0.028776 , -0.11307  ,\n",
              "        0.37219  , -0.054718 , -0.20297  , -0.23974  ,  0.86271  ,\n",
              "        0.25602  , -0.3064   ,  0.014714 , -0.086497 , -0.079054 ,\n",
              "       -0.33109  ,  0.54892  ,  0.20076  ,  0.28064  ,  0.037788 ,\n",
              "        0.0076729, -0.0050123, -0.11619  , -0.23804  ,  0.33027  ,\n",
              "        0.26034  , -0.20615  , -0.35744  ,  0.54125  , -0.3239   ,\n",
              "        0.093441 ,  0.17113  , -0.41533  ,  0.13702  , -0.21765  ,\n",
              "       -0.65442  ,  0.75733  ,  0.359    ,  0.62492  ,  0.019685 ,\n",
              "        0.21156  ,  0.28125  ,  0.22288  ,  0.026787 , -0.1019   ,\n",
              "        0.11178  ,  0.17202  , -0.20403  , -0.01767  , -0.34351  ,\n",
              "        0.11926  ,  0.73156  ,  0.11094  ,  0.12576  ,  0.64825  ,\n",
              "       -0.80004  ,  0.62074  , -0.38557  ,  0.015614 ,  0.2664   ,\n",
              "        0.18254  ,  0.11678  ,  0.58919  , -1.0639   , -0.29969  ,\n",
              "        0.14827  , -0.42925  , -0.090766 ,  0.12313  , -0.024253 ,\n",
              "       -0.21265  , -0.10331  ,  0.91988  , -1.4097   , -0.0542   ,\n",
              "       -0.071201 ,  0.66878  , -0.24651  , -0.46788  , -0.23991  ,\n",
              "       -0.14138  , -0.038911 , -0.48678  ,  0.22975  ,  0.36074  ,\n",
              "        0.13024  , -0.40091  ,  0.19673  ,  0.016017 ,  0.30575  ,\n",
              "       -2.1901   , -0.55468  ,  0.26955  ,  0.63815  ,  0.42724  ,\n",
              "       -0.070186 , -0.11196  ,  0.14079  , -0.022228 ,  0.070456 ,\n",
              "        0.17229  ,  0.099383 , -0.12258  , -0.23416  , -0.26525  ,\n",
              "       -0.088991 , -0.061554 ,  0.26582  , -0.53112  , -0.4106   ,\n",
              "        0.45211  , -0.39669  , -0.43746  , -0.6632   , -0.048135 ,\n",
              "        0.23171  , -0.37665  , -0.38261  , -0.29286  , -0.036613 ,\n",
              "        0.25354  ,  0.49775  ,  0.3359   , -0.11285  , -0.17228  ,\n",
              "        0.85991  , -0.34081  ,  0.27959  ,  0.03698  ,  0.61782  ,\n",
              "        0.23739  , -0.32049  , -0.073717 ,  0.015991 , -0.37395  ,\n",
              "       -0.4152   ,  0.049221 , -0.3137   ,  0.091128 , -0.38258  ,\n",
              "       -0.036783 ,  0.10902  , -0.38332  , -0.74754  ,  0.016473 ,\n",
              "        0.55256  , -0.29053  , -0.50617  ,  0.83599  , -0.31783  ,\n",
              "       -0.77465  , -0.0049272, -0.17103  , -0.38067  ,  0.44987  ,\n",
              "       -0.12497  ,  0.60263  , -0.12026  ,  0.37368  , -0.079952 ,\n",
              "       -0.15785  ,  0.37684  , -0.18679  ,  0.18855  , -0.4759   ,\n",
              "       -0.11708  ,  0.36999  ,  0.54134  ,  0.42752  ,  0.038618 ,\n",
              "        0.043483 ,  0.31435  , -0.24491  , -0.67818  , -0.33833  ,\n",
              "        0.039218 , -0.11964  ,  0.8474   ,  0.09451  ,  0.070523 ,\n",
              "       -0.2806   ,  0.296    , -0.17554  , -0.41087  ,  0.70748  ,\n",
              "        0.17686  ,  0.043479 , -0.31902  ,  0.64584  , -0.45268  ,\n",
              "       -0.7967   ,  0.099817 , -0.1734   ,  0.11404  , -0.36809  ,\n",
              "        0.12035  , -0.048582 ,  0.55945  , -0.51508  ,  0.072704 ,\n",
              "        0.18106  ,  0.07802  , -0.31526  ,  0.38189  ,  0.092801 ,\n",
              "       -0.044227 , -0.66154  , -0.020428 ,  0.059836 , -0.23628  ,\n",
              "       -0.017592 , -0.56481  , -0.52934  , -0.16392  ,  0.077331 ,\n",
              "        0.24583  , -0.32195  , -0.36811  , -0.037208 ,  0.26702  ,\n",
              "       -0.57907  ,  0.46457  , -0.54636  ,  0.11855  ,  0.092475 ,\n",
              "       -0.10469  ,  0.03319  ,  0.62616  , -0.33684  ,  0.045742 ,\n",
              "        0.25089  ,  0.28973  ,  0.060633 , -0.4096   ,  0.39198  ,\n",
              "        0.58276  ,  0.496    , -0.75881  ,  0.13655  ,  0.21704  ,\n",
              "       -0.37978  , -0.54051  , -0.22813  ,  0.28393  , -0.58739  ,\n",
              "        1.0472   , -0.13318  , -0.07325  ,  0.12991  , -0.44999  ],\n",
              "      dtype=float32)"
            ]
          },
          "execution_count": 77,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Vector representation of \"cats\"\n",
        "doc[2].vector"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "BoOkLYt2y7xr"
      },
      "outputs": [],
      "source": [
        "# can also be done using sklearn's linear kernel (equivilant to cosine similarity)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QXraRi6Jy7xr"
      },
      "source": [
        "# n-grams: Unigram, bigrams, trigrams\n",
        "\n",
        "- Unigram = one word, eg the, and, of, hotel\n",
        "- Bigrams = two consecutive words, eg the hotel, in seattle, the city\n",
        "- Trigrams = three consecutive words, eg easy access to, high speed internet, the heart of\n",
        "\n",
        "Credits: https://towardsdatascience.com/building-a-content-based-recommender-system-for-hotels-in-seattle-d724f0a32070"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w5K-w2ANy7xr"
      },
      "source": [
        "## Get all unigrams"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fCrzOwVHy7xr",
        "outputId": "711cea1c-c3a4-49c2-ef60-e4f690cea8d1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 299
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7fbae1ff3510>"
            ]
          },
          "metadata": {},
          "execution_count": 28
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEICAYAAABBBrPDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAWpElEQVR4nO3debRdZZ3m8e9DgiiDAQUVwhABxQlFCA4LsLGwnRDbZanYouUIWlpVapUtoKjYajelq6pdTZVDxBJoQHCsVnFi0QzlBCSIhFEtDAVBClEIk4qGX/+x96WOl3tfbsw599zcfD9rnZU9nf3+9r7Jfs777n1PUlVIkjSdTcZdgCRpbjMoJElNBoUkqcmgkCQ1GRSSpCaDQpLUZFBo3ktyYJLr/4j3fSLJe4ZUw7FJThnGvqTZZlBsRJLcMfC6J8mvB+YPG1IbJya5e1JbC4ax79lWVW+qqg+Mu465KEkl2X3cdWh2GBQbkaracuIF/BtwyMCyU4fY1IcH26qqtUPcd1OShbPVlkZvQ/2QMd8YFCLJZkk+muSG/vXRJJv16w5Mcn2SdyW5OcmqIfY+rk2yTz99WP8p9fH9/OuT/PM61HdkkhuBzyR5UN+zuSXJFcC+k9o9MsnqJLcnuTrJQdPUd2KSD05q52+S3JTk50le2zi2RyY5r2/jLGDbSetfmOTyJLcmOTfJYwfW7ZTkS0l+keSXSf6hX/4Hw1dJlvTnbGE/f26SDyb5Xt+T+2qShyY5NcltSS5KsmTg/Y9JclaSX/Xn4WWTjv0fk5zZH8MFSXbr153fb/ajvp1DpzkHhye5sn//FUn27pc/tq/11v4cvHBSux9P8vUkdwLP7Jd9oq/19v687jLVORg4D2/op3fvt1/T//09Y7qfmaZnUAjg3cDTgL2AJwFPAY4ZWP8IugvdYuDVwLIkezT29+b+4rMiyZ82tjsPOLCf/k/ANcAzBubPW4f6HgLsAhwBvA/YrX89p68ZgL7uvwD2raqt+vWrGjUOegSwiO48vB74xyTbTLPtacAKuvP2gUk1PBr4LPA2YDvg68BXkzwg3SforwHXAkv6tk6fYX0ALwde1b9vN+D7wGfozs+VdOeGJFsAZ/V1Pqx/38eSPG7Svt4PbAP8FPgQQFVN/Iye1PcY73PxTfJS4Fjgz4AHAy8EfplkU+CrwLf7dv8SOHXS36dX9G1tBXynX3YY3XncFrgEmGkP+AN9W9sAOwLHz/B9GlRVvjbCF93F8Vn99L8Czx9Y9xxgVT99IPB7YIuB9Z8D3jPNfvcGHgosBJ4P3A7sN822rwe+0k9fCbwBOL2fvxbYe4b13Q08cGD9NcBzB+aPAK7vp3cHbgKeBWx6P+foROCDA+38Glg4sP4m4GlTvG/nKc7ZacAp/fR7gM8NrNsEWN238XTgF4PtDGx37MQ++vklQE1sC5wLvHtg/d8B3xiYPwS4pJ8+FPiXSfv/JPC+gWM/YWDd84GrBuYL2L1x7r4FvHWK5QcANwKbDCz7LHDsQLsnT/FzOH1gfktgLbDT5HMwcB7e0E+fDCwDdhz3v7kN+WWPQgA70F2YJ1zbL5twS1Xd2Vh/r6q6uKp+WVW/r6qv033ye/E07Z4HHJBke2ABXQDt1w+PLKL75DiT+n5RVb+ZdDzXTdp+or6f0n2SPxa4KcnpSaY8lin8sqp+PzB/F91Fa7IdmPqcDa4frOmevt7FdBe/aye1sy7+fWD611PMT9S7C/DUfvjn1iS30n1qf8TA9jcOTE93rNPZiS7gJ9sBuK4/5gnX0h37hOu4r3uXVdUdwK+Y5u/gJO8EAlzYD3O9bgbv0SQGhQBuoLtwTNi5XzZhm36oYrr1LUX3D/W+K7qL9l10ww/nV9VtdBenI4DvDFxM7q++yV+B/HO6C9Xg9oPtnlZV+/f7LOBvZ3gsM/Vzpj5nE/7geJKkr3c13QVx50x9U/5OYPOB+UdMsc1MXQecV1VbD7y2rKo/X499Tt7/blMsvwHYKcngtWdnumOfMNVXWt/780yyJd1Q2g105wSmOS9VdWNVHV5VOwBvpBte82mtdWRQCLqu/zFJtkuyLfBeYPIz/+/vx9APAF4AfH6qHSV5SZItk2yS5NnAK4GvNNo+j+6ewcT9iHMnzc+0vkGfA45Osk2SHemCaKK+PZL8Sbqb4b+h+5R9zzT7+aNU1bXAcv7jnO1PN+wzWN/BSQ7qx+z/Bvgt8D3gQrqgOS7JFkkemGS//n2XAM9IsnOSRcDR61Hm14BHJ3lVkk37174ZuKl+P/4d2LWx/gTgHUn2SWf3/gb0BXQfDt7Zt3kg3bm5v/swz0+yf5IH0N13+EFVXVdVv6ALmVcmWdD3GO4NqCQv7f8OANxCF0JD/XlvDAwKAXyQ7sJ2KbASuLhfNuFGun9kN9ANJb2pqq6aZl9vpfuHeyvwEeDwqjq30fZ5dDctz59mfib1TfZ+uuGMn9HdyPw/A+s2A44Dbu6P62Gs3wV3Oq8Anko3RPI+urFyAKrqaroAPb6v4xC6R5Xvru5R4kPo7qX8G3A93f0Equos4Ay687CC7mL/R6mq24Fn092wvoHuXPwt3fmZiWOBk/phq5dNXllVn6e7IX0a3X2qfwYeUlV30x3f8+iO/WPAnzX+Pk04je48/grYh+78TTgc+G/AL4HH0wXuhH2BC5LcQfeB5a1Vdc0Mj1G99Dd8pCn1n/hOqaod729baRSSnEj3MMIx97etRsMehSSpyaCQJDU59CRJarJHIUlqmpdfoLbtttvWkiVLxl2GJG0wVqxYcXNVbTfVunkZFEuWLGH58uXjLkOSNhhJrp1unUNPkqQmg0KS1GRQSJKaDApJUpNBIUlqMigkSU0GhSSpyaCQJDUZFJKkpnn5m9krV69hyVFnjrsMSZo1q447eGT7tkchSWoyKCRJTQaFJKnJoJAkNRkUkqQmg0KS1GRQSJKaDApJUpNBIUlqGnlQJFmS5KokJyb5cZJTkzwryXeT/CTJU/rX95P8MMn3kuzRv/c1Sb6U5Jv9th8edb2SpD80Wz2K3YG/Ax7Tv14B7A+8A3gXcBVwQFU9GXgv8D8G3rsXcCiwJ3Bokp2maiDJEUmWJ1m+9q41IzsQSdrYzNZ3Pf2sqlYCJLkcOLuqKslKYAmwCDgpyaOAAjYdeO/ZVbWmf+8VwC7AdZMbqKplwDKAzbZ/VI3wWCRpozJbPYrfDkzfMzB/D11YfQA4p6qeABwCPHCa965lnn6RoSTNVXPlZvYiYHU//Zox1iFJmmSuBMWHgf+Z5IfYY5CkOSVV8284f7PtH1Xbv/qj4y5DkmbN+v5/FElWVNXSqdbNlR6FJGmOMigkSU0GhSSpyaCQJDUZFJKkpnn5KOqeixexfD2fAJAkdexRSJKaDApJUpNBIUlqMigkSU0GhSSpyaCQJDUZFJKkJoNCktRkUEiSmgwKSVKTQSFJajIoJElNBoUkqcmgkCQ1GRSSpCaDQpLUZFBIkpoMCklSk0EhSWoyKCRJTQaFJKlp4bgLGIWVq9ew5Kgzx12GpDlq1XEHj7uEDYo9CklSk0EhSWoyKCRJTQaFJKnJoJAkNRkUkqSmsQZFkmOTvGOcNUiS2uxRSJKaZj0okrw7yY+TfAfYo1+2V5IfJLk0yZeTbNMv37dfdkmSjyS5bLbrlaSN3awGRZJ9gJcDewHPB/btV50MHFlVTwRWAu/rl38GeGNV7QWsnc1aJUmd2e5RHAB8uaruqqrbgK8AWwBbV9V5/TYnAc9IsjWwVVV9v19+WmvHSY5IsjzJ8rV3rRlV/ZK00Zk39yiqallVLa2qpQs2XzTuciRp3pjtoDgfeFGSByXZCjgEuBO4JckB/TavAs6rqluB25M8tV/+8lmuVZLELH97bFVdnOQM4EfATcBF/apXA59IsjlwDfDafvnrgU8luQc4D3BMSZJm2ax/zXhVfQj40BSrnjbFssv7G9wkOQpYPsraJEn3Ndf/P4qDkxxNV+e1wGvGW44kbXzmdFBU1RnAGeOuQ5I2ZvPmqSdJ0mgYFJKkJoNCktQ0p+9R/LH2XLyI5f7n6ZI0FPYoJElNBoUkqcmgkCQ1GRSSpCaDQpLUZFBIkpoMCklSk0EhSWoyKCRJTQaFJKnJoJAkNRkUkqQmg0KS1GRQSJKaDApJUpNBIUlqMigkSU0GhSSpyaCQJDUZFJKkJoNCktS0cNwFjMLK1WtYctSZ4y5DmhdWHXfwuEvQmNmjkCQ1GRSSpKYZBUWStyTZemB+myRvHl1ZkqS5YqY9isOr6taJmaq6BTh8NCVJkuaSmQbFgiSZmEmyAHjAaEqSJM0lM33q6ZvAGUk+2c+/sV8mSZrnZhoURwJHAH/ez58FnDCSiiRJc8qMhp6q6p6q+kRVvYQuML5fVWtHVVSStyXZfFT7lyTN3Eyfejo3yYOTPARYAXwqyf8aYV1vAwwKSZoDZnoze1FV3Qa8GDi5qp4KHLQuDSV5ZZILk1yS5JNJFiT5eJLlSS5P8v5+u78CdgDOSXJOv92JSS5LsjLJ29elXUnS+pnpPYqFSbYHXga8e10bSfJY4FBgv6r6XZKPAYcB766qX/VPUZ2d5IlV9b+T/DXwzKq6Ock+wOKqekK/r62naeMIumExFjx4u3UtUZI0jZn2KP478C3gX6vqoiS7Aj9Zh3YOAvYBLkpyST+/K/CyJBcDPwQeDzxuivdeA+ya5PgkzwVum6qBqlpWVUuraumCzRetQ2mSpJYZ9Siq6vPA5wfmrwH+dB3aCXBSVR1974LkkXRPT+1bVbckORF44BRt35LkScBzgDfR9Wpetw5tS5LWw0xvZj86ydlJLuvnn5jkmHVo52zgJUke1r//IcDOwJ3AmiQPB543sP3twFb9ttsCm1TVF4FjgL3XoV1J0nqa6dDTp4Cjgd8BVNWlwMtn2khVXUF3kf92kkvpehK/pRtyugo4DfjuwFuWAd9Mcg6wGDi3H7I6pa9DkjRLZnoze/OqunDgWzwAfr8uDVXVGcAZkxb/YJptjweOH1hkL0KSxmSmPYqbk+wGFECSlwA/H1lVkqQ5Y6Y9irfQDQc9Jslq4Gd0j7dKkua5ZlD0v88w4evAOXS9kDvpnnr6+9GVJkmaC+6vR7FV/+cewL7A/6V71PVVwIUjrEuSNEc0g6KqJr5W43xg76q6vZ8/Fjhz5NVJksZupvcoHg7cPTB/d79sTtpz8SKWH3fwuMuQpHlhpkFxMnBhki/38y8CThxJRZKkOWWmX+HxoSTfAA7oF722qn44urIkSXPFTHsUVNXFwMUjrEWSNAfN9BfuJEkbKYNCktRkUEiSmgwKSVKTQSFJajIoJElNBoUkqcmgkCQ1GRSSpCaDQpLUZFBIkpoMCklSk0EhSWoyKCRJTQaFJKnJoJAkNRkUkqQmg0KS1DTj/wp1Q7Jy9RqWHHXmuMuQ5oVVxx087hI0ZvYoJElNBoUkqcmgkCQ1GRSSpCaDQpLUZFBIkppGFhRJliS5bIrlJyR5XD/9rlG1L0kajlnvUVTVG6rqin7WoJCkOW7UQbEwyalJrkzyhSSbJzk3ydIkxwEPSnJJklMBkrwyyYX9sk8mWdC/TkxyWZKVSd4+4polSQNGHRR7AB+rqscCtwFvnlhRVUcBv66qvarqsCSPBQ4F9quqvYC1wGHAXsDiqnpCVe0JfGbENUuSBow6KK6rqu/206cA+ze2PQjYB7goySX9/K7ANcCuSY5P8ly6wLmPJEckWZ5k+dq71gzvCCRpIzfq73qq+5kfFOCkqjr6PiuSJwHPAd4EvAx43X0aqloGLAPYbPtHtdqRJK2DUfcodk7y9H76FcB3Jq3/XZJN++mzgZckeRhAkock2SXJtsAmVfVF4Bhg7xHXLEkaMOqguBp4S5IrgW2Aj09avwy4NMmp/ZNQxwDfTnIpcBawPbAYOLcfjjoFuE+PQ5I0OiMbeqqqVcBjplh14MA2RwJHDsyfAZwxxXvsRUjSmPib2ZKkJoNCktRkUEiSmgwKSVKTQSFJahr1L9yNxZ6LF7Hc/xBekobCHoUkqcmgkCQ1GRSSpCaDQpLUZFBIkpoMCklSk0EhSWoyKCRJTQaFJKnJoJAkNRkUkqQmg0KS1GRQSJKaDApJUpNBIUlqMigkSU0GhSSpyaCQJDUZFJKkJoNCktRkUEiSmhaOu4BRWLl6DUuOOnPcZWgjtOq4g8ddgjR09igkSU0GhSSpyaCQJDUZFJKkJoNCktRkUEiSmgwKSVLTWIIiyV8luTLJLUmOamz3miT/MJu1SZL+0Lh+4e7NwLOq6voxtS9JmqFZ71Ek+QSwK/CNJG+f6DEkeWmSy5L8KMn5A2/ZIck3k/wkyYdnu15J2tjNelBU1ZuAG4BnArcMrHov8JyqehLwwoHlewGHAnsChybZaar9JjkiyfIky9fetWY0xUvSRmgu3cz+LnBiksOBBQPLz66qNVX1G+AKYJep3lxVy6pqaVUtXbD5olkoV5I2DnMmKPqexjHATsCKJA/tV/12YLO1zNMvMpSkuWrOXHST7FZVFwAXJHkeXWBIksZszvQogI8kWZnkMuB7wI/GXZAkaUw9iqpa0k+e2L+oqhdPsem96/ttXjDSwiRJ9zGXehSSpDnIoJAkNRkUkqQmg0KS1GRQSJKa5szvUQzTnosXsfy4g8ddhiTNC/YoJElNBoUkqcmgkCQ1GRSSpCaDQpLUZFBIkpoMCklSk0EhSWoyKCRJTQaFJKkpVTXuGoYuye3A1eOuY8i2BW4edxFD5jFtGDymDcP6HtMuVbXdVCvm5Xc9AVdX1dJxFzFMSZZ7THOfx7Rh8JjWjUNPkqQmg0KS1DRfg2LZuAsYAY9pw+AxbRg8pnUwL29mS5KGZ772KCRJQ2JQSJKa5lVQJHlukquT/DTJUeOuZxiS/FOSm5JcNu5ahiHJTknOSXJFksuTvHXcNQ1DkgcmuTDJj/rjev+4axqGJAuS/DDJ18Zdy7AkWZVkZZJLkiwfdz3DkGTrJF9IclWSK5M8faj7ny/3KJIsAH4M/GfgeuAi4L9W1RVjLWw9JXkGcAdwclU9Ydz1rK8k2wPbV9XFSbYCVgAvmgc/pwBbVNUdSTYFvgO8tap+MObS1kuSvwaWAg+uqheMu55hSLIKWFpV8+YX7pKcBPxLVZ2Q5AHA5lV167D2P596FE8BflpV11TV3cDpwH8Zc03rrarOB3417jqGpap+XlUX99O3A1cCi8db1fqrzh397Kb9a4P+FJZkR+Bg4IRx16LpJVkEPAP4NEBV3T3MkID5FRSLgesG5q9nHlyA5rMkS4AnAxeMt5Lh6IdpLgFuAs6qqg39uD4KvBO4Z9yFDFkB306yIskR4y5mCB4J/AL4TD9MeEKSLYbZwHwKCm1AkmwJfBF4W1XdNu56hqGq1lbVXsCOwFOSbLBDhUleANxUVSvGXcsI7F9VewPPA97SD+9uyBYCewMfr6onA3cCQ71HO5+CYjWw08D8jv0yzTH9GP4XgVOr6kvjrmfY+m7/OcBzx13LetgPeGE/nn868CdJThlvScNRVav7P28Cvkw3bL0hux64fqAH+wW64Bia+RQUFwGPSvLI/mbOy4GvjLkmTdLf9P00cGVV/f246xmWJNsl2bqffhDdQxVXjbeqP15VHV1VO1bVErp/S/+vql455rLWW5It+oco6Idnng1s0E8UVtWNwHVJ9ugXHQQM9eGQefPtsVX1+yR/AXwLWAD8U1VdPuay1luSzwIHAtsmuR54X1V9erxVrZf9gFcBK/vxfIB3VdXXx1jTMGwPnNQ/fbcJ8LmqmjePlM4jDwe+3H1eYSFwWlV9c7wlDcVfAqf2H5KvAV47zJ3Pm8djJUmjMZ+GniRJI2BQSJKaDApJUpNBIUlqMigkSU0GhSSpyaCQJDX9f0+STy5PVZIIAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "import pandas as pd \n",
        "\n",
        "def get_top_n_words(corpus, n=None):\n",
        "    vec = CountVectorizer(stop_words='english').fit(corpus)\n",
        "    bag_of_words = vec.transform(corpus)\n",
        "    sum_words = bag_of_words.sum(axis=0) \n",
        "    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\n",
        "    words_freq = sorted(words_freq, key = lambda x: x[1], reverse=True)\n",
        "    return words_freq[:n]\n",
        "\n",
        "document_corpus = [\n",
        "  \"Dog bites man\",\n",
        "  \"Dog bites man after man eats fish\", \n",
        "  \"Dog bites fish\",\n",
        "  \"Man bites dog\", \n",
        "  \"Dog eats meat\", \n",
        "  \"Man eats food\",\n",
        "  \"Man eats fish\"\n",
        "]\n",
        "\n",
        "common_words = get_top_n_words(document_corpus, 5) # or use df['desc'] \n",
        "\n",
        "df2 = pd.DataFrame(common_words, columns = ['desc' , 'count'])\n",
        "df2.groupby('desc').sum()['count'].sort_values().plot(kind='barh', title='Top 5 words in document corpus')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rOkG0Ir6y7xr"
      },
      "source": [
        "## Get all bigrams"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "UrbRLvHfy7xs",
        "outputId": "4ee8a008-a839-4bc7-87b2-63fada8836a3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 299
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7fbae2086950>"
            ]
          },
          "metadata": {},
          "execution_count": 27
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaMAAAEICAYAAADyTpvZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZhcVZ3G8e9LEsISSMAghoA0S2QNhAAR1Gh4AMUgwkjGMAISURFBQWZAEBABUbbBQXFGDA4EZBcRUTaRVfZ0NkIgIGAQAoogWSAMS/KbP+5puCmququX6tOdfj/PU0/f5dyz3Hvr/uqce6taEYGZmVlOK+WugJmZmYORmZll52BkZmbZORiZmVl2DkZmZpadg5GZmWXnYLSCkTRJ0j2trL9J0kHdWafOkjRW0uO569EXSNpM0kxJiyUdkbs+3UnS+ZK+m7sevY2kJkkhqX9n8unUxgaSXi3Nrga8ASxN81+LiMu6oIwpwBeAN0uLB0fE0upb1BYRn+5sfbpbRPwJ2Cx3PfqIbwN3RMQoAEnzgK9ExB+z1qobRMSh3V2mpDuBSyPiF91ddk/jnlEnRcSglhfwV2Cv0rJOB6KSs8pldSQQdYYKffp86ewnv+7Ot4M2BOZ0RUb1nDM9rO3WhkYerz59cWkkSQMlnSvp+fQ6V9LAtG6cpOckHS/pJUnzJO3ftcXrp5IWSporadfSijslfSVN95N0TqrDXyR9o9zdTml/IOleYAmwsaQvSXosDeM8Lelrpbxb2vVtSS9KekHSPpLGS3pC0j8lHV9KP0ZSs6RFkv4u6Uc1GjNO0nOl+XmSjpb0cGrjVZJWqbHtSpJOlPRMqtMlkgZXy7eU925p+mRJ10i6VNIiYFKV/FdN+/CZVJd7JK2a1n1W0hxJC9K+3KKinGMlPQy8Jql/WvYdSY9KekXSRS3tqjb8mo7Vpml6fNpusaT5ko6usT82kXS7pJfTcb9M0pC07nZgF+Cnkl6VdAXwQeB3af7bKd1Oku5L7ZolaVwp//ecM1XqUK3tbeV5Wlr/qqTfSXpfqvsiSVMlNZXSfyQtW5j+fiQtnyipuaIuR0m6Pk1PkXRamm45l/+jdC5/qbTd+1I9Wso/rfL4lNKuks6hl1P7pkpaV9IPgLGl/f3T1upf2henS3oolf1bSWvXKPcuSfum6Y+m82XPNL+rpJlpurX3SMsQ3Jcl/RW4XcV14z/T+fM0sGdFuZNUXBsWq7iu1Hdtiwi/uugFzAN2S9OnAg8A7wfWAe4Dvp/WjQPeBn4EDAQ+AbwGbFYj3ynAP9NrGrBvK3WYlPI+ChgATAQWAmun9XdSDLsAHAo8CqwPrAX8EQigfyntX4GtKIZ0B6QTbxNAqd5LgNEV7Toppf0q8A/gcmCNlM/rwEYp/f3AgWl6ELBTjTaNA56r2M8PAesBawOPAYfW2PZg4EmKi+Ig4Frgl9XyrXIMTwbeAvah+OC2apX8/zvtp+FAP+Aj6Zh+KB3T3dO++Haqx8qlcmYCG7Tkm5Y9kpatDdwLnFY6rvdUlB3Apmn6BWBsml6r5ZhUqe+mqU4DKc7Lu4FzS+vfOT8q90eaHw68DIxP+2T3NL9OrXOmxvvknbbXmeeTFOfdYIpz9glgt1TGJcBFKe3awCvAgWndv6X591EMoy8GRpTqMhXYr/Q+a9nf4yjO5VPT8RtPca6vldZfmV6rAVsCz1Yen1IZXwN+l9L2A7YH1qyxv2vWv5R+PrA1sDrwa4phvmrlngqcl6aPB54Cziyt+3Ed75EmivPsklTeqhTXjbm8e57ekdL0T2kWka5lwDBgq7qun919wV6RXyx/IXsKGF9a9ylgXsWJvnpp/dXAd2vkOzq9mfqnN8Vi4KM10k4CngdUWvYQ71703zn5gdsp7mu1pNuN9wajU9to83XAkaV2vQ70S/NrpPw+XEo/DdgnTd8NnAIMbaOMcbw3GB1Qmj8LOL/GtrcBh5XmN6MIMP0r861yDE8G7m6lXiul9m5bZd13gasr0s4HxpXKObhK2YeW5scDT5WOa2vB6K8UF70123nO7gPMKM2/c35U7o80fyzpQlVadgtwUDvOmeXaXmeeJ5TWnQPcVJrfC5iZpg8EHqrI635gUpq+FDgpTY+geC+tluansHwwep30XkjLXgR2oggob1H68AicVnl8SusOpvgwuk2VdZX7u6363wmcUVq3JcW95H5V8t4VeDhN3wx8BXggzd8FfK6O90hTOs82Lq2/neXP00+yfDBaAOxLlQ9vrb08TNc46wHPlOafSctavBIRr7Wy/h0RMT0iXo6ItyPiRuAy4HOtlD0/0lnSRt7rUXyia/FslTTLLZP0aUkPqBhyW0BxwRxaSvJyvHs/6/X09++l9a9TfPoC+DJFD2JuGo74TCttqvS30vSSUp6Vqh2H/sC6dZZTbZ+0GAqsQvHBo9VyI2JZymt4G3mXl9U8J6rYl+JYPJOGZ3auligND12ZhvIWUVych1ZLW8OGwL+m4aYF6Rz4GMUn4GptqKWcpp48K8+hWudU5fEmzbfs98spehtQPBR0XUQsqVHHlyPi7dJ8y3m2DsU51NZ7p8UvKYLrlSqG7M+SNKBG2rbqX1nWMxQ9t2rH8H7gQ5LWBUZR9G42kDQUGEPxYbBamdXeI+UyK68b5fP8NYrRmEOBFyTdIGnzKnV7Dwejxnme4k3W4oNpWYu1JK3eyvrWBMUwWS3DJZXX18r7BYohuhYb1CgLKO6DUQwL/CewbkQMAW5soy41RcSfI+LfKIYyzwSuqdgnXaHacXib4mL2GsXQCVDcQ6O40CxXzVbyfgn4P4rho1bLTcdjA4reUWt5l49B+bhV1vUDy1UyYmpE7E2xL6+j6GlX88NU7siIWBM4gNaPX2Udn6XoxQwpvVaPiDPaaFdr+daTZ70qjzcU+7Flv98KrCNpFEVQurwDZfyD4hxq670DQES8FRGnRMSWFMO4nwG+2LK6nfWvLOuDFL2Yl6qUu4RiJOJI4JGIeJOih/bvFD3ulm1ae4+8k11p+oUqdSiXe0tE7E7xYWIucEFl3apxMGqcK4ATJa2TPomcRPEptOwUSStLGktxgv6qWkaSJkgalG40fpLiAnJ9K2W/HzhC0gBJ/wpsQRE0Kl0NHClpuIqb2Me20aaVKe41/AN4W9KnKbroHSLpAEnrpF7DgrR4WUfzq+EK4ChJG0kaRHExvip94n0CWEXSnumT6okU7atLqveFwI8krZdu7O6cgvbVwJ7pRvEA4D8oHvu/r41sD5e0fropfQJwVVo+C9hK0igVDzWc3LJBOof2lzQ4It6iGLOvtR/XAF4FFkoaDhzTRn3+zvIPIVwK7CXpU6m9q6i42b9+je3r0ZV53kjRG/iCigcjJlIMZf0eisBA8T47m+J+x63tLSD1/K8FTpa0Wvrk/8Va6SXtImlk+rCziCJ4tByfyv3bav2TAyRtKWk1ins/10Ttp2vvAr6R/kIxzFeeh9bfI9VcTXF9WV/SWsBxpbauK2nv9KHyDYpzra73tINR45wGNAMPA7OB6WlZi79R3Jh8nmLY7dCImFsjryMpPhktoHgTfTUi7myl7AcpxsNfAn4ATIiIl6ukuwD4Q6rjDIo3wtu8+z2p5UTEYuAIipPxFYphjtaCYlv2AOao+K7WjyluJL/exjbtdSHFMMndwF8oejLfBIiIhcBhwC8o9u9rwHPVs6npaIrjO5XiAZMzgZUi4nGKDw3nURyHvSge+3+zVkbJ5RTH5GmK4b/TUl2foLjw/BH4M1D55NaBwLw09HYoUOsJplMo7kEuBG6guKi25nSKD1ULJB0dEc8Ce1PcEP8HRa/mGDpxLenKPNN5/hmK4P8yxYMjnyn1AqDYx7sBv2rlgtuWb1A8TPE3ivPrCoqLbzUfAK6hCESPUQSCX6Z1PwYmqHh68id11v+XFPe3/kYxTNzal5PvovgAcneNeWjlPVLDBRTDjrMormvlc2glip7X8xTvh08AX28lr3do+VsL1h1UPLZ6aUR05tNkl0s9nfMjonKYwLqB+tAXTFc0ks4EPhARBzW4nDtZQb8k655RH6biOzLj03DAcOB7wG9y18usp5O0uaRtVBhD8TCO3zud4GDUt4li2OYVimG6xyjubZlZ69agGJ56jeK+3jnAb7PWqJfzMJ2ZmWXnnpGZmWXnHynsoKFDh0ZTU1PuapiZ9SrTpk17KSIqv8/nYNRRTU1NNDc3t53QzMzeIanyFyYAD9OZmVkP4GBkZmbZORiZmVl2DkZmZpadg5GZmWXnYGRmZtk5GJmZWXYORmZmlp2DkZmZZedfYOig2fMX0nTcDbmrYWbWreadsWdD8nXPyMzMsnMwMjOz7ByMzMwsOwcjMzPLzsHIzMyyczAyM7PsHIzMzCy7PhuMJO0jacvc9TAzsz4cjIB9AAcjM7MeoOHBSFKTpLmSpkh6QtJlknaTdK+kP0sak9KNkXS/pBmS7pO0WVo+SdK1km5O6c+qUc72ku6SNE3SLZKGpeVflTRV0ixJv5a0mqSPAJ8FzpY0U9Imko6Q9KikhyVd2ej9YmZm7+quntGmwDnA5un1BeBjwNHA8SnNXGBsRGwHnAT8sLT9KGAiMBKYKGmDcuaSBgDnARMiYnvgQuAHafW1EbFjRGwLPAZ8OSLuA64HjomIURHxFHAcsF1EbAMcWq0Rkg6R1CypeemShZ3YHWZmVtZdv033l4iYDSBpDnBbRISk2UBTSjMYuFjSCCCAAaXtb4uIhWn7R4ENgWdL6zcDtgZulQTQD3ghrdta0mnAEGAQcEuNOj4MXCbpOuC6agkiYjIwGWDgsBFRX9PNzKwt3RWM3ihNLyvNLyvV4fvAHRHxL5KagDtrbL+U99ZbwJyI2LlK2VOAfSJilqRJwLgaddwT+DiwF3CCpJER8XbNFpmZWZfpSQ8wDAbmp+lJ7dz2cWAdSTtDMWwnaau0bg3ghTSUt39pm8VpHZJWAjaIiDuAY1NdBnWkEWZm1n49KRidBZwuaQbt7LFFxJvABOBMSbOAmcBH0urvAg8C91Lcl2pxJXBMKm8EcGkaNpwB/CQiFnSmMWZmVj9F+NZHRwwcNiKGHXRu7mqYmXWrzv4/I0nTImKHyuU9qWdkZmZ9lIORmZll52BkZmbZORiZmVl2DkZmZpZdd33pdYUzcvhgmjv5VImZmRXcMzIzs+wcjMzMLDsHIzMzy87ByMzMsnMwMjOz7ByMzMwsOwcjMzPLzsHIzMyyczAyM7PsHIzMzCw7ByMzM8vOwcjMzLJzMDIzs+wcjMzMLDsHIzMzy87ByMzMsnMwMjOz7ByMzMwsOwcjMzPLzsHIzMyyczAyM7Ps+ueuQG81e/5Cmo67IXc1zMy61bwz9mxIvu4ZmZlZdg5GZmaWnYORmZll52BkZmbZORiZmVl2DkZmZpZdlmAk6WRJR3dBPuMk/b7GuhslDUmvwzpblpmZNc4K2zOKiPERsQAYAjgYmZn1YN0WjCSdIOkJSfcAm5WWj5L0gKSHJf1G0lpp+Y5p2UxJZ0t6pEbWa0q6QdLjks6XtFLafp6kocAZwCYt+aR1x0iamvI/JS1bPeUzS9IjkiY2cn+Ymdm7uiUYSdoe2A8YBYwHdiytvgQ4NiK2AWYD30vLLwK+FhGjgKWtZD8G+CawJbAJ8LmK9ccBT0XEqIg4RtIngRFpu1HA9pI+DuwBPB8R20bE1sDNHW6wmZm1S3f1jMYCv4mIJRGxCLgeQNJgYEhE3JXSXQx8XNIQYI2IuD8tv7yVvB+KiKcjYilwBfCxNuryyfSaAUwHNqcITrOB3SWdKWlsRCys3FDSIZKaJTUvXfKe1WZm1kErwm/TRRvzlQScHhE/f88KaTRFz+00SbdFxKnLZRwxGZgMMHDYiLbKMTOzOnVXz+huYB9Jq0paA9gLIPU+XpE0NqU7ELgrPXiwWNKH0/L9Wsl7jKSN0r2iicA9FesXA2uU5m8BDpY0CEDScEnvl7QesCQiLgXOBkZ3uLVmZtYu3dIziojpkq4CZgEvAlNLqw8Czpe0GvA08KW0/MvABZKWAXcBtcbFpgI/BTYF7gB+U1H2y5LuTQ9A3JTuG20B3C8J4FXggLT92am8t4Cvd7LZZmZWJ0X0zNEmSYMi4tU0fRwwLCKOzFytdwwcNiKGHXRu7mqYmXWrzv4LCUnTImKHyuU9+Z7RnpK+Q1HHZ4BJeatjZmaN0mODUURcBVyVux5mZtZ4K+wvMJiZWe/hYGRmZtk5GJmZWXY99p5RTzdy+GCaO/lUiZmZFdwzMjOz7ByMzMwsOwcjMzPLzsHIzMyyczAyM7PsHIzMzCw7ByMzM8vOwcjMzLJzMDIzs+wcjMzMLDsHIzMzy87ByMzMsnMwMjOz7ByMzMwsOwcjMzPLzsHIzMyyczAyM7PsHIzMzCw7ByMzM8vOwcjMzLJzMDIzs+z6565AbzV7/kKajrshdzXMVgjzztgzdxUsM/eMzMwsOwcjMzPLrq5gJOlwSUNK82tJOqxx1TIzs76k3p7RVyNiQctMRLwCfLUxVTIzs76m3mDUT5JaZiT1A1ZuTJXMzKyvqfdpupuBqyT9PM1/LS0zMzPrtHqD0bHAIcDX0/ytwC8aUiMzM+tz6hqmi4hlEXF+REygCEr3R8TSxlbtXZK+JWm1dqRfR9KDkmZIGivpxvIDGFXSz5M0tGtqa2Zm7VXv03R3SlpT0trANOACSf/V2Kot51tA3cEI2BWYHRHbRcSfImJ8+QEMMzPrWep9gGFwRCwCPgdcEhEfprjgd4ikAyQ9JGmmpJ+nByKQ9DNJzZLmSDolLTsCWA+4Q9IdkvpJmiLpEUmzJR1Vkfco4Cxg75T/qi09H0mrS7pB0qy0/cTSpt+UND3luXlH22ZmZu1XbzDqL2kY8Hng950pUNIWwETgoxExClgK7J9WnxAROwDbAJ+QtE1E/AR4HtglInYBRgHDI2LriBgJXFTOPyJmAicBV0XEqIh4vbR6D+D5iNg2IrZm+YcwXoqI0cDPgKNr1P2QFCybly5Z2JndYGZmJfUGo1OBW4CnImKqpI2BP3ewzF2B7YGpkmam+Y3Tus9Lmg7MALYCtqyy/dPAxpLOk7QHsKgdZc8Gdpd0pqSxEVGOKNemv9OApmobR8TkiNghInbot9rgdhRrZmatqetpuoj4FfCr0vzTwL4dLFPAxRHxneUWShtR9Eh2jIhXJE0BVqlSl1ckbQt8CjiUord2cJ3teELSaGA8cJqk2yLi1LT6jfR3Kf4BWTOzblXvAwwfknSbpEfS/DaSTuxgmbcBEyS9P+W1tqQNgTWB14CFktYFPl3aZjGwRko/FFgpIn4NnAiMrrdgSesBSyLiUuDs9mxrZmaNU28P4ALgGODnABHxsKTLgdPaW2BEPJoC2R8krQS8BRweEQ9ImgHMBZ4F7i1tNhm4WdLzFE/WXZS2BViuh9WGkcDZkpalcr/eRnozM+sGioi2E0lTI2JHSTMiYru0bGZ6AKFPGjhsRAw76Nzc1TBbIfj/GfUdkqalB9WWU+8DDC9J2gSIlNkE4IUurJ+ZmfVh9Q7THU4xVLa5pPnAX3j3cWwzM7NOaTUYSfr30uyNwB0UvanXKJ6m+1HjqmZmZn1FWz2jNdLfzYAdgd9SPJp9IPBQA+tlZmZ9SKvBKCJafpLnbmB0RCxO8ycDNzS8dmZm1ifUe89oXeDN0vybaVmfNXL4YJr9BJCZWZeoNxhdAjwk6Tdpfh9gSkNqZGZmfU69Pwf0A0k3AWPToi9FxIzGVcvMzPqSun+DLSKmA9MbWBczM+uj6v3Sq5mZWcM4GJmZWXYORmZmlp2DkZmZZedgZGZm2TkYmZlZdg5GZmaWnYORmZll52BkZmbZORiZmVl2DkZmZpadg5GZmWXnYGRmZtk5GJmZWXYORmZmlp2DkZmZZedgZGZm2TkYmZlZdnX/23Fb3uz5C2k67obc1TBbIcw7Y8/cVbDM3DMyM7PsHIzMzCw7ByMzM8vOwcjMzLJzMDIzs+wcjMzMLLuGBCNJTZIeqbHuF5K2TNPHN6J8MzPrXbq9ZxQRX4mIR9Osg5GZmTU0GPWXdJmkxyRdI2k1AEl3StpB0hnAqpJmSrosrTtA0kNp2c8l9UuvKZIekTRb0lGVBaX1P5P0gKSnJY2TdGEqe0op3c8kNUuaI+mU0vJ5kk6RND2VsXkD94uZmVVoZDDaDPifiNgCWAQcVl4ZEccBr0fEqIjYX9IWwETgoxExClgK7A+MAoZHxNYRMRK4qEZ5awE7A0cB1wP/BWwFjJQ0KqU5ISJ2ALYBPiFpm9L2L0XEaOBnwNGdbbyZmdWvkcHo2Yi4N01fCnysjfS7AtsDUyXNTPMbA08DG0s6T9IeFIGtmt9FRACzgb9HxOyIWAbMAZpSms9Lmg7MoAhUW5a2vzb9nVZKvxxJh6SeVfPSJQvbaI6ZmdWrkb9NF23MVxJwcUR85z0rpG2BTwGHAp8HDq6y/Rvp77LSdMt8f0kbUfR4doyIV9Lw3SpVtl9Kjf0SEZOByQADh41oqz1mZlanRvaMPihp5zT9BeCeKmnekjQgTd8GTJD0fgBJa0vaUNJQYKWI+DVwIjC6g/VZE3gNWChpXeDTHczHzMy6WCN7Ro8Dh0u6EHiU4l5MpcnAw5Kmp/tGJwJ/kLQS8BZwOPA6cFFaBvCenlM9ImKWpBnAXOBZ4N42NjEzs26i4jaLtdfAYSNi2EHn5q6G2QrB/0Ki75A0LT1Ithz/AoOZmWXnYGRmZtk5GJmZWXYORmZmlp2DkZmZZdfIR7tXaCOHD6bZTwCZmXUJ94zMzCw7ByMzM8vOwcjMzLJzMDIzs+wcjMzMLDsHIzMzy87ByMzMsnMwMjOz7ByMzMwsOwcjMzPLzsHIzMyyczAyM7PsHIzMzCw7ByMzM8vOwcjMzLJzMDIzs+wcjMzMLDsHIzMzy87ByMzMsnMwMjOz7ByMzMwsu/65K9BbzZ6/kKbjbshdDWvDvDP2zF0FM6uDe0ZmZpadg5GZmWXnYGRmZtk5GJmZWXYORmZmlp2DkZmZZedgZGZm2TkYmZlZdg5GZmaWXUODkaQmSXMlTZH0hKTLJO0m6V5Jf5Y0JqUbI+l+STMk3Sdps7R8kqRrJd2c0p9Vo5x5kk6XNFNSs6TRkm6R9JSkQ1OaQZJukzRd0mxJe5fq+JikCyTNkfQHSas2cr+YmdnyuqNntClwDrB5en0B+BhwNHB8SjMXGBsR2wEnAT8sbT8KmAiMBCZK2qBGOX+NiFHAn4ApwARgJ+CUtP7/gH+JiNHALsA5kpTWjQD+OyK2AhYA+1YrQNIhKdg1L12ysP49YGZmreqO36b7S0TMBpA0B7gtIkLSbKAppRkMXCxpBBDAgNL2t0XEwrT9o8CGwLNVyrk+/Z0NDIqIxcBiSW9IGgK8BvxQ0seBZcBwYN1SHWem6Wmlei0nIiYDkwEGDhsR9e8CMzNrTXf0jN4oTS8rzS/j3WD4feCOiNga2AtYpcb2S6kdQMv5VpbZH9gfWAfYPvWg/l4qp94yzMysAXrKAwyDgflpelIDy3gxIt6StAtFD8vMzHqAnhKMzgJOlzSDxvVKLgN2SMODX6S4T2VmZj2AInzroyMGDhsRww46N3c1rA3+f0ZmPYukaRGxQ+XyntIzMjOzPszByMzMsnMwMjOz7ByMzMwsOwcjMzPLzl/u7KCRwwfT7Ce1zMy6hHtGZmaWnYORmZll52BkZmbZORiZmVl2DkZmZpadg5GZmWXnYGRmZtk5GJmZWXYORmZmlp2DkZmZZed/rtdBkhYDj+euRxcbCryUuxJdzG3qHVbENsGK2a7OtmnDiFincqF/m67jHq/23wp7M0nNblPP5zb1HitiuxrVJg/TmZlZdg5GZmaWnYNRx03OXYEGcJt6B7ep91gR29WQNvkBBjMzy849IzMzy87ByMzMsnMwaoOkPSQ9LulJScdVWT9Q0lVp/YOSmrq/lu1TR5smSfqHpJnp9ZUc9ayXpAslvSjpkRrrJeknqb0PSxrd3XVsrzraNE7SwtIxOqm769hekjaQdIekRyXNkXRklTS96ljV2abeeKxWkfSQpFmpXadUSdO1176I8KvGC+gHPAVsDKwMzAK2rEhzGHB+mt4PuCp3vbugTZOAn+auazva9HFgNPBIjfXjgZsAATsBD+aucxe0aRzw+9z1bGebhgGj0/QawBNVzr1edazqbFNvPFYCBqXpAcCDwE4Vabr02ueeUevGAE9GxNMR8SZwJbB3RZq9gYvT9DXArpLUjXVsr3ra1KtExN3AP1tJsjdwSRQeAIZIGtY9teuYOtrU60TECxExPU0vBh4Dhlck61XHqs429Tpp/7+aZgekV+XTbl167XMwat1w4NnS/HO890R7J01EvA0sBN7XLbXrmHraBLBvGia5RtIG3VO1hqm3zb3NzmkY5SZJW+WuTHukIZ3tKD5xl/XaY9VKm6AXHitJ/STNBF4Ebo2ImseqK659DkZWze+ApojYBriVdz/9WM8xneI3vrYFzgOuy1yfukkaBPwa+FZELMpdn67QRpt65bGKiKURMQpYHxgjaetGludg1Lr5QLlXsH5aVjWNpP7AYODlbqldx7TZpoh4OSLeSLO/ALbvpro1Sj3HsVeJiEUtwygRcSMwQNLQzNVqk6QBFBftyyLi2ipJet2xaqtNvfVYtYiIBcAdwB4Vq7r02udg1LqpwAhJG0lameIm3fUVaa4HDkrTE4DbI93R66HabFPFGP1nKcbBe7PrgS+mJ7V2AhZGxAu5K9UZkj7QMj4vaQzFe7knfwgi1fd/gcci4kc1kvWqY1VPm3rpsVpH0pA0vSqwOzC3IlmXXvv8q92tiIi3JX0DuIXiKbQLI2KOpFOB5oi4nuJE/KWkJyluOO+Xr8Ztq7NNR0j6LPA2RZsmZatwHSRdQfHE0lBJzwHfo7jhSkScD9xI8ZTWk8AS4Et5alq/Oto0Afi6pLeB14H9eviHIICPAgcCs9O9CIDjgQ9Crz1W9bSpNx6rYcDFkvpRBM+rI+L3jbz2+eeAzMwsOw/TmZlZdg5GZmaWnYORmZll52BkZmbZORiZmVl2DkZmZpadg9HRL6IAAAAKSURBVJGZmWX3//gf5Y9JiqloAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "def get_top_n_bigram(corpus, n=None):\n",
        "    vec = CountVectorizer(ngram_range=(2, 2), stop_words='english').fit(corpus)\n",
        "    bag_of_words = vec.transform(corpus)\n",
        "    sum_words = bag_of_words.sum(axis=0) \n",
        "    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\n",
        "    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\n",
        "    return words_freq[:n]\n",
        "\n",
        "document_corpus = [\n",
        "  \"Dog bites man\",\n",
        "  \"Dog bites man after man eats fish\", \n",
        "  \"Dog bites fish\",\n",
        "  \"Man bites dog\", \n",
        "  \"Dog eats meat\", \n",
        "  \"Man eats food\",\n",
        "  \"Man eats fish\"\n",
        "]\n",
        "\n",
        "common_words = get_top_n_bigram(document_corpus, 5)\n",
        "\n",
        "df4 = pd.DataFrame(common_words, columns = ['desc' , 'count'])\n",
        "df4.groupby('desc').sum()['count'].sort_values().plot(kind='barh', title='Top 5 bigrams in our corpus after removing stop words')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dCuydfSny7xs"
      },
      "source": [
        "## Get all trigrams"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "-6fa50qfy7xs",
        "outputId": "b61b8188-56c8-469c-e015-f9268059fb38",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 299
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7fbae20fb150>"
            ]
          },
          "metadata": {},
          "execution_count": 26
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAb0AAAEICAYAAADLKSqCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3debxVZd338c9XUExBQDFvRJNUckQRQcvEqLzVMNPSJ7wdEjXn5juL0krNkvIprbvB1Ax9HKKcMmdvBcwBZZQDziBmOKUiQpQa/J4/1nVksdl7n304Z+99cH3fr9d+nbXWtdZ1/da111q/NZ1zFBGYmZkVwTrNDsDMzKxRnPTMzKwwnPTMzKwwnPTMzKwwnPTMzKwwnPTMzKwwnPSsLiRdJOk7zY6jPSS9T9JSSd2aHcu7naTNJN0raYmknzQ7nkaS9G1JlzY7jrWRpJC0bYfq8O/pdW2SluZGNwDeBJan8ZMi4qpOaGM8cATwVm5y74hYXmbeMcDnI2LvjrZrxZVOiHYDDo2IkDQJuDIinAzqIO3jf4uIM5sdS0dICmBQRDy9pnX4Sq+Li4ierR/gr8BBuWkdTng5P863VS7h1aqjV0pFv9KS1L1O9UpSV9nntwIejU46625rm+li625tqNc+AEBE+LOWfIAFwL5puAdwIfB8+lwI9EhlI4G/Ad8GXknLHVml3vHAuTW0vwPwL7IrzaXA67nlfw3cCvwD2Le0TuAbwAsp1s8DAWxbZfkDgZnAG8BzwFm5ugam5Y9NZYuAk4HhwGzgdeAXufm3BSYDi1N/TKiwfq31dk/jk4DvA/cDS4A7gX5V+ucE4GngNeAmYPNy9ebq/nwaHpPauAB4tdx3AXRL3+e8FMt0YMtUthcwNa3fVGCvknZ+kOr/Z+qLScB5wMOpf/8EbJzfdqpsd3sA09JyLwE/rdAXfYGbgb+n7+dmYIvc9/022Z2FpSm25WTb1tLW7w7YHrgr9ecTwGdLttlVtpkyMZRb97bq/BVwWy6u/yDbtxYBjwO7lewPk8i2t7nAp9L0PYEXgW65eT8NzE7DZ5Fd1cLKbeMYspPaV4Azcsu9B7g8tf8Y2X70twp9LrJt6OX0/bQAOwMnlvT3n6vFn+uLi1JfLSHbf7aq0O7lwH+n4QFpfU5L49ukvl6n2j6SygI4DXgKeCZNO52Vx43jWPW4MQp4NMW3EPh6TcfRehyc/anPh1UPPucAU4D3ApsCDwDfT2UjgX8DPyVLjh8hOzBsV6He8WkjfI3sYHpolRjGAPeVWX4x8GGyuwfrk0t6wAFkB4GdyG7RXsnqSa90+ZHA4DS+C9kB9pA0/8C0/EVp3v3IDpg3pv4YQLbjfyTNfw1wRq7uvSusW2u9+aQ3D/gA2cFnEjCuwrIfIztgDU19/j/AveXqzdWdT3r/Br4IdAfeU6b+08kOYtuRHdx2BTYBNiY7IB6dlv2vNL5Jrp2/pr7vDqybpi0kOyBuCFzHyoPwSKonvQeBo9NwT+CDFfpjE+DQ9H33Av4I3FiyzZxbrj/S+IZkJzTHprh3S/27Y6VtpkwMpeveu4Y6XwF2J9tO7gGeAT5HdtJxLjAxzbsu2cH728B66ftfQtrHyLab/8zF8kdgbBo+i9WT3iVk29iuZI8wdkjl48gSTl9gC7KTukpJb3+y/bcP2TayA9C/Qn+3Ff/4NL4P2fb8M0r2+1xdx7EykR6R1n1CruxPbe0jqTzIkuzGqS8OINvvW7fTq1n1uPECMCIN9wWG1nQcbdQB25+Of1j14DMPGJUr2x9YkIZHkh1EN8yV/wH4ToV6h5IdpLqTnT0tAT5cYd4xpRt/2kGuKDOtNeldBpyXK9uW1ZPeFZXWO81zIXBBGh6Ylh+QK38VGJ0bvw74Shq+AriYdKVRpY3WevNJ78xc+anA7RWW/S3ZLeLW8Z5kZ9cDS+vN1Z1Pen9tI7YngIPLTD8aeLhk2oPAmFw755SUTyKXvIEdya4CutF20rsXOJsqV7wV4h8CLCq3fZT2RxofDfylpI7fAN9rxzazyrrXWOclubIvAo/lxgez8u7GCLITuXVy5deQ7kiQJcjL0nAvspPOrdL4Waye9LbI1fMwcHgang/snyv7fOn3kyv7GPAk8MF8XBX6u634xwO/L9mel5PuLpTUvQ3ZidY6ZCeiJ7XGSHYV+LW29pE0HsDHcuWXsep2+gFWPW78NbW1UXu2Rd/jXnttDjybG382TWu1KCL+UaX8HRExIyJejYh/R8StwFXAZ9oZz3NtxJovLzfvKtMk7SlpoqS/S1pMdvuyX8kyL+WG/1lmvGca/gbZme/DkuZKOq5KrKVezA0vy9VZapXvIyKWkiXiATW2U63/ALYkO9Gp2m7ybEm7bfX3s2Rn/qX9W87xZAefxyVNlfTJcjNJ2kDSbyQ9K+kNsmTZpx3Pa7cC9pT0eusHOJLsdmO5dagkP08tdda6TW0OPBcRK3Ll+X6/GviMpB5k+9KMiCj9nvIqbWe17DsARMQ9wC+AXwIvS7pY0kYVZm8r/lXaStvza5Q5hkTEPLKkPoQsmd4MPC9pO7K7TJNzbba1j+TXr3TdS/vvULKT9GclTZb0oQrrugonvbXX82Q7cav3pWmt+krasEp5NUGWJCqVtWc6ZLchtsiNb1nD8leT3fPfMiJ6k51BVoqpqoh4MSJOiIjNyc4Mf9XR157LWOX7SH2/CdltxNaTjw1y8+cPtFC9/yDb+bdpq93kfandanXnv4P3kZ1xv5JifSfOlKQ2faeiiKci4r/IbiP/CLi2ZDtr9d9kt2L3jIiNyG6TQe3b1XPA5Ijok/v0jIhT2livavXWUmetnge2LHk55p1+j4hHyQ7SnyC75Xf1GrQBte0774iIn0fE7mRX7x8guy0Oq/dV1fhL25LUk+y2Y6VjyGTgMGC9iFiYxo8hu+04K9dmpX3knVXIDb/A6ttpfl2nRsTBZNvijWR3s9rkpLf2ugY4U9KmkvoB3yV7VpZ3tqT1JI0APkn2XGE1kg6T1FPSOpL2A44iSzjlvARsIWm9dsT6B+BYSTtI2gCo5ff3egGvRcS/JO1BduBYI5L+j6TWA8cish1rRZVF1sQ1ZOs4JJ3d/xB4KCIWRMTfyXbsoyR1S1ea5RJYNZcC35c0KL2JuIukTche5PiApCMkdZc0muyAd3Mb9R0lacf0fZwDXBvZG7tPAutLOlDSusCZZM9fAJB0lKRN0xXC62lyub7sRXZl9LqkjYHvtRHPS8DWufGb03odLWnd9BkuaYc26qmmM+t8iOyK7BupnpHAQcDvc/NcDXyZLOGX3fdq8AfgW5L6ShoAfKHSjGld9kzf2z/InnO3fjel/VtL/KMk7Z329e8DUyKi0pXm5BTbvWl8Uhq/L1a+CV5xH6my7mNy2+k721A6rh0pqXdEvE324k5N+7ST3trrXLK36GaTveAwI01r9SLZAf55stuVJ0fE4xXq+jLZQfl14HzghIiYVGHee8je9HpR0iu1BBoRtwE/ByaSPTyfkorerLLYqcA5kpaQJfSazuIqGA48lH7n8SbgyxExvwP1rSYi/pcsmV9Hdoa6DXB4bpYTyM66XyV7seKBdjbxU7I+uJNsB/8t2Qsvr5Kd0Px3qvsbwCcjoq3v5v+RPbd5keyljS+l9VhM1veXsvIq9W+55Q4A5qa+/BnZs6d/lqn/QrKXEV4h+75vbyOenwGHSVok6ecRsYTsBaXDybbhF8muLHtUqaOqzqwzIt4iSxKfIFvHXwGfK9nHriG7vXdPDd9HJeeQ9f8zwP8C11J5v9mI7IWYRWRXma+S7c+QbS87ptu6N9YY/9VkieY1spd7jqoS52SyE53WpHcf2R2D1vFa9pFVpOPGhWTHnKfTz7yjgQXp9vnJZLeq2+RfTn8XSmdtV0bEFm3N2wzpzHoO2a9Y/LvZ8RSN/Ivgay1Jp5CdaHykzu2M513wy+zl+ErPGkLSpyX1kNSX7Oz6z054ZtVJ6i/pw+nRw3ZkV/Q3NDuutZmTnjXKSWS/OzeP7NXnNXl5wKxo1iP7tYolZLf3/kR2K9LWkG9vmplZYfhKz8zMCqN+f9TTOqxfv34xcODAZodhZrZWmT59+isRsWm5Mie9LmzgwIFMmzat2WGYma1VJFX86ze+vWlmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhv8jShbUsXMzAsbc0Owwzs4ZaMO7AutXtKz0zMysMJz0zMysMJz0zMysMJz0zMysMJz0zMysMJz0zMysMJz0zMysMJ71E0iGSdmznMtdImi3pq5LOkbRvlXnHSzqs45Gamdma8i+nr3QIcDPwaC0zS/oPYHhEbFvXqMzMrNM0/EpP0kBJj6crnyclXSVpX0n3S3pK0h5pvj0kPShppqQHJG2Xpo+RdL2k29P8P67Qzu6SJkuaLukOSf3T9BMkTZX0iKTrJG0gaS/gU8D5kmZJ2kbSlyQ9mq7kfl+miTuBAWn+EfkrOUnjcsv+39wy+6R1me+rPjOzxmvW7c1tgZ8A26fPEcDewNeBb6d5HgdGRMRuwHeBH+aWHwKMBgYDoyVtma9c0rrA/wCHRcTuwGXAD1Lx9RExPCJ2BR4Djo+IB4CbgNMjYkhEzAPGArtFxC7AyWXW4VPAvDT/X3JtbwJ8GtgpLXtubpn+aT0/CYwr1zGSTpQ0TdK05csWl5vFzMzWULNubz4TES0AkuYCd0dESGoBBqZ5egOXSxoEBLBubvm7I2JxWv5RYCvguVz5dsDOwF2SALoBL6SynSWdC/QBegJ3VIhxNnCVpBuBG9uxbouBfwG/lXQz2S3TVjdGxArgUUmblVs4Ii4GLgbo0X9QtKNdMzNrQ7Ou9N7MDa/Ija9gZSL+PjAxInYGDgLWr7D8clZP3gLmpquwIRExOCL2S2XjgS9ExGDg7JJ68w4EfgkMBaZKqukEISL+DewBXEt2RXd7hbhVS31mZtZ5uvLbm72BhWl4TDuXfQLYVNKHILvdKWmnVNYLeCHdAj0yt8ySVIakdYAtI2Ii8M0US89aGpbUE+gdEbcCXwV2bWfsZmZWJ1056f0YOE/STNp5GzYi3gIOA34k6RFgFrBXKv4O8BBwP9lzw1a/B05P7Q0Crky3W2cCP4+I12tsvhdws6TZwH3A19oTu5mZ1Y8i/Nioq+rRf1D0P+bCZodhZtZQHf1/epKmR8SwcmVd+UrPzMysUznpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYfgPTndhgwf0ZloH32IyM7OVfKVnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF4aRnZmaF0b3ZAVhlLQsXM3DsLc0Ow8ysoRaMO7BudftKz8zMCsNJz8zMCsNJz8zMCsNJz8zMCsNJz8zMCsNJz8zMCqMpSU/SWZK+3gn1jJR0c4WyWyX1SZ9TO9qWmZmt/d61V3oRMSoiXgf6AE56ZmbWuKQn6QxJT0q6D9guN32IpCmSZku6QVLfNH14mjZL0vmS5lSoeiNJt0h6QtJFktZJyy+Q1A8YB2zTWk8qO13S1FT/2WnahqmeRyTNkTS6zDpMknSBpGmSHksxXi/pKUnn5ua7UdJ0SXMlnZibvlTSD1IbUyRt1vGeNTOzWjUk6UnaHTgcGAKMAobniq8AvhkRuwAtwPfS9N8BJ0XEEGB5ler3AL4I7AhsA3ympHwsMC8ihkTE6ZL2Awal5YYAu0vaBzgAeD4ido2InYHbK7T3VkQMAy4C/gScBuwMjJG0SZrnuIjYHRgGfCk3fUNgSkTsCtwLnFBlvczMrJM16kpvBHBDRCyLiDeAmwAk9Qb6RMTkNN/lwD6S+gC9IuLBNP3qKnU/HBHzI2I5cA2wdxux7Jc+M4EZwPZkSbAF+E9JP5I0IiIWV1j+pvSzBZgbES9ExJvAfGDLVPYlSY8AU9K0QWn6W0DrM8jpwMDSyiWdmK4kpy1fVikEMzNbE++Gv70ZbYyXEnBeRPxmtQJpKNmV6LmS7o6Ic8os/2b6uSI33DreXdJIYF/gQxGxTNIkYP00z9sR0Rrfcsr0f0RcDFwM0KP/oLbWxczM2qFRV3r3AodIeo+kXsBBAOlqapGkEWm+o4HJ6QWUJZL2TNMPr1L3HpLen57ljQbuKylfAvTKjd8BHCepJ4CkAZLeK2lzYFlEXAmcDwxdw3XtDSxKCW974INrWI+ZmXWyhlzpRcQMSROAR4CXgam54mOAiyRtQHaL8Ng0/XjgEkkrgMlApXt9U4FfANsCE4EbStp+VdL96UWY29JzvR2AByUBLAWOSsufn9p7GzhlDVf3duBkSY8BT5Dd4jQzsy5AK++2dS2SekbE0jQ8FugfEV9uclgN1aP/oOh/zIXNDsPMrKE6+q+FJE1PLxyupis/0ztQ0rfIYnwWGNPccMzMbG3XZZNeREwAJjQ7DjMze/d41/5FFjMzs1JOemZmVhhOemZmVhhd9pmeweABvZnWwbeYzMxsJV/pmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYTjpmZlZYXRvdgBWWcvCxQwce0uzw7CCWjDuwGaHYNbpfKVnZmaF4aRnZmaFUVPSk3SapD658b6STq1fWGZmZp2v1iu9EyLi9daRiFgEnFCfkMzMzOqj1qTXTZJaRyR1A9arT0hmZmb1Uevbm7cDEyT9Jo2flKaZmZmtNWpNet8ETgROSeN3AZfWJSIzM7M6qSnpRcQK4CLgIkkbA1tExPK6RvYuIGkIsHlE3NrsWMzMrPa3NydJ2iglvOnAJZIuqG9o7wpDgFHNDsLMzDK1vsjSOyLeAD4DXBERewIfr2VBSQMlPS5pvKQnJV0laV9J90t6StIeab49JD0oaaakByRtl6aPkXS9pNvT/D+u0M4CSedJmiVpmqShku6QNE/SyWmenpLuljRDUoukg3MxPibpEklzJd0p6T1l2thU0nWSpqbPhyvFLmk94BxgdIpptKSPpOFZad5eNfa/mZl1glqTXndJ/YHPAjevQTvbAj8Btk+fI4C9ga8D307zPA6MiIjdgO8CP8wtPwQYDQwmSyJbVmjnrxExBPgLMB44DPggcHYq/xfw6YgYCnwU+EnurdRBwC8jYifgdeDQMvX/DLggIoan8tbnmqvFHhFvpeEJETEkIiak9T0txTgC+GdpA5JOTEl72vJliyusppmZrYlaX2Q5B7gDuD8ipkraGniqHe08ExEtAJLmAndHREhqAQameXoDl0saBASwbm75uyNicVr+UWAr4Lky7dyUfrYAPSNiCbBE0pvpl+v/AfxQ0j7ACmAAsFkuxllpeHourrx9gR1zv72xkaSebcSedz/wU0lXAddHxN9KZ4iIi4GLAXr0HxQV6jEzszVQ64ssfwT+mBufT/kroUrezA2vyI2vyMXwfWBiRHxa0kBgUoXll1eJO19vaZvdgSOBTYHdI+JtSQuA9Su0sdrtTbIr4w9GxL/yEyX9okrs74iIcZJuIXvOd7+k/SPi8QrrYmZmnazWF1k+kJ6FzUnju0g6s5Nj6Q0sTMNjOrnufBsvp4T3UbIrxva4E/hi60h6O7O13nKxLwF65ebfJiJaIuJHwFSyW71mZtYgtT7TuwT4FvA2QETMBg7v5Fh+DJwnaSb1+5dHVwHD0m3Vz5E9i2uPL6XlZ6fbrCen6ZVin0h2O3SWpNHAVyTNkTSbrC9v68jKmJlZ+yii7cdGkqZGxHBJM9PLGkialV7IsDrp0X9Q9D/mwmaHYQXl/6dnaytJ0yNiWLmyWq/0XpG0DdlLGkg6DHihk+IzMzNriFpvI55G9kbh9pIWAs+QvRRiZma21qia9CR9LTd6K9kzqnXIXv0/FPhp/UIzMzPrXG1d6bW+ebgdMBz4EyDgaODhOsZlZmbW6aomvYg4G0DSvcDQ9MveSDoLuKXu0ZmZmXWiWp/pbQa8lRt/i5V/ycTqZPCA3kzzG3RmZp2m1qR3BfCwpBvS+CFkf9vSzMxsrVHrnyH7gaTbyP5IMsCxETGzfmGZmZl1vpr/8klEzABm1DEWMzOzuqr1l9PNzMzWek56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGE56ZmZWGN2bHYBV1rJwMQPH3tLsMKygFow7sNkhmHU6X+mZmVlhOOmZmVlhOOmZmVlhOOmZmVlhOOmZmVlhOOmZmVlhdImkJ+ksSV/vhHpGSrq5Qtmtkvqkz6kdbSvVub2kWZJmStpG0gNtzL+0M9o1M7M10yWSXiNExKiIeB3oA3RK0gMOAa6NiN0iYl5E7NVJ9ZqZWR00LelJOkPSk5LuA7bLTR8iaYqk2ZJukNQ3TR+eps2SdL6kORWq3kjSLZKekHSRpHXS8gsk9QPGAdu01pPKTpc0NdV/dpq2YarnEUlzJI0uiX8U8BXgFEkT07Sl6Wd/SfemNuZIGpFb7gepzimSNuuc3jQzs1o0JelJ2h04HBgCjAKG54qvAL4ZEbsALcD30vTfASdFxBBgeZXq9wC+COwIbAN8pqR8LDAvIoZExOmS9gMGpeWGALtL2gc4AHg+InaNiJ2B2/OVRMStwEXABRHx0ZI2jgDuSLHuCsxK0zcEpkTErsC9wAlV1sPMzDpZs670RgA3RMSyiHgDuAlAUm+gT0RMTvNdDuwjqQ/QKyIeTNOvrlL3wxExPyKWA9cAe7cRy37pMxOYAWxPlgRbgP+U9CNJIyJicTvWbypwrKSzgMERsSRNfwtofeY4HRhYuqCkEyVNkzRt+bL2NGlmZm15Nz7TizbGSwk4L135DYmIbSPitxHxJDCULPmdK+m7NQcQcS+wD7AQGC/pc6no7YhojWc5Zf72aURcHBHDImJYtw1619qkmZnVoFlJ717gEEnvkdQLOAggXU0tyj0DOxqYnF5AWSJpzzT98Cp17yHp/elZ3mjgvpLyJUCv3PgdwHGSegJIGiDpvZI2B5ZFxJXA+WQJsCaStgJeiohLgEvbs6yZmdVPU/7LQkTMkDQBeAR4mex2YKtjgIskbQDMB45N048HLpG0ApgMVLr3NxX4BbAtMBG4oaTtVyXdn16EuS0919sBeFASwFLgqLT8+am9t4FT2rGKI4HTJb2d6vtc9dnNzKwRtPJuW9cmqWdEtL4dORboHxFfbnJYddWj/6Dof8yFzQ7DCsr/WsjWVpKmR8SwcmVr0//TO1DSt8hifhYY09xwzMxsbbPWJL2ImABMaHYcZma29no3vr1pZmZWlpOemZkVhpOemZkVxlrzTK+IBg/ozTS/QWdm1ml8pWdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXhpGdmZoXRvdkBWGUtCxczcOwtzQ7DCmrBuAObHYJZp/OVnpmZFYaTnpmZFYaTnpmZFYaTnpmZFYaTnpmZFYaTnpmZFYaTnpmZFUZdkp6kgZLmVCi7VNKOafjb9WjfzMysnIZf6UXE5yPi0TTqpGdmZg1Tz6TXXdJVkh6TdK2kDQAkTZI0TNI44D2SZkm6KpUdJenhNO03krqlz3hJcyS1SPpqaUOp/NeSpkiaL2mkpMtS2+Nz8/1a0jRJcyWdnZu+QNLZkmakNrYv08YYSTdKuivN/wVJX5M0M7W7cZrvBElTJT0i6brceo+X9HNJD6QYD+vsDjczs+rqmfS2A34VETsAbwCn5gsjYizwz4gYEhFHStoBGA18OCKGAMuBI4EhwICI2DkiBgO/q9BeX+BDwFeBm4ALgJ2AwZKGpHnOiIhhwC7ARyTtklv+lYgYCvwa+HqFNnYGPgMMB34ALIuI3YAHgc+lea6PiOERsSvwGHB8bvn+wN7AJ4Fx5RqQdGJKzNOWL1tcIQwzM1sT9Ux6z0XE/Wn4SrKDfTUfB3YHpkqalca3BuYDW0v6H0kHkCXQcv4cEQG0AC9FREtErADmAgPTPJ+VNAOYSZYQd8wtf336OT03f6mJEbEkIv4OLAb+nKa35JbZWdJfJLWQJe2dcsvfGBEr0u3dzco1EBEXR8SwiBjWbYPeFcIwM7M1Uc8/OB1tjJcScHlEfGu1AmlXYH/gZOCzwHFlln8z/VyRG24d7y7p/WRXcMMjYlG67bl+meWXU7lfSuvNt9m6zHjgkIh4RNIYYGSF5VWhDTMzq5N6Xum9T9KH0vARwH1l5nlb0rpp+G7gMEnvBZC0saStJPUD1omI64AzgaFrGM9GwD+AxZI2Az6xhvW0pRfwQlqvI+vUhpmZrYF6Xuk9AZwm6TLgUbJnZaUuBmZLmpGe650J3ClpHeBt4DTgn8Dv0jSA1a4Ea5GuvGYCjwPPAfe3scia+g7wEPD39LNXndoxM7N2UvYYzLqiHv0HRf9jLmx2GFZQ/n96traSND29tLga/0UWMzMrDCc9MzMrDCc9MzMrDCc9MzMrDCc9MzMrjHr+yoJ10OABvZnmN+jMzDqNr/TMzKwwnPTMzKwwnPTMzKwwnPTMzKwwnPTMzKwwnPTMzKwwnPTMzKwwnPTMzKwwnPTMzKwwnPTMzKww/E9kuzBJS8j+A31X1Q94pdlBVOH4OqYrx9eVYwPH11EdjW+riNi0XIH/9mbX9kSl//7bFUia5vjWnONbc105NnB8HVXP+Hx708zMCsNJz8zMCsNJr2u7uNkBtMHxdYzjW5mi+pkAAAUISURBVHNdOTZwfB1Vt/j8IouZmRWGr/TMzKwwnPTMzKwwnPSaRNIBkp6Q9LSksWXKe0iakMofkjQwV/atNP0JSfs3IbavSXpU0mxJd0vaKle2XNKs9Lmps2OrMb4xkv6ei+PzubJjJD2VPsc0Kb4LcrE9Ken1XFkj+u8ySS9LmlOhXJJ+nuKfLWlorqyu/VdDbEemmFokPSBp11zZgjR9lqRpnR1bjfGNlLQ49x1+N1dWdbtoUHyn52Kbk7a3jVNZI/pvS0kT0/FjrqQvl5mnvttfRPjT4A/QDZgHbA2sBzwC7Fgyz6nARWn4cGBCGt4xzd8DeH+qp1uDY/sosEEaPqU1tjS+tAv03RjgF2WW3RiYn372TcN9Gx1fyfxfBC5rVP+lNvYBhgJzKpSPAm4DBHwQeKiB/ddWbHu1tgl8ojW2NL4A6NfkvhsJ3NzR7aJe8ZXMexBwT4P7rz8wNA33Ap4ss//WdfvzlV5z7AE8HRHzI+It4PfAwSXzHAxcnoavBT4uSWn67yPizYh4Bng61dew2CJiYkQsS6NTgC06sf0Ox1fF/sBdEfFaRCwC7gIOaHJ8/wVc08kxVBUR9wKvVZnlYOCKyEwB+kjqTwP6r63YIuKB1DY0fturpe8q6ch2W7N2xteMbe+FiJiRhpcAjwEDSmar6/bnpNccA4DncuN/Y/Uv/p15IuLfwGJgkxqXrXdseceTnZW1Wl/SNElTJB3SiXG1N75D062RayVt2c5lGxEf6bbw+4F7cpPr3X+1qLQOjei/9ijd9gK4U9J0SSc2KSaAD0l6RNJtknZK07pU30nagCxhXJeb3ND+U/bIZjfgoZKium5//jNktsYkHQUMAz6Sm7xVRCyUtDVwj6SWiJjX4ND+DFwTEW9KOonsivljDY6hFocD10bE8ty0rtB/XZ6kj5Ilvb1zk/dOffde4C5Jj6crn0aaQfYdLpU0CrgRGNTgGGpxEHB/ROSvChvWf5J6kiXcr0TEG/VooxJf6TXHQmDL3PgWaVrZeSR1B3oDr9a4bL1jQ9K+wBnApyLizdbpEbEw/ZwPTCI7k+tMbcYXEa/mYroU2L3WZRsRX87hlNxeakD/1aLSOjSi/9okaRey7/XgiHi1dXqu714GbqBzb/vXJCLeiIilafhWYF1J/egifZdTbdura/9JWpcs4V0VEdeXmaW+2189H1r6U/Fhbneyh7DvZ+VD7Z1K5jmNVV9k+UMa3olVX2SZT+e+yFJLbLuRPZQfVDK9L9AjDfcDnqKTH9bXGF//3PCngSlpeGPgmRRn3zS8caPjS/NtT/bigBrZf7m2BlL5ZYwDWfVFgocb1X81xPY+sufYe5VM3xDolRt+ADigCX33H63fKVnS+Gvqx5q2i3rHl8p7kz3327DR/Zf64grgwirz1HX76/QO96fmL38U2ZtL84Az0rRzyK6cANYH/ph28IeBrXPLnpGWewL4RBNi+1/gJWBW+tyUpu8FtKQdugU4vkl9dx4wN8UxEdg+t+xxqU+fBo5tRnxp/CxgXMlyjeq/a4AXgLfJnoscD5wMnJzKBfwyxd8CDGtU/9UQ26XAoty2Ny1N3zr12yPpuz+jSX33hdy2N4Vcci63XTQ6vjTPGLKX4fLLNar/9iZ7djg79x2OauT25z9DZmZmheFnemZmVhhOemZmVhhOemZmVhhOemZmVhhOemZmVhhOemZmVhhOemZmVhj/H7MmcWHXpzILAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "def get_top_n_trigram(corpus, n=None):\n",
        "    vec = CountVectorizer(ngram_range=(3, 3), stop_words='english').fit(corpus)\n",
        "    bag_of_words = vec.transform(corpus)\n",
        "    sum_words = bag_of_words.sum(axis=0) \n",
        "    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\n",
        "    words_freq = sorted(words_freq, key = lambda x: x[1], reverse=True)\n",
        "    return words_freq[:n]\n",
        "\n",
        "document_corpus = [\n",
        "  \"Dog bites man\",\n",
        "  \"Dog bites man after man eats fish\", \n",
        "  \"Dog bites fish\",\n",
        "  \"Man bites dog\", \n",
        "  \"Dog eats meat\", \n",
        "  \"Man eats food\",\n",
        "  \"Man eats fish\"\n",
        "]\n",
        "\n",
        "common_words = get_top_n_trigram(document_corpus, 5)\n",
        "\n",
        "df6 = pd.DataFrame(common_words, columns = ['desc' , 'count'])\n",
        "df6.groupby('desc').sum()['count'].sort_values().plot(kind='barh', title='Top 5 trigrams in our corpus after removing stop words')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r_sx0rray7xs"
      },
      "source": [
        "# Visualization"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "KlNT8LfRy7xs"
      },
      "outputs": [],
      "source": [
        "from spacy import displacy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YclNGdaUy7xt",
        "outputId": "4285d768-b6ad-4d79-9a1b-dbc5f0b36555"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"0bf72b5ccfc143b4871b353f00ec3470-0\" class=\"displacy\" width=\"750\" height=\"312.0\" direction=\"ltr\" style=\"max-width: none; height: 312.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">This</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">DET</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">is</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">VERB</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">a</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">DET</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">sentence</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">NOUN</tspan>\n",
              "</text>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-0bf72b5ccfc143b4871b353f00ec3470-0-0\" stroke-width=\"2px\" d=\"M70,177.0 C70,89.5 220.0,89.5 220.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-0bf72b5ccfc143b4871b353f00ec3470-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M70,179.0 L62,167.0 78,167.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-0bf72b5ccfc143b4871b353f00ec3470-0-1\" stroke-width=\"2px\" d=\"M420,177.0 C420,89.5 570.0,89.5 570.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-0bf72b5ccfc143b4871b353f00ec3470-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M420,179.0 L412,167.0 428,167.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-0bf72b5ccfc143b4871b353f00ec3470-0-2\" stroke-width=\"2px\" d=\"M245,177.0 C245,2.0 575.0,2.0 575.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-0bf72b5ccfc143b4871b353f00ec3470-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">attr</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M575.0,179.0 L583.0,167.0 567.0,167.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "</svg>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "doc = nlp(\"This is a sentence\")\n",
        "displacy.render(doc, style=\"dep\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AiAhnvhsy7xt",
        "outputId": "bdd7ec6c-cbc5-4184-f370-58aa020b1e57"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
              "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Larry Page\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
              "</mark>\n",
              " founded \n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Google\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " in the \n",
              "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    US\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
              "</mark>\n",
              " in \n",
              "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    early 1990\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
              "</mark>\n",
              ".</div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "doc = nlp(\"Larry Page founded Google in the US in early 1990.\")\n",
        "displacy.render(doc, style=\"ent\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yRG2OkMTy7xu"
      },
      "source": [
        "Inspired by: https://www.datacamp.com/community/blog/spacy-cheatsheet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "36_h_-Gcy7xu"
      },
      "source": [
        "# Kernels \n",
        "\n",
        "Used by \n",
        "- Support Vector Machines (SVMs)\n",
        "- Principal Component Analysis (PCA)\n",
        "\n",
        "Useful for\n",
        "- classification tasks\n",
        "\n",
        "Also known as\n",
        "- kernel function\n",
        "- similarity function\n",
        "\n",
        "Opposite of kernels: vectors\n",
        "\n",
        "Source:\n",
        "- [Wikipedia](https://en.wikipedia.org/wiki/Kernel_method)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PnrfMOQQy7xu"
      },
      "source": [
        "## Linear Kernel\n",
        "\n",
        "Linear Kernel is used when the data is Linearly separable, that is, it can be separated using a single Line.\n",
        "\n",
        "Compute the linear kernel between X and Y: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.linear_kernel.html#sklearn.metrics.pairwise.linear_kernel"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tmpoDv0jy7xu"
      },
      "source": [
        "## Non-linear Kernel\n",
        "\n",
        "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/c/cc/Kernel_trick_idea.svg/1280px-Kernel_trick_idea.svg.png\" width=\"600\">\n",
        "\n",
        "[Credits](https://en.wikipedia.org/wiki/Kernel_method)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r3ipOMCay7xu"
      },
      "source": [
        "# Spearman's Rank Correlation Coefficient\n",
        "\n",
        "\n",
        "\n",
        "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/4/4e/Spearman_fig1.svg/1280px-Spearman_fig1.svg.png\" width=\"300\">\n",
        "\n",
        "Credits: https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SOoUgTGmy7xu"
      },
      "source": [
        "# kNN\n",
        "\n",
        "k-nearest neighbors algoritm\n",
        "\n",
        "Useful for\n",
        "- classification"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dDXTtlbYy7xu"
      },
      "source": [
        "# Text Summarization\n",
        "\n",
        "- [How to Make a Text Summarizer](https://github.com/llSourcell/How_to_make_a_text_summarizer)\n",
        "- [How to Prepare News Articles for Text Summarization](https://machinelearningmastery.com/prepare-news-articles-text-summarization/)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RwAE28kwy7xv"
      },
      "source": [
        "# Sentiment Analysis\n",
        "\n",
        "Is text fact or opinion? Only perform sentiment analysis on opinion, not facts.\n",
        "\n",
        "Sentiments:\n",
        "- positive\n",
        "- neutral\n",
        "- negative\n",
        "\n",
        "2 ways:\n",
        "- rule-based uses lexicon with polarity score per word. Count positive and negative words. Doesn't provide training data.\n",
        "- automatic using machine learning (=classification problem). Needs training data.\n",
        "\n",
        "Sentiment analysis can be performed with ntlk's `SentimentIntensityAnalyzer`\n",
        "\n",
        "See: https://www.nltk.org/api/nltk.sentiment.html#module-nltk.sentiment.vader\n",
        "\n",
        "Learning resources: \n",
        "- https://www.youtube.com/watch?v=3Pzni2yfGUQ\n",
        "- https://towardsdatascience.com/sentiment-analysis-with-python-part-1-5ce197074184"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XLoHwQMfy7xv",
        "outputId": "94260b69-395f-467a-eb64-271effadf2e0"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/Users/jan/PycharmProjects/playground/nlp-cheat-sheet/venv/lib/python3.6/site-packages/nltk/twitter/__init__.py:20: UserWarning: The twython library has not been installed. Some functionality from the twitter package will not be available.\n",
            "  warnings.warn(\"The twython library has not been installed. \"\n"
          ]
        }
      ],
      "source": [
        "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
        "# nltk.download('vader_lexicon')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xzqYJouRy7xv"
      },
      "source": [
        "# Logistic Regression\n",
        "\n",
        "A classification model that uses a sigmoid function to convert a linear model's raw prediction () into a value between 0 and 1. You can interpret the value between 0 and 1 in either of the following two ways:\n",
        "\n",
        "- As a probability that the example belongs to the positive class in a binary classification problem.\n",
        "- As a value to be compared against a classification threshold. If the value is equal to or above the classification threshold, the system classifies the example as the positive class. Conversely, if the value is below the given threshold, the system classifies the example as the negative class.\n",
        "\n",
        "https://developers.google.com/machine-learning/glossary/#logistic-regression"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fuar7BXgy7xv"
      },
      "source": [
        "# RNN\n",
        "\n",
        "Recurrent neural networks\n",
        "- Size changes depending on input/output (in contrast to neural network like CNN)\n",
        "\n",
        "## LSTM\n",
        "\n",
        "Long Short-Term Mermoy\n",
        "\n",
        "ToDo"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yrW6HitLy7xv",
        "outputId": "f69d7b75-e13c-461c-c96e-26aed5f2dcde"
      },
      "outputs": [
        {
          "ename": "ModuleNotFoundError",
          "evalue": "No module named 'keras'",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-63-61823dfb33e8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecurrent\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mLSTM\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mSequential\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcore\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mDense\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mActivation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mDropout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mRepeatVector\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mMerge\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTimeDistributedDense\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'keras'"
          ]
        }
      ],
      "source": [
        "from keras.layers.recurrent import LSTM\n",
        "from keras.models import Sequential\n",
        "from keras.layers.core import Dense, Activation, Dropout, RepeatVector, Merge, TimeDistributedDense"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K--Q9LAQy7xv"
      },
      "source": [
        "# Levenshtein distance"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GHAWezXRy7xw"
      },
      "outputs": [],
      "source": [
        "import Levenshtein"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i9LQAWu5y7xw"
      },
      "source": [
        "# Regularization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "rLI5VSRCy7xw"
      },
      "source": [
        "# Markov Decision Process\n",
        "\n",
        "- State -> action -> state -> action ...\n",
        "- Agent\n",
        "- Set of actions\n",
        "- Transitions\n",
        "- Discount factor\n",
        "- Reward"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vfHAtMrsy7xw"
      },
      "source": [
        "# Probability to discard words to reduce noise\n",
        "\n",
        "![prob](https://miro.medium.com/max/460/1*h4xJftToHQRc_sl1ejpmfA.png)\n",
        "\n",
        "Credits: https://towardsdatascience.com/how-to-train-custom-word-embeddings-using-gpu-on-aws-f62727a1e3f6"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ox5oS4e1y7xw"
      },
      "source": [
        "# Loss functions\n",
        "\n",
        "A measure of how far a model's predictions are from its label.\n",
        "\n",
        "In contrast to:\n",
        "- reward function\n",
        "\n",
        "## SSE (sum of squared of the errors)\n",
        "\n",
        "## Mean Squared Errors (MSE)\n",
        "\n",
        "Mean Squared Error (MSE) is a common loss function used for regression problems.\n",
        "\n",
        "Mean squared error of an estimator measures the average of the squares of the errors—that is, the average squared difference between the estimated values and the actual value.\n",
        "\n",
        "Can be used for regression problems (say, to predict the price of a house).\n",
        "\n",
        "Alternatives:\n",
        "- Binary Crossentropy Loss (is better for dealing with probabilities)\n",
        "\n",
        "## Binary Crossentropy Loss\n",
        "\n",
        "Used in binary classification tasks, ie model outputs a probability (a single-unit layer with a sigmoid activation), we'll use the binary_crossentropy loss function.\n",
        "\n",
        "## Cross-entropy loss\n",
        "\n",
        "## Sparse Categorical Crossentropy\n",
        "\n",
        "Used in image classification task\n",
        "\n",
        "## Log loss\n",
        "\n",
        "Used in logistic regression tasks"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MtUM4wlvy7xw"
      },
      "source": [
        "# Optimizer\n",
        "\n",
        "This is how the model is updated based on the data it sees and its loss function.\n",
        "\n",
        "## Gradient Descent\n",
        "\n",
        "Optimization algorithm for finding the minimum of a function.\n",
        "\n",
        "## Stochastic Gradient Descent (SGD)\n",
        "\n",
        "## Adam\n",
        "\n",
        "## AdaBoost\n",
        "\n",
        "## AdaGrad"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "112r3ZrIy7xw"
      },
      "source": [
        "# NN Frameworks\n",
        "- Keras (best learning tool for beginners)\n",
        "- PyTorch (dynamic)\n",
        "- Tensorflow (declerative programming, can run on Apache Spark)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z0wJ7ttry7xw"
      },
      "source": [
        "# Classification\n",
        "\n",
        "- Binary\n",
        "- Not binary"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oC8fPUZ9y7xx"
      },
      "source": [
        "# Activation function\n",
        "\n",
        "A function (for example, ReLU or sigmoid) that takes in the weighted sum of all of the inputs from the previous layer and then generates and passes an output value (typically nonlinear) to the next layer.\n",
        "\n",
        "https://developers.google.com/machine-learning/glossary/#activation_function\n",
        "\n",
        "## Softmax Function\n",
        "\n",
        "A function that provides probabilities for each possible class in a multi-class classification model. The probabilities add up to exactly 1.0. For example, softmax might determine that the probability of a particular image being a dog at 0.9, a cat at 0.08, and a horse at 0.02.\n",
        "\n",
        "Example: last layer is a 10-node softmax layer—this returns an array of 10 probability scores that sum to 1.\n",
        "\n",
        "\n",
        "## Sigmoid\n",
        "\n",
        "A function that maps logistic or multinomial regression output (log odds) to probabilities, returning a value between 0 and 1\n",
        "\n",
        "Sigmoid function converts /sigma into a probability between 0 and 1.\n",
        "\n",
        "## ReLU (Rectified Linear Unit)\n",
        "\n",
        "- If input is negative or zero, output is 0.\n",
        "- If input is positive, output is equal to input."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3mPxxSIiy7xx"
      },
      "source": [
        "# Performance measure\n",
        "\n",
        "## Accuracy\n",
        "\n",
        "Used when taining a neural network.\n",
        "\n",
        "- training loss decreases with each epoch\n",
        "- training accuracy increases with each epoch\n",
        "\n",
        "![acc](https://www.tensorflow.org/tutorials/keras/basic_text_classification_files/output_6hXx-xOv-llh_0.png)\n",
        "\n",
        "\n",
        "## Precision\n",
        "\n",
        "TP/(TP+FP)\n",
        "\n",
        "- TP=true positive\n",
        "- FP=false positive\n",
        "\n",
        "## Recall\n",
        "\n",
        "TP/(TP+FN)\n",
        "\n",
        "## F1 score\n",
        "\n",
        "(2 × Precision × Recall) / (Precision + Recall)\n",
        "\n",
        "## Mean Absolute Error\n",
        "\n",
        "A common regression metric is Mean Absolute Error (MAE).\n",
        "\n",
        "## Mean Squared Error\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PLtF2wZxy7xx"
      },
      "source": [
        "# Early stopping\n",
        "\n",
        "Early stopping is a useful technique to prevent overfitting."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "scgRiTkgy7xx"
      },
      "source": [
        "# Regularization\n",
        "\n",
        "## L1 Regularization\n",
        "penalizes weights in proportion to the sum of the absolute values of the weights\n",
        "\n",
        "https://developers.google.com/machine-learning/glossary/#L1_regularization\n",
        "\n",
        "## L2 Regularization\n",
        "\n",
        "penalizes weights in proportion to the sum of the squares of the weights"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DmNKLm0ey7xx"
      },
      "source": [
        "# Sparsity\n",
        "\n",
        "The number of elements set to zero (or null) in a vector or matrix divided by the total number of entries in that vector or matrix."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qdJe2NDiy7xx"
      },
      "source": [
        "# Ranking\n",
        "\n",
        "## Wilson-Score Interval\n",
        "\n",
        "Used by Reddit to rank comments.\n",
        "\n",
        "## Euclidean Ranking\n",
        "\n",
        "## Cosine Ranking"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nlQvAcxZy7xx"
      },
      "source": [
        "# XLNet + BERT in spacy\n",
        "\n",
        "https://spacy.io/models/en#en_pytt_xlnetbasecased_lg"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eDKpMYKqy7xy"
      },
      "source": [
        "# Latent Dirichlet Allocation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "osnehyfyy7xy"
      },
      "source": [
        "# Confusion Matrix\n",
        "\n",
        "A confusion matrix is a table where each cell `[i,j]` indicates how often label `j` was predicted when the correct label was `i`."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Entropy & Information Gain\n",
        "\n",
        "Information gain measures how much more organized the input values become when we divide them up using a given feature. To measure how disorganized the original set of input values are, we calculate **entropy** of their labels, which will be high if the input values have highly varied labels,\n",
        "\n",
        "Entropy is defined as the sum of the probability of each label times the log probability of that same label:\n",
        "\n",
        "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAN8AAAAvCAYAAABjax81AAABRGlDQ1BJQ0MgUHJvZmlsZQAAKJFjYGASSCwoyGFhYGDIzSspCnJ3UoiIjFJgf8bAzSDEwMcgwyCbmFxc4BgQ4ANUwgCjUcG3awyMIPqyLsgs1619QSyeWhKnsuv+te/QccNUjwK4UlKLk4H0HyBOSS4oKmFgYEwAspXLSwpA7BYgW6QI6CggewaInQ5hrwGxkyDsA2A1IUHOQPYVIFsgOSMxBch+AmTrJCGJpyOxofaC3RDhZGRu6BGq4EjAsaSCktSKEhDtnF9QWZSZnlGi4AgMoVQFz7xkPR0FIwMjIwYGUHhDVH8OAocjo9g+hFj+EgYGi28MDMwTEWJJUxgYtrcxMEjcQoipzGNg4AeG1bZDBYlFiXAHMH5jKU4zNoKweewZGFjv/v//WYOBgX0iA8Pfif///178///fxUDzbzMwHKgEAF/JXuMCB+4PAAAAVmVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAADkoYABwAAABIAAABEoAIABAAAAAEAAADfoAMABAAAAAEAAAAvAAAAAEFTQ0lJAAAAU2NyZWVuc2hvdH1L8vUAAAHVaVRYdFhNTDpjb20uYWRvYmUueG1wAAAAAAA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4OnhtcHRrPSJYTVAgQ29yZSA2LjAuMCI+CiAgIDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+CiAgICAgIDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiCiAgICAgICAgICAgIHhtbG5zOmV4aWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20vZXhpZi8xLjAvIj4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjQ3PC9leGlmOlBpeGVsWURpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6UGl4ZWxYRGltZW5zaW9uPjIyMzwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgr2RmJVAAANRUlEQVR4Ae1bBaxVRxMerMWhOBRSWijuXqyhNC1QSqA4CZ4Q3D3BgwQIFAIU7SNFihYPLsHdg7u7awvsP9/k35Nz7z1Xnp4Hdye57+xZP9/OzM7M7kugmMiQQcAgEOcIJIzzEc2ABgGDgCBghM8wgkHAJQSM8LkEvBnWIGCEz/CAQcAlBIzwuQS8GdYgYITP8IBBwCUEjPC5BLwZ1iBghM/wgEHAJQSM8LkEvBnWIGCEz/CAQcAlBIzwuQS8GdYgYITP8IBBwCUEjPC5BLwZ1iBghM/wgEHAJQSM8LkEfHwc9t9//6UPHz7Ex6m5Oqc3b97EyvhG+GIFVnc7/e+//+j69ev0/PlzvxNZsWIFzZ49W8r37dtHpUuXpuTJk9OQIUOoW7du9Ndff/ltG5WCGzdu0LRp06hXr14B5xWVvkNtE1lcZsyYQXnz5qVkyZLR1q1bYx4X/DNtKPTtt9/in27l991336nXr19Ls6lTp6qECRNKfvr06dXcuXND6S5G69y+fVtNnz7d8bdw4cIYHSs+dzZq1Cj15ZdfynrkyJFDnnny5FErV670mPaWLVsUM5W6f/++lf/u3TuVIkUKNWfOHLV8+XKVIEECwdOqEM3E6NGjVdmyZYVP7ty5E83eItc8Orjs3btX5szKLMZxoch8Bha2Q4cOPk3wcUmTJlW8PfuUxUUGGAaKIWfOnKpmzZqqRo0a6scff1QFChRQX3zxRVxMIV6McfnyZTV58mQRnKdPn6rHjx+rFi1ayNrwziNzRB6U5K5duzzmfOzYMcHw7Nmzkt+zZ09pd+LECY960XlZs2aNK8IXHVwmTJigsmXLZn12TOISsvBBS4LBZ82aZU1EJ5o0aaLKlCmjX115NmvWTKVMmVIdPXrUGv/u3bsqSZIk6u3bt1bep57o2LGj+uabb6zP3Llzp6yb3v369eunmjdvbpXrBCwHKCr2+SSLTTRZ099++01X8XnC+tEWkC589OiRTvo83RI+TCSquIC3a9eubX1LKLhYlYMkQvb5Dh8+LKZ1sWLF5Gn/wwxPJUqUsGd5pFkIiE1Dnx8c/JiimTNnEisA4p2Pbt26Jd1mypSJNmzYQJ999llMDRPv+zl06BAVL17cmmeqVKkkXahQIXry5AmxJqfWrVtb5ToBvw/4sbkpWYkTJ6YGDRoQfEP4a07EAkv169cnNlmlOCIign755Renqn7zXr58SV27diV2awhzLVeuHLFZ7FH/wIEDVKVKFUqbNi3lz5+fqlatSmzlUIUKFTzqBXqJDi5sLltdh4KLVTlYIohwWsUjR45UiRIlUjBnsJPoHy+o5ENz+qN8+fKpdOnS+fx27Njhr0mU8jGXggULKmY+xcGGKPXxMTeC38ZBEzVs2DDrM9q2bSuYIGP79u1iSurdzarECeA2cOBAe5bC+jD/KOxYToTx6tWrp5o2baqWLFmismfPri5evOhUVfK8dz7Mo2LFioqFSK1bt07BxK1Vq5bw0/Hjx6UN/EP4osOHD1esVNWqVavkGxYtWqTWr1/vdyx7QVRx0dYefGQ7BcPFXjdQOmSzEyBjIfz9WLMEGidSZfBPYO44/YJ1dOnSJTE1x40b51EVCzBmzBjxgeDfxARB2MePH69atWoVVNg5eqjat2+v9u/fH6mht23bpnr06KHQPhidPHlS1ocjlmrQoEGqaNGiqkiRIurq1avSlKN3Hiap7u/Zs2cSnPEWMt6VRBDgR/ojKGEIEHz+YP6ht/DxrirzPXPmjNX9vXv3xGfl3VnyFixYoD7//HMP1+HXX3+VdbQaBUlEFZfVq1cLLsDHTqHgYq/vL5042M6oy2F2sq8gZovOwxPh4wEDBhDMGn/EABL7Bj7F1atXpyxZsvjkp06dmho2bOiTH0oGB3+INTCxre5RHWHmUqVKEWtQatOmjUeZ/YWBJmZ42rNnD7EgW0VoA3PHTvqb/vzzT+JoHrHPaS+20jg7w2/ZsmX0ww8/SFjfKgySwBgHDx4kZnLiHSZgbe0arF27luAewJyrW7eumHNoeO3aNeLItE8fMOswP5iddsLRQ+bMmYn9Znu2R/rIkSN05coVKly4MP3999+Cr0eFAC8YF2uFcL6mjBkzytw58CNZHBySb2fBppIlS0oeCytxUE03keemTZuId09ZA/AOzFNNUcUFpjgH7Sz8dH+h4KLrBnqGJHzwFcCIvXv3pjRp0nj0d+rUKRG8QH4VmzvE5qpHO7zAlnYSPtj+8DciSxxMINZWxGaBMA3aw1+BMOOXO3duUQL+/NObN2+KgMGuh/CCcTV9/fXXOmk9MXcIBM7FAhEYHopr7Nixgao5llWrVo14x3As884Ek3FkThSHdxneIUhsForvB/9JE5gsV65clCFDBp0lTwgk70QewmGvwBaEYMTHOSJ8UCzAuU+fPvZqftNs1tGrV69E8O1KAXPR/AKFV7lyZapUqZI8IZQcdaf+/ftb/ULhzps3j7p06UIcYBLlxuaqpQyjg4vd39MDBsNF1wv2DEn4tObAzuFN0Mp87ued7fE+ZcoUj/fYeAFj41AUgs7RPmuIwYMHE5tFxCF34vC6aE82Y6xye+Lnn3+mn376iSZOnGjPjnT69OnTtHHjRoLWRgDIrrDAZAgCQWlBwUBYNLF5KMoDygeCz76OLvJ4gmGh6dkvIvbViP0kYn+cEFTQu4NHg/+/IEjBJpDspHwUY1XRwRY+hhAtD+UDOn/+vART7DuT1YgTYHDs+jrwwT6YWEL2OoHSUGgcHZV545AfhPlhDbXlguAZvpN9SkJwpnz58iJ89n7/+OMPGReKCspw8eLFgg9HKaVaVHDBPNhFkHWAksCOrCkYLrpesKevDeLQAuYBTA9v0xImGhiNfQuHVnGXhSgbtC1uZ+BWB0xG7H7YBWEKZc2aVSYD4dOM4j07mE4wZ7p37+5dFKl33AzBogMvvnBA7dq182jPwRDR0jBtwXxgKhAWGswDM3P37t0S/Xv//r1HW/0CweSzK2EI9jcJzME+LSHqbI906vr6iYghdiaMZScIHwjl9sgmzGQIKXZMJ9LKSpdB2dh3JJ2vn2BoEHYOUKNGjYh9RTFVX7x4QeCnESNGENJ6HTBXvCNiDhMV87tw4YK013+gzDB3EAQUuOl1jiou586dEwsBlgLcIzt54wK+Y3/eXiW0NAPilxCkYFNCHGDuTbGZpfh4QOrPnz9fopfIZ22u2Pfz209sF3Tu3NlvIAiBAERoQWxuyi0Fp/kgegbHHpFSBCrYh1GsbOSgHtFaON9O9ODBAxkbT9A///yjli5dKmkES1jwJY0/6M8eFe7UqZN1hsQ7rpo0aZJVFxcamKnkHWdU+IFYOCWiyQIq77zDKsydNbOFAZtqUub0B5ck2LxUCGpp+uqrrxRuxNijhxgH0UtESGOCcBGDLQCZI3Dmq23SLaKX+FbebSWqCYzsFwBwcQORckTawWv6xwpBsUXmM7WWLVtK9BUF0cEFlw3YlZJbOQjiaXLCBXMBfqxUdLWQniFHO0PqLR5WYq0ps8LRAxYY0TQnQrQWIW3eyRXvKhIyZw2rWAMqROO0AHu39RY+lCO6hgghbpfg4F8TGAsheU0QVD6LlFcID66CQfDxg0Dog3G78KEyjhLwLbzzyFi6v1CeDx8+FEZhP0pBuYK0QrW379u3r9wWsufFZhrHCIgeexOi7I0bN1a8myk2txXWk31zxWaqgtK1E44j2OyWOvb8UNJOuCCS601OuOBigeYz7/qB3j954dMfzz6Swv1UfwRGZJNMsX/qr4pjvrfw4QYJGAB3XhHaDyR8uO8IIQBBMDlo4DiGXfigXcF8UAgIx2Nnx93WyBCb5YoDLor9YcdmUAp82B30+MSxcQxmIqSPnY59Oo9ecXSCXRm7piYoJNw5xvW5qFJc4xI2wjd06FBVp04dxeF3hStCTqQPilGXnWqnKj553sIH01YzCy5143obNDYIB9kcDJA0ND0HsCxhx5gc0bMOqbFT6/Mlu/CxD6S0WQlB5AiusptF0nkIf7Cj8019x5owabETxAfCoTssEvaHxZyEkoJC+/333y0zj6Pwio9WxAqA1YIfFFRUKC5xCRvhwwLBd4CvGog2b96sOBAgl7S1f4Gnk0/LQRoFXw3lMBVhpuKSN3wqHG5z6Ftx2Fx2XDAEzERcaoaAwkew7zwQRtxG0f4Dn1MJA4HJYJri3iUHM2QMmKQo//77731upQT6to+xDAoGu3xERISCpYCDeu//isBdVvtaIc3X3uL95ybADHmynzzhM3khJSQf2x+LA3FE3HAYizSOAXT4HvNAdBL3Tp0I7RD1Y2FzKrbycCyAyCX6NvRxIhA2wvdxLo+Z9aeMQEjnfJ8yAObbDAJuIWCEzy3kzbhhj4ARvrBnAQOAWwgY4XMLeTNu2CNghC/sWcAA4BYCRvjcQt6MG/YIGOELexYwALiFgBE+t5A344Y9Akb4wp4FDABuIWCEzy3kzbhhj4ARvrBnAQOAWwgY4XMLeTNu2CNghC/sWcAA4BYC/wMTxbTq87gpXQAAAABJRU5ErkJggg==)\n",
        "\n",
        "Labels that have low frequency do not contribute much to the entropy (since P(l) is small), and labels with high frequency also do not contribute much to the entropy (since log2P(l) is small).\n",
        "\n"
      ],
      "metadata": {
        "id": "AsgWtq-5vq06"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAWAAAAEoCAYAAABmaitNAAABRGlDQ1BJQ0MgUHJvZmlsZQAAKJFjYGASSCwoyGFhYGDIzSspCnJ3UoiIjFJgf8bAzSDEwMcgwyCbmFxc4BgQ4ANUwgCjUcG3awyMIPqyLsgs1619QSyeWhKnsuv+te/QccNUjwK4UlKLk4H0HyBOSS4oKmFgYEwAspXLSwpA7BYgW6QI6CggewaInQ5hrwGxkyDsA2A1IUHOQPYVIFsgOSMxBch+AmTrJCGJpyOxofaC3RDhZGRu6BGq4EjAsaSCktSKEhDtnF9QWZSZnlGi4AgMoVQFz7xkPR0FIwMjIwYGUHhDVH8OAocjo9g+hFj+EgYGi28MDMwTEWJJUxgYtrcxMEjcQoipzGNg4AeG1bZDBYlFiXAHMH5jKU4zNoKweewZGFjv/v//WYOBgX0iA8Pfif///178///fxUDzbzMwHKgEAF/JXuMCB+4PAAAAVmVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAADkoYABwAAABIAAABEoAIABAAAAAEAAAFgoAMABAAAAAEAAAEoAAAAAEFTQ0lJAAAAU2NyZWVuc2hvdBTk8xEAAAHWaVRYdFhNTDpjb20uYWRvYmUueG1wAAAAAAA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4OnhtcHRrPSJYTVAgQ29yZSA2LjAuMCI+CiAgIDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+CiAgICAgIDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiCiAgICAgICAgICAgIHhtbG5zOmV4aWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20vZXhpZi8xLjAvIj4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjI5NjwvZXhpZjpQaXhlbFlEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlBpeGVsWERpbWVuc2lvbj4zNTI8L2V4aWY6UGl4ZWxYRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpVc2VyQ29tbWVudD5TY3JlZW5zaG90PC9leGlmOlVzZXJDb21tZW50PgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KYOk3fgAAQABJREFUeAHs3Qm0fUlVH/5rYiZFJIAyCPhrJpmnBhkbGlRAQGUSTRQiREHiMmrAWcGV5RCzCCFOEUkMjTJoBBQEBBMaZGpohibYQDM0DagQFaNkHv//3+fA97lf9Tn3nnvf/b3hvtprnbtr2HvXrl1V+9apU6fO5/x/Z2HRoVugW6BboFvg0C3wVw69xF5gt0C3QLdAt8Bgge6Ae0foFugW6BY4Igt0B3xEhu/Fdgt0C3QLdAfc+0C3QLdAt8ARWaA74JmGX/Ws8v/9v/+3+IM/+IPF933f9y2e/exnz5TayboFugVOswU+p++CmNf8HLDrr/yVz/xnCX/O53zOkAb/9//+3xevetWrFo95zGMWt7/97RdvfOMbF5//+Z8/T3in6hboFjiVFugz4JnNzsnG4WJJ2MyXM/7IRz6y+Cf/5J8sxK+88srFM57xjCE8U3wn6xboFjiFFugOeI1G53TB//2//3dwupwt+LM/+7PFv/t3/25x6aWXDo75P//n/7z4pV/6pcUHPvCBxf/5P/9noOk/3QLdAt0CrQX6EkRrkSXxLEMg4Xz/6l/9q4MjvuSSSxYPe9jDFv/pP/2ngdsyxV/7a39t8cAHPnDxW7/1W4NTXiK2Z3ULdAucUgv0GfCaDc8Jc76ZDX/84x9fvPCFL9xzvtI54P/5P//n4vWvf/3id37nd9YsoZN3C3QLnBYL9BnwGi3N+Wb54XM/93MX//t//+9h6cGDt//6X//r1SSZId/sZjdbXHHFFVfL6wndAt0C3QJ9BrxGHzC7dWUpwsO25z73uaPOl1jO+mMf+9jiJ3/yJ9copZN2C3QLnBYLnEgHbAngf/yP/zE4OLf6IA/EOMcxyMz1f/2v/zXQos9ywhRPlROnGydsxuuh28te9rJKdrUw/X7u535uYakiOpAl3OHcWGCsPcfS1i19GzLWLbPT77YFTpQDjuOKE9Q0f/2v//WhhTIzhcfAcoA8D8es0brIiyMe42nT6gB897vfvXjWs541rPW2dDWO50/+5E8WP/VTPzU43ejOAXcnXC21nTB7py+QKJ60tgTpoal5bXriVW6l7+FugU0tcGIcMEdpzTWOlOPlRLPNK4Msg4VBEkZjxsrheWGCnPCRR84qMPjiuP/4j/948epXv3rxnve8ZxXbkK+sX/mVX1n8/u///l7ZccSzBHSi2RaIk6xYuIJ+kb5RccJohcMXWTW/yuvhboFNLXCiHsJxwpzZf/tv/23xkpe8ZHHRRRcNM1C3+He5y12uNmAygAwc19//+39/cdlllw10nDFZ0r1A8bVf+7WTNkRTZ8p4//RP/3TY/5vBaUnEmvA3fuM3Lr7kS75keCuOw47OZt4eyP3Nv/k3h3LIBMFoo++Q0X+2YoHYN8LaeNpPfvKkjbXFWFrkdtwtsIkFPncTpqPi4aT+43/8j4t/9I/+0d7s84/+6I/2HoJlgHB6CUdXs973ve99i5vf/OaL+973vosv+IIvGLIMutve9rYhG8VkKRugF7/BDW6wuP71rz+EOWRpKZOzvd3tbjfE0dMn/GiFE6+DfrTwnri2BdJG1bZtOG1FuLD8tEkKDM2YvNB03C1wEAucKAecQXTrW9968U3f9E2Ll770pcPFABlEdbBkAMEf/OAHF//lv/yXxVd/9VcvHvrQhy7+1t/6WwMPh/h5n/d5K21YZY2VIc3WNHSRHb0scwCOOEso4niqc5bWYbsWGGu3toRK0+YlnjZPvONugW1Y4MQ4YAOAs7zOda6zeNKTnjQ8TPOiA5DnMoMxmDLLjIHEP/GJTyzsgPjSL/3Sxd/+23974McDMgBDvwpXerKBtOhgtl1BevKl44mzjs6Vvoc3t0BsHQmJVyxcIfHg2r6hq+03lh+6jrsF1rHAiXHAKmWAePh27Wtfe1h28ECtQgZQTRPm5D760Y8OD+AuvvjixVVXXbX4si/7smE54hrXuMawLsshTkHktgNPvL1tNcO1BNECGaGtM+JWZsvX4+tZYMqe0j/96U8vPvnJTw53QuLW8f/wD/9w6Ff+lG9yk5sMhTnF7sY3vvFwJ5PStd+U7NB03C2wrgWmvc66kg6B3gCot+xxmtJzjakhz8MvTtdDuN/93d8d5Bhwj3/84xcXXnjhcHRkHG1wlSXNzNXstuYLkw8scXiwZ++vi36wmbc/DnThDa5ptbxNwtEDb9UrZaTMKdnhD0bH3iAyhsjED5opurbs0FUs7M8Lbe4SFOWPS5or+ghLZ2980l3Z5+1Zwfvf//5h6YmT5Xid1eFPGw/w4NQDXbL+xt/4Gwt/xsAfpLDrpje96eKGN7zh4ja3uc3wh/2FX/iFe0tMyo/+0TH8NT13SaHXh5I2FNj84B0DekbuWH5Nq/okHX8LVV5b7hh9+FvapFe8jB8dGSkf1n6Rqz0yUakydy18YhxwGgtOR64NJD0dVEPKS1yjPeABDxgcMIdoIL7hDW8YDsr5l//yXy6+6Iu+aPHlX/7le43v/Ibf+I3fGGZItcEzyGtaDf/FX/zFUKbB/6hHPWpw6o997GMXt7zlLYcBVwdFdFvVSav8TcKxm3JWlYU2EHuu4gk9XPnn8qUcmH0MPM5JGE6YPH9oaXPp2iM0bO+tw/e+972Lyy+/fM/h/vmf//nwx8jRtktDVfexsPI4XDNid11myZyxB7kesnp4S08XvWD1iG5xKNLIUgd0+mD+BMbKTRq+QOwpLeHkjWG6tHRtvOUbK6+laeOVp+atKiu00ZMcdou88847b6+tQ7uL+MQ4YMavjSWexqrOVqdH14KBY+cCQHPHO95xGJAvf/nLh2Mjzz///L1Bgr/KUE46iFmTzmVARR8ypWVQpfPBOpVZGVCutFzopVXHMhBu8EOOsshmDxeIjYTlLQO0aGDy4rDMSmtdp2SkrLYccbq56KW+KSuyxGsa26RdYyez18xSOVSXIz/tx/ZijD8+ywpefOHkDgrK/dSnPjVcnDv9PGDliO2AsUzBCd/73vceZsr0ZSdOX53VMy8K6QPyyEh9lukXW5CDHo7t8IkvA/wgdJGnDcgjyxW6ZbKW5ZGVdtVPyNN31gE84WMnOq8rY53yjhPtiXLAGkZD6ci5GFMHCMivnS7pwRoWzY1udKNh8Lzyla8cPiXkDF8DC9hTzFkbxGgBbInBINdxv/iLv3hvcMkn18zrbW972yDn6U9/+nA7baCizxIE2nQwt8WciNkVx6JOmwJdnUusA5Nnmx2do78yY5dlZYSes6Mf3dVhbF27lZPy2rIMUk7JGuw1r3nN4UFqeFOeeMKwNoXZ1UUmXehhtqvd3vKWtyyuOruezzlylP4wwhv5wZFdbXCta11raEd8+PWBmh/eitHqAy6zbUta9HAHdcEFFyzuec97Dv1GXT0wVt84Xji2WVVO9PWHr12Vq3+St4qXvilLGD15sLs/l/6mn/iTqPkpN3zwFLTtet3rXnevnDk6pqxg7aw91XUO/5ReJyn9RDngNJTGyRVji8sP1jnEDVoNakDHCUrX2GhggxpdeC1J6EwtGFTWE8kx+3G7HIgcsqXf9a53HcqXrxwdvYKyDCwO2KzKre5BHDA5HAg9bKsjjy4VlLkKwkNfb/yxi8GaF0iW8YcXTS2L/TkSf2Dk0C1QecInLfywtoPNbD1EzeH3HC+5sVtk0Zn+2sEfrZ0v7oD8aV7vetfbK5/d/VFl+UL5nCY72jWjPPvMPcC96qxjYFt6KCc6OuPDZY/5m970poU7Kc8arBmrK9vRJTypI7wK8Oib7JZ25YDnAN5AdIbZy5+hPqzusPRAy5f0Maxd/VGzf9uuVeYYr7Ralrg6kiW9zZO/i7DfK5yAGqaRagNrtBrXKZ73vOcNa3ePeMQjBodksBqAnJMBbfbyute9bhhwt7rVrYZ0jR45GdTVJBnYcDpK8sWlAzJqBxqTlXy4hiNvXRw5FbcyUk6bPhVHn3puwhs7VJ0SXlZm8rQ1MDv97d/+7cVrX/vaYbnBwzV/agEOTlkus3V3MNZoOV5vJfozNdvjvDj/dt+3P0IfVDUjRh8n5Q/XbJEj9serz1jqUD4nRj+Xcs2IOe13vetdizNnzgxO2JuX97nPfQYbxo503sSWsdu6vLW8KiPhmi9cYU5ZkdPStvEqdyocnuApul1KP3EOWOPkSkNkoIrr6GYtHq4ZmPe///2HwfLzP//zw0zTYOSg3/nOdw632H/n7/ydYbaKtoJBBYLJFU680oaOYwdxxEPk7M8Uj/y2LuHZFC/TcZVMvG3nX6b7KnkHzTcT9SfphRtfHfnQhz40OLxWrpmuJYA73/nOw7KSF3U4X21thrcKtK32h7Wdh24uzjhgtmem++EPf3i46ONsj3e84x0DSexktiqdk/bHYXnC6+m3uMUtBvmRtwxHVtsWy3im8sbadIp2k/Tougkvnqqf+h5U3qZ6HBXffq9zVFqsUW46pUFnvY3zdXuZhoPNdNwKmum4zeQYbSfyNpyZCh6zFLPjC89uQTMzrh2hqpPyatpUODq0PElv+aTnavM2iddyhFs91pFZeavcuTJanjYeOWN6mt3+h//wHxb/9t/+22HWa119DCwXfOVXfuUw473Tne40ODl9Qd9YF6b0ixxLAfqQfnOve91rePDGGVsD9gVsu2rysBWPOvjz8IDQH4dPVvlE1djSVsqYg8fstYwv7Ri+VfVcJmudvJS3iif6Vbq5vJXnpIZPlAOujWXNyeC7+93vPtx2mr0ANNb5nBdhFuOW0+B5whOesPewxizHHmBrvcJ4quzIGQR+9kencHHeLuExkJ6ZcPLHaJMGJxz6TXBkbFtedCG3tVHy5mJtpC2mgF09+HPQknVeSw5mwS1whJYYLB19/dd//bDeSi4d2zuZlncszlma4dZljTE6aWxgVm2Wba33K77iKxb3u9/99pzwm9/85r29xejNgv2ReEBrecKfvh04+uZcm6KrMJcPT+VNGE44chPfpI3H5EXuOjg6bEveOmUfFe2JcsCMlA7CcVrXc43B7W9/+33JZsMG7CqI/DE6HSMOGA6k48yNo6s8NRwZm+BtyUnZ5G0is/JEBpxb/civ2LqrWe+/+Tf/ZmFroDXXgDbB70/T7NOebmu9/kC1qbucZe0WOWOYXG0ZByy+TJa8OPksWXDA97jHPYZDnrwe/5rXvGbx9re/fZgRk5cZvRkzR2xt2J1Xdt3Qa1mZ8slxbRMiL3hd2ZvyTZVD3io7TPGe1PQT54BPqqG73vstYLBxfAach11mvL/6q786vBxTB7Z8SwoPechDBqflrscM2Hf26t7a/dIPJ0Y3F33dkTllz1r0eWdfIjhzdqnCsoQHc5w7cCb1q171quGZhOcU6uRPhSMnB+5wuizQHfDpau8jrS2HazYIW6aBOSKfdXr+858/zBqr8+WQrO1yuh5k2drl9j+Ou9IeZsXidJUfJwwDs/Esi3kOYR3YGrHdFNHXgzk7KzzU+4Zv+IZhFp/tYO7sOpweC3QHfHra+shryul6YYQDteRgp4BZr9e+OeKAW3z7Zz20ssb78Ic/fODhnPByZGTBcWrhPWyc8oOVn+Wxb/mWbxmWTH7rt35r2Mlx1dm9xHlDz4O55579oKs1b3R5aBxHHnzY9enlHa4FugM+XHuf+tI4Kg/W7GzggH7zN39zcMYxjOUGM11nNnNMlhsApxynywmD1kmR3aYNhOfgRzmu6BIHDHPAliTkcawe2Dk/4td//deHettDDMyCPaCzD1ldvUlnl8Rh1eEcmKWLXNMC3QGvabBOvrkFOCfOxi35L//yLw9vtXGqAbfv9m1bbnDgviWI6lQ5tsSF8WbdVPphQspL+cqW5vLA1wzfsgJnalukXThnzq4LW2p5xSteMWyHxIPO4U+2R3oj72u+5muGl4O6E2ad3YfugHe/jY9NDc383vrWty5e+MIXDi8rZPbI2Xgt9sEPfvDie77ne4YdBXFwrSMS5/TgNu+wK6p8elacMOdr1l4dtN0bHrpxyGbDHtDh90fihQ4vD3mwyAnXl0AOu169vMOzQHfA58DWrWNo44qsaTV8EHUiB074oPI2kYOnOlCO1rkSZn4vfvGLF9ZCA2ayDj76uq/7usH5+nApmFNupallRvZcHDlwwqt4Q9difOruYWNsUGVZinjKU54ybEF70YteNLxVx+mi5YTNgt0lfPM3f/PwNh9eZXDk2f6WMqvcueGD8LZlVFk13NLNjUdG8Fy+k0zXHfDM1tMpcmFZ1knavDYefukGVp0lzVTnamRk5bpa5gYJdIq84HXE4AFmd572e9DmfA7rngHrvc5K9jq4ma910znAWbkygw5Pykx8Dk7dgufwrKJxMJItZ7A6+pOp4ICgH/iBHxgOCrIU43wJrzADD+ee/exnDw/rvvVbv3VYjpDO+XrZZN06tvSpZzDZLY20uYA3faXybCIzPHDCVeYuhrsDntmqGfQzyfeR4V0Gq/KX8U7lkbluJ6480QlOeKqsqXTO10lidjp44Fadr/VeZzd8+7d/+8Kh9etA6hUc3qp/0ubgTes3JbvKq+FKz5n64/HW5i/8wi8svMDBaQN/WF5IwWsmnJnvnDf1ahk1TBZ7TelTadcJkzcmM+WtkjWXbpWck5rfHfCGLTfW6SKqnZklvXa28MMJh24TvA051aFFXnSDa/4qHTlfJ9BddNFFi+c85znDEkQcgC1m3mj7B//gHwxLD6tktfnRiZ0Tbmnmxiu/sGudeo6VEzmt7CpXntmxr3SbQbKXh5Nxwt4EZDt1fPSjHz3QVv6xcpelhbfqtky/ZbLaPHKm+nxLuywefaLjMtpdyesOeM2WTCcJW+LBSa84nT+45qWzBde8dcOrZCzTsS2rlTWHN/XLsgMH4hbbzFeeQcr5Oj/h+7//+4dT6KpcNDXe6lTj6FJe0ufyt3zhT9nBSd8UVzltmeJxWh7OpT7enuOExR2RaZsaR+3BXD3sp8oe06+WF9rgSp809MLhS3qlbcOVJnyVpubX9ITxVL5V9OHbJdwd8AatqaMYPBxNBhExbVjnkubWsXY0tJHRhsU3gZRNLr1cgZS9qoPLRxu9o2Nbz8htsZkcHrM3Sw7/+l//671D3b2AYNnBuQlxvpn5kRPe1KOVnXjVTVm1ruJzodqEjPBWu82V1dKRlauVF/smP33jq77qq4a1YvXnhB2ZisYJfrau+ePyUoozi2ODttwar/Wr6eSTC9MtW/sqzRRvpREmI/LIqnWVHjktX+L00O4BPKcNugNeo8UzaLzb72m1QZJOI8+ZBkBHlG/mgsbMxTqeDokOwNLx5wHMqg47ME78kOVSdm5jyYvM6D7Bvi8ZT95Yk6FeHiitAnVUb0/3f+VXfmVv2YEsD9i8ousgGmfjkhfnk4dUyrUvdg6wW2yet8uqbadkxB5w6FM3cuxAOChoB7JruyovZeeBmvKkqQtH5BwJ9qGPF1XIAZywNWHO15KFfrVsPThlpX6wNDiOXdm2BdKl5kdH5YZfeAz0Ne1Ff9jr1gDfKt7Iax0wmevwR85Jxd0Br9FyOqcOo7N5hRSko8FJ04msf8YBc9g5hrHSC5Op4xoMtfOvodaeHpHtCESOJPLg5K2SG51Cry72q0bWMn7O16E69rh6kAQ4EXbwlQovWHizjZ2qTdg0jiB/HsvKiW74vMCwTv3IbelTZ86J3nPqukw/Dgm07Rq5/ozoLh4nHMfjdLfHPOYxw9uCDnaXjs7hQz/7sz872MmHQCNrSg/yY6cW08+fA1uTk/pHVtISn8L4wutP2huOc3mrzNSFXpGXtEq3i+HugNdsVR2Es3A7XW/fpLtNBDpPziG2Id8gy1tR8tPRzHQ4IrJqPpp1wYwosxEOzwXo4qKfaw6gJ88gVUdvdmWWOsWvjp7k2/EQ56s8dfPAzZttXjGml7pWedleJa2mj5VFN/Yz4GF2jvNep37VJnTnfMlJG46VPTeN3divbVdluqRzkKHhZAH95ZvlCtsdYVuafHV1XKe0H//xHx/2CKOdgjhgciIXrbrqd9ogfwTSK03kSlsG9CJPH9au6gVW8VWZykp55KVd15FR5Z20cHfAG7SYjuajnLXD6TC5BeNQzpx97TRgFqjDVzCgrrzyyuGW3bm2vu6wyvlU/jZsNuNAGwPC1iby0rFb2qm4OoSHPOfXckre3jJYk1f58RiAnO/rzp78ZeYfUGcH6ljzZQ8zc2+8+WoFWXWQRbavk6wCA9X32di1bQe8kbVKjnw6eEhouxzn60WQdfjHyuA0ORJ1ceavdl1Xph0izjz+0R/90WEJgp76jD3DPq/1S7/0S3tf15BH/lgZ1cZ0dcfAbv64vACjjSrNmIyxOkrT17xgo13Vs769F52meMfS1c+sn9x19BiTdVLS/nIF/KRo3PU8VhYwaC699NLFT//0Tw9OOMpx3D5I+R3f8R3DbofM+JLf8XILcEC+nvEjP/Ijw6vLoXaX4zjLn/mZnxkcFfsDf4IdTp4FugM+eW12LDR2+2wmapb8z//5P9/nfDlbn9154hOfOByug86si1PuMM8CZs3s9aAHPWjxvd/7vftsZxbrIaflHjNNbeGui407nCwLdAd8strrWGhr1sVBeOjyrGc9a/HqV796cAKUM3Nz3sGTn/zkYQYnzpFYR85s7VhU4gQowXaWRdxJmA37YwtYMvln/+yfDbsl2JfztZzgz67DybHAX7boydG5a3qEFuBEOQazLrfBPhmf7VbUsv7s9eK/+3f/7kDHIeCxrneQNe4jrPKRFW12y37WVn05w1Gd1QnbGWGJx7MHzlebdBsfWXNtVHB3wBuZ7XQyxfmabTnVzJWtdyziSfjjH//4xeMe97jBIWT2yynkyf/ptNzmtTa75XR9gfm7v/u7F2fKw10O2hLQU5/61OFPDm2Hk2WB7oBPVnsdqbYcAef7vve9b3gK74l1hSc96UnDjMwTcbSZjXHEoM7eKl8Pj1sgdxpy7ULxpe9nPOMZ+74E7u7jd3/3d4eXX8al9NTjbIHugI9z6xxD3Qz4n/iJn1hccskl+7YvOavA6V62NnG0nIdLOM64O+D1G9RdhZmt3Q+2jvls0dOe9rQ9QWbBXpT5x//4Hw/f2NvL6IETYYHugE9EMx2OkpmpKs3aoyfr1hbjUG3g/6Ef+qHhbbe80ormrne96/DJHd8/i5PlGHLZIuWBHf4Oqy0Qu6G0rst+2sYS0LWvfe3hK9EeyuUPTht5+UXb2CFR3yZEUy8y+1IFKxwP6A74eLTDsdOCYzXg81SdU/hX/+pfLV7zmtcML49Q2OzMRyQdHO4wmQxstBXEyWrTK00Pr7YA+3GmDrH/ru/6rr3P2VsWkudlGIe5p81aidpAm8EdjocFugM+Hu1w7LQwk43TNLgvv/zyxa/92q8NRyTGEcj/tm/7tsXDHvawyTfljl3FdkAha+t3utOdhhmv6viztMvE23dOoHvPe94z1FI7xRnDnHd2owh3OHoLdAd89G1wLDXgXA1Sa4+WD37sx35scdlll+3t9zW4HRrjVtjWszxwO5aV2TGltItXum1LswPCEgX7W66wHvzDP/zDQ5vF6cr3hyruLiV/rDtmlhNZne6AT2SznXulDdoM2Gc+85nD1xrqGq7Tzez1vfWtb7239HDuteolxAIcqrMiHvWoRy2svQe02zvf+c7FL/7iLw5JuVuJM5aIt8+AY7Gjxd0BH639j2XpZkhmSgazz+R42eJP//RPB13NtOzpte7rDS1rih2OxgLawvGVT3/604dZbZyqnSoO7Lnq7NenM9vliHNxwB2OhwV6SxyPdjhWWhjI1gqd7/vcs1+2+MhHPrK3lmhAf+d3fudwypnXizscrQX8Ad7tbncbzougibbjaH3O6Ad/8AeH2a6liYC8DsfHAt0BH5+2OFaamF298IUv3Dv2McrZ7fB1X/d1wyfVpfUBHcscDeZwvfjiVWVLEWkPSw7O6PBNPiDdzBf2J9rheFigO+AN2kEnTkevt3hTojIrCU+w9Fx4I3cTPFZ25CQvcRgEjw1Is958IDL8HrZ9y7d8y/BGlifvwECPHPEaFm8h+XDCaBKfg6s9q4yUtUxGaJbhZfxz8lrZ0TFY/jI54a80Na2GQ+NLI84Qtic4YCnCYUnOieZ8XQ73qdsFo1PFkbkKa4cxWEcW/tBH1pTc5O8S7g74gK2ZzhJMXA2L62DSkh4sL9B2wqTPxS1/W+aYnJYnNNZ+/+k//afDOQ+V5rGPfeyw7msQpz5ZTwzdWN0iF675bbjGK08bnltWy7csnrIjexntqrzICI7s4KSvkhP6YPQ1LJ4/T8tB7k4e+chHSh5AOQ7ssTc4yxBTZVe5NRxZY5isMXnr8Fe5kRVc83Y13E/vWLNldS6d3ub33NKlI9qyBZJvpoHOJQ0d/tAbFMLy8caZranSQF7L4EDJU1YdDCkXQ/RIPv2EzWyf85znDG+71V0P97jHPRYPfehDh8PBaz3xREZwLUdZ6pm60s1bdGjRBSpv0sYwutgSv7Xq2g5jPDUt5QSTQSeY3NSt8qwbzl1B6poZZy2z1n1KPvq2XdlOfS0R0Tf1F77mNa857Mv2Xb48NKWLc4Mf/ehHD3uH1Q+tdLzCIPpEx5o2EDQ/6NQv7QrHdmRFXsO2L5qy4PDQJ+n7iHc00h3wBg1rUHjlM4MgnU0a0Ll9mt3toE6powujS2dDl9d5fZCzvj4qb13IgFKGD3LmA4nkKDOdOgNOunCcvjBen5gxY6ITmfisMT784Q8fziJQx8hLfuIwGS5QMZuJ00s4tHCF8NS0GkZPV2UD+nBw+Fbxoq/lRYfooz08vDooxBE5JtIyQGysvOg/V1d11X/UV7t62aI6YE4wMvVHrypbD7YLIuDEOl8scYaHcsmil3pHt6oPeaCmRVZwyqQbOu1KFyC+jLfKQFd1SLuGZtdxd8AzW1iHS8c0KOq/fzqbWQAQ17kB7ArNkHj2R9xFZmYRkR+adXAcKJmRh7/qLR490CcvaTq/L+864pAMA0Oeg3Y8ac+MHp90V2TUQZQ85QUysJTLHpEhX3guVNn4tEOt+zqyUmbajZw4keRtgukI1LnaKLolf0p28tEL0wvQU5itOVt59BVXVnTXXtbvfcQTyPdA7r73ve/wgVQ7J8KbMgbCsz/ibVryWkyX6Kb89PmkkTMHUl6t9xy+XaDpDnhmK+oc6SA6sA9pul1PGmz2CAyOnAqmY2YGXItCb2biNt8M07pqnFilmxtWhhmSQXita11reFMqAyBYmS7xDGTyk/byl798cfHFFw/6hsdBO3/v7/29xe1ud7uhXtJdeEDogqWlnORzHGbUZkne4DJLC6TsxOdguvv4J7keDMahbCJLebbbaTtHPmq3g8Kydq22W1ZO6sKp1XbN+ru+goYtQgu7HOD+fd/3fcO5zPL1R7P7F7zgBcPBSeLaQb/L8khk0Gmujttq15RHh6vO7l02Zk4LdAe8ZkvrJByvgdB+FVmnBgaH9bgAp5hdA0kjx8DigMlx1KCBsSlYwuDgdF4PZDjh6hRbuQam/HR+t8u2LHFGZBiYdLZ26PtunGYdpAnDgVpeTVd/DsBtL2eprim38kTOMkyuy1IBXu2Q/cjSN5GXmaNloui2TIdVeVl/3bRdYzt10T+m2jX1DaZX2tUDOTPhl73sZXu2dn6zs4Mf8IAHDF/NVtf0y1rmqvol359DbVd9LlB1StoqnD8Ljv20QN8FsUFL61xj1zJRLT1aaYE2f914Ky9yx9J19JQNi3tQ89a3vnWYVeKR5pb1y7/8y4c/CHQcQuWtZaSc6J34HJrwzMFj8mraHBloKohXxz1XxhRdK7vSyavxqXBLV2UmHJ2DpecuivN/ylOeMpxWlz9Ud0nPf/7zhz8vabXsyKxpc8Lhq3gOX0tT+Wt9avouhrsDPoRW1dl0qnpJC9Rw0tbF68qgCx4O1Z7fX/iFXxiWHlKu21MvXJj5Rm95wmM4aclPfCD+7E90rDTJH0tL3hQmr/JF/hT9snS8B+FfJjt5VdekjeHQBbc0q/TEZznFn+cTnvCEwSnj4XQ/9KEPLV772tfu+45fW04bb8vfdrwtb1X9tl3+UcrrDngD6+swY9cGovZYxuStm7YnrAmQU8EsySXd7Z6Pa37wgx8cnHHonHLm3NnMqJIenRKHx9JqfsKVTriF5K/CBmiliZzIrHlT4fCM4SmeuelTMpO+Sk7o4NDWtDlhSwuWe7yccebMmX1t64FcHrSSH4dXy0p4GZ7SYxnPWN6UnNOQ3h3waWjliTpa+3z729++uOiii/YNUN8e+6Zv+qatrIdOFN2TD8kCN7nJTfY+kpoirZ+/6lWvGh6MSosDDg5dx+feAt0Bn3sbH8sSLD14uOOoSXtLA2YoPq5p9uuhVGbKye/45FngW86+Pn7b295230PeV77ylcM35DxEq9CdcLXGuQ93B3zubXwsSzD7fctb3rL47d/+7X36eePNFy7sLjAY2yWIfcQ9ciIscMMb3nB4Qy67RSjtRSHHjGbrpD9k4A+4w+FZoDvgw7P1sSmJY7VV6l/8i3+xt3mecp6c2z/6xV/8xcNsqTvfY9NkGyuSGa3thO3h+S95yUuG0+7sjli2u2XjwjvjSgt0B7zSRLtH4FVZ5wW4Kjz4wQ8eDnThiDMjqvk9fDItYFZrz6+lCHc2AaekuQPKK/R99hvLHB7uDvjwbH0sSjIj+pM/+ZO9T9ZEKdvOHOBt+xLwUkhmT6Hp+ORZIE5Ve/p8kZdqrO0HfvM3f3P4iKeXKvosOFY5PNwd8OHZerKkDJJJgpkZkQMnHFbO1D5Qb6OZ+b7tbW/bo0Fr3fcOd7jD3oOaLD9s4oRr2QkHR5+D4GWyluVNlbkJz5Ss45au/dKG17nOdYZPSfmzDZgF25b2qU99akgKbfKX4V2227J6bzOvO+ADWrN28IhqO2YbR5e04PBuiiMHTrjKoqfXWj/wgQ8snvGMZwwP1zhZlzXff/gP/+Hea6n4OGty4HO9HDGmb9V9LLyOoxjjn0rbRJcpWUk/qEz8kbFuvdGnLc2C733vey9ucYtb7JsF25KmX2jn/PFG97k4+s2lH6OLjOAxml1L6w54iy2awREc0W1cetKCQ3sQrOMuk2f2+zu/8zvD9iO3oQamzfrf/M3fPJwVO9XxyVwmt9W50iYc3NJuEt+mrFr+VP0rzSbhg+qLPzLW1RF9eMiwvu+MCGvCQN6VV145HMJkaWodiE54angdGaedtjvgI+oBGRTbLp7crOVlUMCcrQ34XroA6MyInP711Kc+de9M3apP+GvaJuExOTWthlfJX0W7Kv+g8lfxT+UfVK8qd52+U/tDwr5m7cQ0By7pA9KzI2LuQTipT/A6OtW61HBkwQnX/F0Mdwe8xVZdpxPWDpYOdxCsGrX8LBukHKecveY1rxleOZZmOeLzP//zF49//OMX17/+9WdZYZV+U0LoVXWLrpFX+ZI2hascNGMwxbuMvspZxj8nr8pKuNV7mZzwTOFlvMmrvMqWrk8Iu+u58MILhxP7OFx5XkV3FGl2RETOFI781AtdC1O8Nb3lEY/MsbxdS+sO+IAtqrOs6jCr8g+owh57OwjElW3gOT/3ec973kAbnZ1p/OQnP3mPv9VTvJW5R7wk0MpBGjnB0qLHGL38MUidklflJW1TvE1ZVYd16lf5thVO+cHkfv3Xf/3ChzwzC7b2axb8vve9b+gv2yp7mZyqT+iknat2SBnHCffzgNdojW12jHQ+uF5rqDNJSk9ON4dtOwzdzgcPWoDBZruZk7I8GZ8CcqIbmug8Rb8sfRlvyplbRtqh6payl5UTmikc3uApuoOmr5Kf+tVyWp42XmlrmCzLDLkjwue67nWvu3DgkmUpr6LbhmYWfNllly3ufOc7L+0XkV91qOGx/KStwqn7mLxVvCcxvzvgDVpNJ3Hp1OkwiUecuE6ExvqrQVAhvOFDcxDIACODrMgXvursVway9itfntnvE5/4xEF/etIDrXD0lhb6Kn9IHPnBFwivON7wp74pIzzBlS+yWhxZY7Jb2jaecpKuvFzkxgbJ3wSnDql3KzP5q2THRtEPbmVNyUg98fjDTZl0Sty3417xilcs3v3ud+/l/8Zv/MbiLne5y+KCCy5YuSOCrOiWukYf8eiQtBa3+ZEnPfq2PLsW7w54jRZNh7Fu5iCbvMJJhM6TQ20SttbmsBOzi3x5gIx0NOcx6GjWYw0K16ZAlsFJtjfdrPly+ma/thm9973v3RPtTABn/doJoWy60VF9AB3JoJu6qlc+NLknZCSAz5VBGVl0I1s6TKfQhSaY/quAnNBl9lbLXMav3Ar4tJF0erLbQaG2K7l50EVu6q3cVYBWu6RttZW0OZC+pBx3Qnm4lnYkVz940IMeNPxBS2dTh/L/3u/93uLMmTP7TsMjpy2bXukncGyHdm791CVylZ92Tdqcup5kmu6AN2g9nc2tW+1kwp/4xCcGaTqmQ85bB6xTpXNWXucyZCP8BuoMLJYU8hYbp+QTQ/TgeOvslw4+nWQ/qMO51cXrqZxEBlP0JJie6jpnQFSaWr/IgTnf1gFLB7Xcz6SM/5IdWofKJDxOffXUVs/o6rNO2u2gEHltu1anGJp1ysrBOXN4UsdaTsKw9jaJ8K0/d0MeviXfIT12SZgJZ2KgLwn7s0bnkhYebcoBKzdOdI6eaMglJ3zrtufcco4jXXfAG7SKGQVnl45DhA6U9+x1bt+EM8PUYXVUPOh1snRSMy95ZKHNoNlApUE+/pQDGxQf/ehHB0cbmZn1OJ6Qzpwxff1ZcEBkuMyYMuPCQ/c5gDeDMvTqbOarruqpLHQVEm95K41w8jkPPHZy0P8gQDd/PmxG3kGBHdmvbdd164g+tnN3k/4UOcv0DE3sBUtLXbUp/Xzt+p73vOfwkNafNvA15avOLls5Gc++YYCXDLbWF9TPlXbVpuSlvIFpxQ/a9CvyyWO7jJEV7DuR3R3wGs2ow7h0NAdd65xJ02lyC6aTesIccLtn8ATCYwM8J2kG4tM/B3EknJsBRA+b7L1uam3vjW98Y4odBpG33p72tKctvvRLv3RYsqAXfQyAdHxhL234agKdbnzjGw913hM0EcBXgVxANzMss3x6OR4RbcoNjg5VxlTYBybZVTtwJmSkvCke6cqoeuLxAoI/Ks73Zje72TL2WXnspi+wtfqyYS13HV39Cbqz8gdhu2A+jrpKEY4t9oBjW7No8vQR+vnTcfj+JZdcMvSfyPVAjtM/77zzBt44yuSTyf5pVx/kvNGNbjRkz61fK1N5l19++d6dWMraZTxvWrPLFphZNx1Yh0mnqZ2biKQL1wG+LF5lCh/koo8ODJNjdv2e97xn34lnZuXWfjnUWhc6gpT/mdhn4uikB4dmCoc38sIX3PKFXnp4WpoaJ6fSDZHP/lS6ZeHKU2VN6bhM1lReZMEJ13Kn+Jalhz/yVtHKT/3gypew9Hvd617Dge3+yAJ2zeThnD7lzzkXmpQdOSkr/JUmtC0mr0Ly/WGRexrgdNRyCy2pE+aqnS1heVPQ5iUeeYlP8c9N16HJsuxhBpN9v+E30/7u7/7uIUrvdPLUQcaULu1gicwWV1nyolPkwvVC0/JIm4Iqp9KsIyN8razolfyD4MiCY4N15UW/lm9uW4QvulQ9qr30g8c+9rGLm9/85mEZZsNvetOb9pavUmb6TGTCkVX1TdqewJFAlSU7MpWV8kbYdiqpO+CZzalDpVPpKG3nkZcOGBzR4avxpMHbuMhWLlnW0q644orhzaaU6fb6rne963DLzkGbLauD5YFA+MWjX/LQrtIztBWHr6ZFTmwoHpslbxmOnuEZkz3FP0VLFn2i7xT/nHRlhE449RQGyVuGQzcwND/L+JJXWaTRIUsh8mpY3BeU3RlJD7zuda8b7qLiuKu9azlJlxaQFpopjDZ0wik77SBt16E74DVbOJ0pbBmw4umA6UihGcNVTvjG6Oam1UFgDfLlL3/5vlnE9a53vcV3fMd3XE1HzjhQ9RBO3Wp6aJdh9LlCl3hk1nThdcqIrNaJROa6OPLW5Zuir/JSr+Apnql0fHFIre2meGp6W664y9pvBQ9jH/jABw5OOOn60fvf//5ht0TS0s8Sj7xajjBd50DlC31bRtJ3Ec+z0i7WfM066RTpGGOdhrikhy5FJD3xiuXlqumbhpVt4Dj1LGD2a4bjqXbVrYbRVj0T3oZu25CRulQ9l+lf6Y8qHBtuUv4y3mV5Y2VVemGXP96azmE6E7o+PDbzdVawvcFo5zrVMR3WSat6rcN3Emm7A16z1Qz6XC1rHEJw8sfWs3JbBycc+incyg2ddDI8Mfcuv1tH27QyE/faqSMIzXpq565hsqr86LSOftEnssibulqayrsqXGVW2qTXtFXh8MCp8yqeVflVZsKreGp+5YletR2kzYXICo4c/cMyFLCVTLolCPvD3S0FvJrsYS4a/aX2mcgMDg8sbV2gA4A34V+3vONA3x3wMWiFOZ0Njc4Ph76GdVoDyuzXJ8cDtpn5GCMHvC6knG3wbSprrOxtyiJ/2/LGdD5uaXG+cPqVP+yHPOQhw4Ht0defuq1hDnPqsH0LdAe8fZueE4kZJIRnFgInzIl4A86BO/Z5Js/e0Uc+8pFbecHgnFSsCz0SC9R+U5cW7nSnOw0O2F53oF+5o/IJqw7bt0B3wNu36TmTGKeqAAOj3qqJm6UYKHZByLPkYE3voQ996KGt352zynfBW7WA/gL0KQ44DtkbbV/5lV+5b0ua78ZZisir9ltV5JQL6w74BHaAOniivreSnO3gDTHAAXs76X73u9++Nb3Qd9wtwAKZ/QbrN/e///33OWB/6O985zuHpYhute1aoDvg7drznEqL400hmbXAZr+XXnrp3sMStGa/3/iN37j3MC58HXcLsIA+ou/kihP2wo5dM14ZD3gQZxbsj77D9izQHfD2bHnokjKAYGu/zn0Qdnnt+I53vONWzjY49Ir1As+5BfSRQBywOCfsYZwti3VLml0Tdtj0h3Gx2nZwd8DbseOhSDFQ4mAVKA6y9vtnf/ZnQxyNg1Gs/dYXLYbM/tMtcNYC1QHHINLijG9zm9ssbnnLW+4dwiTv7W9/+3CXFfqOD26B7oDXtGGc3hhbOnVL08bx1rQaHpNb01IGnLB139e+9rV7cVvPbnGLWwyzmMq7Khx5oVtHr/DAlS9hOOHQJh6c9Ll4jG8s7SDy5vIuozuITpFbZdRw8qdwaFvctrV4aMjyqaoLL7xw8WVf9mV7+8dtcbTM9clPfnLfUgS+you/jUtbBeEJXkW/C/ndAW/YiukkOm46c9bQ5ojEX685PGhSbugde2h9Lt97k24jvTNeHYW4LWjLXSU3dQtd+Nv05K+DIyt2X4d3jDY6BY/RrJMWOcHr8KLFVyHx4Jo3Jxy+ilfZzjKEB7hmwvo1XifsveMd7xgu/NIic44eq2gia9tyV5V7lPndAR/Q+nM6SzrWAYsa2FtZ3tV3apXzYgPO+n3wgx88DI5VAy08cJWdMJxwpV0nvIx/Wd5UGVN/dJvIwrMJ35Ru0slj93Vsv0xezduWrq2csbg95NaCc84vGn/273rXu4Y6Vp5t1nWbsqrtjmO4O+AttErtiGPizkWHItMTaWe2WpsLuHX0BNvB4qv0Cs9Jw9u2Z+QFH9Qe25JzUD0Oyu/5gc8SWQsG6uUDApywTzeJ5zpoWZEfvCs2XGWX/UciraLu+UOHs1fSu/EgHUVaZqHCnKMXIRz3KJxXP2NCfNJgsvBOzezwkCUfj0sZH//4x4fbQV90CJit3P3udx9oyMXj4oyjc2inMHq0ylAW3VLPKR7p1eFXejZwkQe7lW0hvJWvpUkcDVkwPdfhDW2VpX3IUtcx3UI7F9s3Sx655LmdT7mwPNcqQJu+o76Rt4pPvjZMGcHS0w50pBvcgnLxA2cE3+pWtxrusnwlBbjrshZshkwe+bDXltU15QUPTCM/rU3U0XWaoDvgDVrboPcgIp0tHc3nWYBO7WOR3irivNCbTaBLpxPWYWGfEsr32KbUwa88snVSzuLiiy8eBoYw4KRvcIMbDLeMykfvs0no8WcATZWRdDoqh1y8qWvyp3DqJj82ESaD84DzFWPpAXyxTeVLfsVoyaEfWrrFtqt4I6fVk26AQ/IB0oMCOXQxW6ztmnLn6kkPbZA/Q590sh1sDsQBKyv2xUeWNHpxmnG0VSZ66foP8IFOBzql/5j9+oTRne98570+knbNmMA3p57Kin7ota34aYHugDdoaYOCY03HiQidG+hInGvoDHCdvaXX2YA8tMs6XvLjeAwGt4Lefgv4xtftb3/74RtpVRdy8XEMc0EdcpG1TLdVMsmhP6yuBx1k5AA60Y3MpK3SZSy/2la7HaSu5JMHtDu7R17wOrqije3Ii+yhgCU/U2Wl/8BkjTlgYjlSu2nIcZiTXTX5YrQ/AQf0mAmfOXNmsH36V5W3qp7yyY+u+kV4gpdUcSeyugPeoBnNbH0cMR+0JEKHyWfDdV5vEcGcg8ssTUdLxwq9Ae+VYWf21s7bqmV2C9JJvZXkQ5JxtPLMVKz/2v1AnvLo4AJezlgG0Q8mV33o5M0o+q8CfCB1DL2BbreGPw0Hf4/tzpjijYwWm6ka9P50tIcy23JbHvGUU/PM3nxC3jfRyDsouBPyB13bVbmu6Bi8rCz02sFMWh/yIU32m8Or3UIXrCztQJ66kpd+VfVQLv70G2VaC37LW96yNwPX92xL81l7s2k06osv9azlVvk1jBbA+ok7N3U9LdAd8AYtrdPqbPkqMhE6mw4NdNzqZMxcOOsKHKmBYFbKWXp4lg5f6cbCyrL+682kABkevJ05OyO5xjWusVe+jp1ObpAsgwwY9AYVp5S65nSsyJojJzQGFBuQyWbq2sIcueFhOwfEgLSDsHRy1pGlzga+umojuq3Dr9wWtCvHmXbQruvIrO3gD9olTfv54wdz5UVWdNQW/nC0J1ntH+uYXOVabjjv7BeSfbIeWPrhgAGnq10tU4AxGUPGih/tlwN/NpWxoohjl913QWzQJLVz1PAGojZi8cably/qeqWHbwaJgdUOunUKWVafZXlzysAfGcFz+Foa9auykn8QmZFxHPE269XKauNj9UfjtXaz3YCJgwlADn86SJ+LzNOIuwM+ga3+3ve+d1j/za2aAeK9/bvd7W5Xc75zBtgJNEFX+ZAtYG85B+yOI+DlH3vQ43xzB5L8jldboDvg1TY6VhRulz1844QDbiUNjrphvjveWKfjbVjAUkXOh4g8uzI44eyOiCNOfserLdAd8GobHSsKe345Xw96OFmXfZr3ute99mYiBkIfDMeq2U60Mma2gAP2R2/NF3gI6qHZBz/4waG/zX2GMTD3n8EC3QGfoI7Aqdr9kHU3zteDI9uE7nCHOww1qTPfGj5B1eyqHjMLcMAuH+20FmxXTMDDYIe165u9v8Uq83F3wPNtdeSUnqx77dgM2CzEZesZ52vXRTsA2viRV6ArcCItkF0c/uztMzcTDliG8DDOzo8O61ugO+D1bXYkHGYYdj3Y/K6zZ4nBcYF3vetd981AON7ufI+kmXauUP0sSxAq51yIugwhzbKYfpk+Ka3DPAusdMBjRh1Lm1fcdqiUb/0JzhXJtbMkrWIPsfDAaBOvNMcxrL5mv2Yb9Ke7vZeWHwyKrMvJ69AtsC0LZDz5Q9e3vGBkFpz9yMrxwo67stBuq+zTIGelA26NoBHiAA7L4Cmv6mILVtVFGNBpTK+k4RPm0OA44ir7KMOpR6uDzfO2/HDA0d1BKd588/KAp9R4M1Ba/h7vFtjEAvljx5uwfscJB+xL1y/10Q7rWWClA25vZcWTxoFNOQxqyMvVqjWV3tIlzlGClGf2Rw+dwiUsL09iQxeezHSznuX1VW95WdcKz1DAEf7QObaN/rDLe/jOfch5Duhue9vbDq+IHqHKvehTYIGMs/RN50J47pC4Sc2VV145nMx3Csyx1SqudMBjpaVBOLA0QujiMGBOL46vpguDljcyKg6fssLH8bscCuLddnJCJz3/1KGXHycbB1zLOC7h2CN60ythXyKw3Sfg1WNLD2fOvnrcoVvgXFlgrE9ahrAOnFfb0dgWqY92WM8CK8+CiAMg1hNPh4yYPZqBcgIVNESdFXOE+HOFFl2cZNKmMF633OhdeOlgNmg7ltset+G2yHgXnYPFg47zTzn+pW0a57QjK2V6iyzvsSftKDH9K6hv++qx20Az4PZd/srXw90C27RA7ZfOHfHq++/93u8N48lZGrZIOu+j9Qvb1GHXZK10wBwXh8X4L3nJS4YHQf4BOQBP4BnbbbwzCBw+Im62io8DFI4TjGNkxBpeZlSOlFMlQ9jBJK9+9asXP/mTP7l3O67RH//4xy+e/OQnD46U7MojzlE96UlPGk4QcyqYtMzOn/GMZywe/vCHL1Pj0PLo5QLq63LoiSUI29ACbF+3AyW9426BbVtAH6wg7mAeyxCvf/3rh3Ek3wlpZsH3ve99K3kPL7HASgfMuQKzUA7A13dhszLO0D5Ur8D6RzQjO//88/eOCOSUHXsHt8sVbaNO6chJKsuMG1xxxRWLH/7hHx7KfelLXzo4qJ/6qZ9a/PiP//jwp/CYxzxmcGAcNl6OmO5mvvT+hm/4hsVXfdVXDZvJpXN2/kyOE7TLKI4BrMsPbNGXH45Ti+2uLsaHsZqJQcatce9BnAkXPyDdLNhpaRdccMEQ312rbK9mKx1wnAEDf/u3f/viwgsvHPaj2nbytre9bbitdzbBG9/4xsFJcmrelDFL5iScI+rDfl6XNUOukMasaW2YA+XEOVT7DV/84hcPyw4/93M/N5xvaznkB37gBxa/9mu/tvjZn/3ZxUMf+tC9YyE5fWD2a/lBR7nPfe6zuP/977/3tQpO2h/EcYI6A3Yn4btv3jgK+MMw+z1ueke/jnfDAnG+amOs1vEq78zZ5w+W/8yCjU/rwHzCE5/4xOGueDescG5rsdIBx+gcoStvwvjqLuCgn/WsZy1++Zd/efHABz5w71/QSwNum1/2spctHvKQhyy+93u/d7hlQQ8id4is+OEkgQduGttbXw94wAOGma1067f3vve9F695zWv2bt/xxAHrLNavOTPO3JIFPeRzzup1UIjTHJOzTl3xh55MT5fZMbsf5Ptjc01B+Kfyz3W68pfZ46Dln2v5B9VvV/jH+lFt15vc5CaLO93pTsM6sPQ8m/FSRvuyxq7YZNv1WOmAGbY2hLh/uwDnZSbmlsQM2dokcLv/sY99bPHmN795cJi+VUbOus4uDa5MDtTJS/51yeJAOSbY8sfLX/7yYZbcnvSP1mZxsp7//OcPyyj+SO55z3surne96w2z4fowK2Wmjqm/9PZCw5nHoYu3/GN1JjNy8QSk0YU8cuz9zcHXaMhiY0s++WMakx9562BlKzM4vGN6Jm8KtzLQxXaVZ4yu5rdhdqn6VFvX9JZvWXxTvmUy5cWWq+jWzV9HX7TVRlNlrUNHhqXJrAMbb9kDbOLlGY08uyRaG9R4whVP6ber6SsdcHW2jJDG5wwZjrPgCOKEYihLA/YLugBaF6iNHXlDxsiPfJcy8g/L2QY4X2XHCXH6djW0elsWufDs8onXeHWQ5zznOcPZpt///d+/sG6cN3vI8ufBsWeNWPnkmYH7gKbOZ+3LH0/qhcYaGJCGlk5o80WMyImt4kATT51q3EONdvmBA46Tjm3wRBd6uyrE9jWtDaNRzyqTXG2tLnNk4AWVFr+6SoPFAVppwcpW3jJAQ0ZsJBzZq3irXHKA8qutolulXTdMD3Lh2K7KkDcH2IVuqa8webHXMhloQOwrLC2y4MgKXWhqXPe5SaQAAEAASURBVHgMIsvd6Hmf/VIGeSY6liM98BZP+XDaN7qxj7QaD/1YmbuYttIBp9KtYTJjlM6Q4unUeHQWho1xg+XhATVtSBj5CW0aj8Pl1FKujgSiTx1MaFLGwx72sMWDHvSgwZFw5K94xSsWP/ETP7F45jOfuXArZZkEKMdSxkUXXbTva8BkKYszVd+AdJfOZo0Zv054v/vdb1ijtjODwwwdPjqREd38afhjkJYyyHPUn6095AbcSZBlXZhTzEWWesHkuSI/GN8qQBNbw+zplhJEzpSM5Kec4NDLt47PPsLywxP5LU94g+W71Jt+dINX8YW/lhceWLo/V1elCd86OPK0gT/PyAtOuctkRkalIc9OA3Iiq+bXcPLJSXlwbOXtNROGxENfZSwL4wOw1+E9fCPD5MUW0X//7//9cFcc+crWZvp4eKNXaOqYSN4yHXYhb7YDZlxGcQmnwWBO0cX4jGjQiqPlUCqtNAZPI6wyYqU14/TRRA6JIySL41WudV1lyldedCBfPoeGX5514Ec/+tGDg7N+7bYps1QyyTGD5+jR54qzTydCmwuPfHkuM2QPHcNLj4TxpDNKV0c88oG8OD7OKiDfAzhX7KJcIB5+WDyQdOUuA/mRg4dOIGnLeJOHL+UEy6vhKk96eKRXushscfRCG0ectkldW542njJTXvSQPldGKzPxtGXqmbJSv1Xyqy5kRkdhMsAqGcmnS3jIiSz5bBe6mj4UsOIHH7ketptg6If6rDR3fx7M25sPlJOyhQE6uklPWF7aNnoNxDv8s9IBc1wMxFA2XZt5WQJwLqgzCKS7XWe4GLfOUNmOMV1oY9gaXmVftBqXXA/cPJiik0bnOJVrpkiH887eDqVBI5czdRmk5ERPyxIcpZmFh3Rml+Crv/qrhyv8wda5zErpoXPBAWU7F/Xud7/70Cnf8IY3DFnqS0flV1Ana7tkkmUJJB0Qjz+NF7zgBXvLGnjtfLCFThmxY+riD8iHKvGpB3mhqeUuC6sDHhd57Ew+m+bPaxl/zVO/gD7kj8SMi809L6j56NbRFa8Bri3trvGHCqr+Q8KMH7LoddVVVw3PKqytHxS0qzsan/Fxi86G69aPDnhMNnwA092N5xVZKpsrr7WznQo+fEmOflL78Lr1Np4ybs6c3RGRg6LMru2S+tEf/dFBJF3p4TI2K0hLXYTNpPWX0wIrHXC9tbe24wWIGFLn4jx0fANWw5qdcYxxkGhBjCycNOFVgNZFD4v99hj++q//+tDYHBLH5sHcxRdfPOzCULay2oamo7Q4LJ3HINGxLUHkS714q65VP7LjzHXcWgYeOgbXvNiwyko5MFoYhM+tsL2/ebghz5Nlsw11aG1Y5UQ2nnUgZeOJjIrXkYUvEBm1njU/dHMx3siqcqr+68iKflXWXP4xula/deVWeuFcqfNYmVNpVRYaMnK1eVMyptLxk+Wu064cDhiYkPlD8+dhggPQtn026QPBZ2nQ5Ur6LuP9f0cjNeW4GI5RnvCEJyxe9KIXLb7zO79zceGFFw5rnGbE73rXu4Z1n0c84hHDzNh2tB/6oR8a9ubaF5j1TbJcgLw5gD7/iJykpQMN68ULTpQD/cVf/MVB7vd8z/cMcuV/67d+6+IHf/AHF/6NgRc4pJOFx4O45z73ucMDOzN6TlI9U98x3eSZzWZJo9LgzRV9a/5UmMyxMh096Q+NzAAH7A8uwIY1P+Un/6CYXgeF6ASnntE5eN0yIkv7byqjLZOcbclq69mWtU48ekUm3k30DA+bkeU6KHC+JiVm5sZQHdPGnS2jKTd61/hU+bWuUzS7kr5yBqyiDMvY1nvcnn/FV3zF4MgYym3gpZdeOtxycMbWUzlkb29xavi+9mu/dvHUpz51mMHNaYBqXPzkKEvncYv4lKc8ZfHTP/3Tg7O3JHH55Zcvvuu7vmt48EVXzvaVr3zlQO9prGWDb/u2bxscp85iVun22r+2XRC+JpzOAyunDu7Un2N1iXPEOh9Ih/HPH1p2Sb5wHLz6VB70AbYR5+D9cXHAAXp749BbcMqrgCd2FY4+kZ284Mrbhqt+8vBE3ip+5VVdIhtfeGtYfurS6hreFtOv8kQuulZ2yyuechIOT9W9yhyTsU5aZAXHPolPyar6pL54hOUlbRl/LSPlwiCyQhMcuim5Sa9ybDez88idpIeEwNKEu9K8L0B+ysxdaOrQ9rnEU9Yu45UOOMZgcFduwatRzHi9XcZxcFwcDgfHMboVMXPddK0p5SqPQ3OOg9mtr0C87nWvGxzi0572tOFtO2VoZFvffuRHfmQo15qjtS6zZLNK6706AKdrRun2iXOsID/1Tjq50pJHF+Gk60zS0snifMNf65E0GL0LoAHsaI0z29qk2fvsaTMaeoQvPGiSJlwhNME1r4ajR00TDl9rk5ZuKh7+yBJ3KS95LZ6SlfQpeunJC+0qHLu1DmEV39z8Wk88q3Ss7dDSpm5z2gJtlVX1jdzQRG7SK+1U2GTERITdjHFLgnHAJjkmZql7JiH8Q2CqDmmH0O0yXumAV1WesTg+jjnrPXjMTL0lo5EYOmc5RF46Rho+6WNYGekYsLVgXwF2GpM8uw3i/PBzqI973OMGB0kv4BbJ7bsOQ4aHSlnHHdNhWVrNE04nq+lDoRv+eKBo1l47K1u2D4iUt60yW1XPldy2nIPGT4qeB63nceSP7fV/s1+H8LzqVa8aVOVwPRT2IpbX/0Hoh0j/GSxwYAecf7E4oThWzs0FkiZcG6GG5U1BWwa6diae8jlk9NkClrLNVjnpeh5F8qbKPap0DzOydk0HT/l1cA89x+C41mNM1562OxYwzoxhEwV3kiYIxhjnCxyAZRbsmADpJmNwh7+0wMqHcH9JOh2KA4A1SByreJuWvGlpV8+JnOTUeMIw0CmSFnq4poW2Ta/0Rxn29lteVmAvSyVnzpwZ6lb1Sj02sWmV08PdAgexQByxZ0QO3wp4+P66s8uEHHT6aHBoTjveigNm1Bg2js5MtIYTh0Ho5zRAlR9+OJf8ON7Ik5Zb+NDJo9M6ZUfeYWFr5y47NWI/Sy32lAaSfpzrEV073m0LpA/qk2bB1QF7lqEv28cM0HTYb4GtOOA4BKI1SL2m0varsTxWGy6yOdzITn46AywtcbShl84hJ1966JZrcTi5dj94XTd1op+Hb15eSH2Dj5Peh2OdXspxsoA+mnFkTF3/+tcfHoZHR3n2s3spKXTJ6/gzFjiwA45hqzOoadXBocmFxjUH8EQm+siIbHEQeaFN+hh98gbGY/KjE9vCxwFnt4WHh+edfROt3akRlVPnxDvuFjgsC2Rc6bfGogfb7tQ44oAXtGxJtS5s/bf311jmM/jADriKi1MLrnltOI3Xpk/FIzM4dG28po+VMZYWnqPGHrx5FbO+/ebYTNvoasetdRCueUddh17+6bEApwvigPVFyxC2iAa8pfrWt751oJGPtsNfWuDADphR58BcujmypmhSRvAU3UHSx2QnrcUpJ+mJw9Jy6cjWy7w/b+uOrXLSzIKt/+bp8picVlaNC28K0a3iubIqTxsmQ1pw8oeEmT8Z+JFT2cbSan4bruXXcEu3TrzqsA2ZZKhzZFX5q/SqPGNh/DV9lbya7xmLma0JgLD+at+9XQ8B6c6fyGvKabvkV5x6Bde8XQ0f2AHvqmHG6qWjZbZZO0nSgsd4x9IqvZmBPZNevki6Wzl7l691rWvtrWGPyZEWnqrXFO3cdDJzzeVZRhcdK81YWs1fFm55D1r3g/JH11avpG+Co9O2ZJKTaxN9Ko8+y8HCLrrqq3bt1CUzRwD4sABIfaqcGt6WblXmcQ53B7xh64wNiLE04qfSU3Q67yWXXDK8wimOx/KDh2+ZNUibulJOzY/84Jo3Fa60dbBM0df08MI1PeHki4em4oRDvwwPAs7+RFaNz+Fr6cWX8a2TF9lVZk2r6WNyK638wBjtVFp44JYmeTU9aRXX/DZc6RJGY53XkplnFwEO2DrwOkDWaYDugNdo5eqQsNVOWcWs6jzhqzxekXZZiqgO2BkQYI5MdHN1RLsMomPwMto2LzyrdMZXaefQq1942nLZbRVM8a7iO2h+yg1eR17lEZ4LlS88tX9UWaGtODxjuPKSmXZB66jLug6sTzuWwMPlCimrpkVWTdvlcHfAa7Ru7XRhSxqczp200CzDoXWWsI3rkeFNP+/We6gxB9JxIw9PdIrMVXLQVX70kbuKdyx/rFzyc7Xyx+hbuamT9DFdW/p14q28dXhb2lrH5G1qy6rXHBtNlRedqrzQwlPplWYsnHpFN0cF1P3Alim8WMQJzwF6RNYc+pNM098LXLP1aied6iRtehtXpLSkw9bI7JkMOGfZ68e29oDQJn8MV91anrn8oQuu5Yyl1fyEx+hqWg23PGN5oYEPWscqK+G2zDYeuk0wWa28Nt7KXVbH0K6SEToYLZlwrqRXLAzWlW2JzAO4lON1f3vXOWJfxwCWIexxv/DCC4e4n7actt57hDsc6DPgmY2bzpFO1rIlv00Xl9fm1zRvvTn9zJad0N3hDnfYN/sN/TI8VnZNW8Zb88IjDVRc6dpwy5d4ZFQ5Na+GW5ltPLRjslraqXhkBEeW+BTPuumRXWUmbR1Z4QmOrqtkVPrQJg2uaTUcmqStwln2qc7UAzh3bj4eENDHnQ9cgewWxtJaml2K9xnwzNbUwdLJbBNzS5WN5emkttuAnAQlP0f2OZgozjv0ed3493//94fXNXOIiRmFGTCwbuYWzrWscyrHecTo7CcmO/oOgs7+LONHIz/1VMeUqV71qXbktTjlteWQY0O+weqAFie9xRbhIStprdw2Tn500w5sK60tt+VLvJaJx+wMsF90C+0mOO1qScn6Z8oLnqunsvUJesHaNbpG1pR+yW/LSjvAXhE2c20Bb8s3RqMNTBq0K8x2AC/5zgg26wX6kyUIWy2tEeOJ89bf8YiTCVaVPxDtwE93wDMbUafMZVBlu5iOks7iEGqgE3Fa1nENRp0vx3HqZOKcM8Bro3p9+QKt7TzoOGCDz5UOOzCO/EQPt33kZSDBIPkjrPuSwidRmeoaGfsIm0jlq1kpV76BabCGtspNWuUdC5OH1kU3A5iec/jRBKJX5GlXTqnShHYdHHlph/BGbspN+hyMlzxOHUTWFG/ya1nRCw+nzpmHbkrOnHQy/LGmz2kLtrQMAVKGMn1Xsj6go5M/AThtWHkGATv80x3wzMbVQXKZDToYHk4anBmwDmX3QmbAZqeOlNTBgI7POVvz1Smd/8sxBaz9un2zD1jnJRtv+ENXsTIMAs7fuczKS8cPHTnLQD4eF+dPP87NWt42ZsDqSi9fUAApLzqlrolPYXz+mNjDudPbmAFzbtbb/fG1dpvSYyqd3TggdvNnGnnBq9qB3NjGH68/LFi7zn0mMFVW/gDJsVa76QyYjvoaeWPtqm18REAfNi7og/4DH/jA8GFZ/UnfAnUGnDu+OTYamE/4T3fAazRgOgXnycEaXEnT4XLqE8fraL4AZ5azkaUZTDo+R+y2UqfUiQPefrMDQhlTAym0wQYCuQa+WzxfKAhvaKJr4i2Wj8dFHoeuLqlrS9/GU15bjj8Hg5BunG8OFqr8U7yVJmHyOTl2rbq15Ya+xSlLOh6DngPWnnSr+S3vnLi2pBtnri3i5CJ3lZ7JR08Wx+VOiixnQkfOMl1CE1mwNO2g33HmnGPtl5GHLnxJG8PqONaueI0Hup9//vmLV7ziFYOTRe9Zh/Mi0Ohb0ZN8PMaD+p4W6A/httTStSO1IvNPn/T84+uEV539ZJNbN50v4OsXHmJUmcKrrvBP4VX8yQ+/OKg4NGO45Us8OHISH8NjcmtaeMZkVbpl4cgIjqxlPNvIq+XNlVd5hOfqWvlSVtLm4PDMxVVmeNwlcsCJc8DWgTltfV96C0kLbvN3Ld4d8IYtqgO115QodBUqnw5ZZ786rbMf3B5WvsozFa5lCB+EP2VUmUmbwqGt5VY9kh4c+pZmSn7lSzg4MpbxJq+Wm7ABH1mhOwiO3CpjLK3mJxy64LH0pE3h8MJogkM/JEz8hGYVDnvkJx7sbscyhDtGYGbrruqNb3zjoFPkh/60ON3UF+4OuFpjRTgdBE64skx1xLH0pLUO2AdFzX7b27NaztwwHdtOvow3OlWadfin+GKr2C0YfZU/Vn6VKZw6JVzz5/Cjb+miQ/SsMjcJT8lJOevKrPJa3ZfJSnnhIadey3jn5EV+1Q9fyrOk4yyTLDmgsxxlz3toxspBtyx/jOekpnUHfICW01EsJ7QdcJVI9JYcrOt9+MMfHtZGw+P8Bw+WWpltPPTBU/nSc4V2Dg5P8Bye0IQHXgWVdg69gRmeVvYc/jHeOXxtWcviY84j5QYv42/zKs86uoav8lTdanqlTbjVo41XulZWaD10vcc97jFMKMyAPacwAw5v5cNT9YuMXcbdAW/Yum3HSYcaE1dpOV5OG/74xz9+tfVfJ0lxwKDtjCljLm51mcMXHrS1/HV4yRijr7Jbmpo3xtumVfqEW5ktj3gLNW2MftO0lBP+xIOTPoVDV/EU7Vj6Mj55LU+lT7ilaeOhG8OhtQxxt7vdbXjYp8+7PHT24LMFPIEaTtou4u6A12zVdIzWORGTvOAx0fhcnLDD182CA27Z7J20Dqyj1jJCswpvwjMlUz1yTdGsk77MLuvICe025ZG1LdsdVK8xPQ4qs9psW7KqzIQrVo4Z8B3veMdhBpw8M+H2rbjkRbfgpO8q7g54zZYdGxxJa3FEJ13cliTO1fYfB/DUB3DWy6z/cs466TpOWBkpp4ajw7o4MiqeK6PytGEypAUnf0iY+cMu4W9ZIrtNn4qn/IqnaOemVx0idy7vGB0Z6Qvrygv9FFZezRsrfyoNHwh/S8eJ6uc3utGNhu1zobUM8drXvnavH1S+0+J4U+fugGOJQ8QcrJnv+9///n3rv7af2eepQ2fv6CGq1YvqFjgnFnBnZzdEwJ9JzgeOE09eHHCbnvxdw90BH0GL6mQf+tCHhpcJMpujRhyw/NPWEY+gGXqRh2QBb915ucjEA+jz+r+XX2r/53TFa/8/JBWPrJjugI/I9L5+Udd/7ZX0AM5DCx0wnfWI1OvFdgtszQJmwM5/qHd1nK114OqALbudNugO+Iha3AM4bwYFvHzh9WGdNM5XODPh0HXcLXDSLGAd2PMNr40HON6LL754bx04Sw6ZAYdu13F3wEfQwjrZpZdeum8G7PAds19QnW4NH4Gqvchuga1YwCzYa8kBDtd+4Mx6xYXh09TnuwNOjzhE7EhBZ0B4GhzQOetBK5x0ZgWh6bhb4KRawOlnZsEBfdvZwLkLdNenz4Pg0O4y7g74CFrX+b/peIrX+TyAs//3uMBJcf4nRc/j0q7b0mNdu3sQd/e7331f8bZg2oqZma9Ms9+6VryPYQcj/TjKNRtVB9H52ivpc8S9/e1v33vaS46Zr+Mr6/nCuQ2bOxsgB0/4qh7yQHDNa8PL+Ffpsqr86FD1qOHktzrVeNbHwxccGvE2LXlwrV8bjuxV9azy5oSVE50STnyKP7q19RGXt0rHlBP5kZd45EaPYPk1HPoWt/LCN5YuzzqwrWjOOKl3fm94wxsW97rXvQYnbJlCnnOUT4sT7g647Vkj8XRWWemcBqvO5kp+BnArok1/xzveMbyAEVm3utWthtPPWrpWzrJ4eMmMXqHPoAhO+jIceXiEW5nLeGs5rT6RF/5KK62Nh67i6BbZycM7hz/0cHjI4tQSrzQHCUef4FrmMrn0CW21f+TEBstkhBZNbDUmN+VEVuVL2hhu6Wq8hiPfLNi3Ds16A9aBn/rUp+4d+M9R1xlx6HYV9yWIGS2rM6XD1/AUa2jH8g0ASxBOhQrYopMHcEnbBNMt1yb8Lc+yerS0U/HoE7vBB4XIot+25EXmQXXDH70Oolt44bFrXT0jL7tsttG2dCA39V2mk37vteT73Oc++8h8D5HDjT6w+GmB7oBntnRmDnAurEmv4Tat3i6+733vGzpbpbnlLW+59824mepcjSw6wcqr8q9GPJFQ+Q4ir/KOhRXfpk+oNJqMNzYVDlT9k7YKH0SPKdmtzKrjFE9NR1/rF3mp3ybyqgxyIquWu0mYnNZhpqzIE7e0YPnBckOFT3ziE8NLGWg8F/EHcZpmwH0JovaGGWH/+L7RxZGC2tk81QVesLDP1+2UBw1mu8LgpS996b7zH6T5PMwHP/jBvVmAtHWh6vHRj3508bGPfWwQIT2zC+E5oI6RB6trZlCr+MOLrpaXsM821Q9fSg8PPUO3rByDHi3sde6El/G0eSlTeSnTZ44sD82tayszcXqRkXZIWXPrFznB0c/OGRdYpWPyowue1BX2MdP6sVVp4UE7ByIPH0eadlUmZ2tNl1MVNybQja3t/uqv/urikY985DBG6rfi5uhw0mm6A16zBdPp0mET18kCBpp/8fyTy0MnfsUVVwwdUmeU5gB2zrnyR866ODpFVgaUclPeKpmRAasHaOVNyahl4HdVSH4tI2noYqfK04YjF58rNm7LavkSxwMiJ+HoEd0Gog1/oov6RE+i1mmHtmhyYp8qs6VLHA2gS9pPOG0q7EJXcfjnYHwgMhKXJpxy1RvAHjjf+MY3Ho5iHRLP/tgT/zVf8zWDc85YqLJCt4u4O+ANWtWM1ccbPVSoHSVPd/37e9gA5PsYpX92//5myRlI8n0BmRP2EU58m4IPLf7xH//xUJYT1eqeYjKrnsvKyOA2yzfbopMvGrR1HZNRB33ypZkFOf/VBxeddVy/9lzphFfpSR77mZW7s/AA00wLZMAPkSU/qWN4PvWpTy3+4A/+YLCZ9jgosJtP73A0Psyp3WMbslfVEU3otWs+fFnbdZWMzLZDF3n6iJmq/uHNNP1SXuhSdo1LawEP+5OnXb/oi75ocYMb3GCQpR3kRwfjgg30A+EHPOABi4suumhP5B/+4R8Ox7DSyVnB+aPaI9jhwOYjfoeNsqxqOqbOpUNl2xh6nS7LDMmvctBzag5hR6tz6mgcL4dOlmtTMBiU6+I0M7DWlZf6kUFHF1niZK8DGcRwnFDsRlbKCp4jm+2qbuSJbwrKrvyb2q2Wz2bqRzd9IvFKsyxMJ0AGfcghg56b6Bd5ZEYWTBb9Yv9gdKsArSvyolvLhwag43ydeeJ84AqWanyFmwMOXfgq3S6G+0O4NVvVoMiFVTh4qtNwGhzku971rqETmhGbtRlUHsBxwJEzCDuiH/pHj2CqTNVrmZpjPFVmeMfokjeG2azyVJnSa94Y/1jaJjxjcto0up0L2evInEs7ly51bOvW8ieOTpuZ/XK+HPW9733viBmwiYhnJniMlYyNfUQ7GukO+JAaVuey6Zwj1slc1772tYfDqnXM6kgOqlI6/0HlbMp/rssnf5v22rSenW+eBdIfOGIz3Jve9KbDklbltg7MScsPfc3f1XB3wIfUshzue97znr31LZ3svPPO27tFPSQ1ejHdAoduAX+WLmMAHnst2duh7gzNkNGdFifcHfAhdsec/qRzWQ9zOMm2Z7+HWJ1eVLfALAtwuma2AeELLrgg0QF7IYMDRntanK+Kdwe8rxucm4gO5cmzp+3CmQn4AKf1rtPU4c6NhbvU42wBTtXyAww44Lvc5S57cWn2JPtKhiW60wTdAR9Ca3OwWX5IcdJue9vbDrdj6ZjJ67hbYNcsUPs4Z9y+kszx2lpoFnyaoDvgQ2htzvbd7373MPNNcfZN2tdpK1CHboFdtcDY3R0HbH909g2rOzp7gPPG3K7ao61Xd8CtRc5B3JLDm970pr2lBh3Q+q/DScwM6uzgHBTfRXYLHJkF9O0xJyzd8ZS17zslrX4n8ciUPsSCuwPeorF1Jld94EC8NGcMpCPCt7/97Yf0bNTfohpdVLfAsbJAnGzFwhywyUjAuR4ccLZqJn2X8V/WfpdreUh1i4OtxUnLa7jJhz2EsB1HB3Slc1beHu4W2EUL6OsuX4Gp/d7r2w6R4oQzVnax/rVO3QFXaxwwrDPpOJYcArabebhQ0+R5AGcHhPQ6Cwhfx90Cu2yBOGCTECDujbgPf/jDex8rOA1OuDvgLfby/JvXjqNTeQBXwRtwDkKx/MAB4wtvpevhboFdtYD+7uyHm93sZnt3gcZKliF2td5tvboDbi1ygHicaHXAHOxll122T6pP0PvnR482fPuIeqRbYMctoO/7UKdnJu4Cxe2EOE17gbsD3lInn3KiHLCDRipYfjD7DfQ14Fii49NmgfPPP39wwMaPy7GgjuA0bk4DdAd8DlvZP7rOdNVnv2KQojyA8wqyfDDlvEPfcbfALlrAxMNkJHvhxR1L6a1RB/OcBugOeEutHGdaxfkX53w54QrOAM45rDW9h7sFTpMFOFxrwF5Kygw4yxA+5XUaoDvgNVo5TjYYa8I6kA4FhDlfOyAcMtKCr2mg8dABHRmR09IedrzVo+rW5q2rW2QFr8sferY7qIzIOle41U/8OEH0CV5Xt/DBCUdGG096xWis/Xog7Uxsp6CFz/cRzYQTr3y7Ft78UwK7ZomZ9dEp3B7Z25tZLIegA336058epHCsbqNg55xW8Aomfh8xtN+Rk/7zP//zwSEfpMORSZ4y/+Iv/mLvQQbd1nVY6Mkji04OSlE/6asgZaFLffzJGFCwr4KwTSD0+fMKT/JbLB8Pu4HaDi3tVDz1SFl0A+xXdZviX5Vu9ka2/uCBUurWlrtKjnz8tZ9olzlQy4rNYPbXDu7KfE6ovjQUujny0eofY+0qbw7QEe2ZM2cWb3nLW4ZzIMQ/8pGPjG5Fq3UiP/E5ZR1Xmu6A12gZDe4yIHReoDPrxPb0+nYX4Bz+6I/+aMhrZ8BuufDadG4dGPgar0FL1kGBfr7R5QLRWcdeNTDQoqnYIKPvKt7ojTcwxqPerhZSZpu+LI6HzcMbvIxHHjpQ9ZOmDTwE2gaQp11dIGUK13LFxwB96MIbeUkf40taeMTRV3nSOM788YhXaGlr3lgY/Vi7rtIzf0y+OZhzgMk3dvInNlaPMR1Oalp3wBu0nJnvNa5xjcHxpnNL+4Iv+IKho+tYDtrRwa0BV/DU1x5gzjczG3zkHcQBc/pmNRymP4Nsc6tlrxoQaHV4l5mWGZy60K/eIlaZNdwOluSplwGlvvTyCaYWwrtKR3TkcUbq6hbWQ5zwBbfyE0854sLo6ZY/xGte85oh3RizG/uxm3aOo6kC5+rZtit50bvKa8OpZ1uOuuqX2sFZJJkBh26Kb0w+++tz+cRWbdfIa/lqnF2U52zgn/mZn9lrQzLdIbKhfgfIi27iNSx+UqE74DVaLp3KIHCSU3uWL8eqY+g0N7rRjYZbKV98rXC3u91tcebsLRca//QGvocQZB4EDCqd1mDw5eHrXOc6V+uk0X+qnHRydXCraiDQM3Wd4qvpeMlJWeIGklk0/IVf+IWLG97whgOLvArhrWljYQ6Y3WC6cSQgZY7x1LRaLh5LLOSR46vINb/yzQ17m8vSgT8HV5xc5M7VU3kcZpartKuXFyJnmT6hUZYrcXcM/gj9ObBddiAskzWVl3Z1zrV29WwDzK1fdDJuLM3RLbyOb73vfe87jDEyQysMap0+k3Iyf7sDXqPddIJcYUvHSIdIBzJz8TChBZ+rNyA5NjOIzI4ip6WfGyenyhqTN5ZW5df8zE6kJVxpl4XxtLKiX/AUf+Wbook+bTlzeFuZVYYw2QeFyKy4ypS+DlQ56+oX3pSXeMXJWxdvqx3cPZ539vNc/rg4dWDvvEmAPx2Q8SVM94wz8ZMMB+9tJ7n2W9K9doZ0bDMgJ/zXPLMEX8FIB6qdakuqXO0PYltyj6McdqxQbV3T54RbWXN4VtEcROZB6rJKr+OYf6tb3WrfbPyKK64YZv9V1120SXfAtYXXCGdw6RTpGNLcFsP+yS+//PJ9Em23MfsNjVmyeGTtIz6CSOqh6KnwOmrNkVFp5srehGeZ7MjbZjtsU1bVPbrWtKMKL9NlWd6Yvu3boU5FMwMmhy1zjfGe5LTugGe2no6QC0s6WO0Y0jhUmHN1CloFn+POLSQnDOCEK+264egGkwdvE9aVF3q4DSe+qX7hD95UDr7IgHMdRF5kRlbkryNzFc+q/LGywgPnGqNbNy1yW751/4CcDWwpIsD5WhM2kYm+tV+vKz9yjxvuDnhmi1RHO8aSjqgTCessV1555T5Sh7BzwNZ/sz/UAxFrweHfx7BmZBsyapHkbSJzjGdZ2lhe1WOd8CaywhO8TnlzaA8qF3+uOeVVmsrX6pF4aBKv/MvCoQ9/pU1eTVsWzt1hpTGB4YiB8TdWTqU/ieHugLfUanHQ6SSeqn/84x/fJ90ryJkBZ6acjrWPsEe6BU6ZBbKNM+ND9Z0i6KUisK5DH5hOwE93wFtqJI5U5zGbdTnXVFoFT3rTweJ4gytdD3cLnEYL+EJG3Y7pDjLrwBywsbNr46U74C329HQOywtXnX0BQzxgz7DbrHSi6ohD03G3wGm2gLOBvSAS8FaifdCBzILruEreScXdAZ+DluOAbSSvTtZT3oB0nSk7IpLecbfAabbA7W53u30P4rzE5JyU+vDN3aX4rkB3wFtuSZ1DJ3EGhIdtwD+25QcOF+SfXFieeE2T3qFb4DRZQP/ngOsShPr7nBcnDIwt48W1K9Ad8JZbUuewdca/d5yqNG/A1RkxJ80hw7vWqbZs0i7uFFjAOPBqNAdcHayT0eo68K6ZojvgLbcop+uf2kbygA7lxKfMgKVnGSI4zjo8B8G1Ax+FnLHyx9I21W2bsuiwbXmb1us08+n/Lmdx1PMpnJdiqyYwrrY5To6DvbsDntkKGaSwK7NZ7DVP53EgDmcLJ/8Wt7jFHp00yxM6VK6B8AA/0Sn6Rad1RFYe9auyat4cmS1vGyejps2RGZopvqSHbg4OD5w6z+FbRhM5VfYy+jYvfNKjF5lVbsszFY+s4CpDGkhe4lOy2vTwRWbNX1eWcWMWbJzUZQiv85sBe66SCcwuOeHugGuvWRJOo8MujjOQvMThmqYz3uMe9xg6emjkp5N6a67Sh2YdjD8yanhdGaFP/aJj0jfF0a3ln0pv6cbirW4HkTUmf9O0qkcNryNvim8qfa7sg/KPlUNm2xZjdHPSHOOaZyfovYyRvcDiY85e+kmF7oBntlztYOnEcC5ihOO4QiNdpxmD8FbZY3RHlZbZea3LXF1SN/ThT9pUfK5sepEx9scV2XNlRb9N+FaVcRCZ6RNkpB3gpK8qO/mtDuL1mqJL+iocWXSr0JZb88bCeRZy3tmH1fWVZLQ5ZtU42jUH3I+jHOsNTVo6WbBsYYMhWJq42ySOIQNFh/EKcgvpcPjjUIQ3BbxjsqIHuXPko6k80Ye+U38koQlfyqlYuL3whScywpN4i+OE0E0NxmUyanmVTjrZkd+Wu048ciIfnip3Sm7LE/65+oU+9Qomt+qVePLDN6VXTY+cpImHH27zQxccWvHUy06IuhdY3gc+8IFhGcIZxrUMeScdxqdmJ71WW9ZfRzHYc6Xj6Az1ysCrjoFDvvOd77xHRzU8aEArc0jc4IdOubDX8Driat0qX9bfatpYOAOklp9wZMOpNxl45gI9Uga8Dm/KCl/0qfGkzdVnjC51iyw4ZYzRj6XhqXypZ9LHeNq0lIkHsF3lTzh9MfFWzlQ8cpNf49E3eatwbAY7xL4uQ1gH9mr/uvqtKvM45PcZ8IxWSEeOgx3rXOkcwREr7hxgPOno8iLTbNnsMjOA8K2LySMjukV+9El8lVx00VM4MjNjX8Uvv5YVGdV2ta70C48w+mUQXvq05azijdzWJpGTuoZuU0y3tCnZwrFpZM7VtdLjoSOI3ZLf4uTXcsIP54q8lr/ytXmJR0bF8ubwoouOwmlX4Rvf+MYL5wEbG0A4X0ARb20p7aRCd8AzWi4dBdb4Pv/jtgjobNL9Y+so8nWmdGx59jd6M044HQ1fliq8cmm7jfxNgTyyYVvg8nHJyJS+CtCii57k0VG9ImeZjNC0ZYm7yPMVY58AqmWROcU7Vl5ksTXdMiseox1LS1nyqiwPe3yJ4aAQxzHWrqn33DJiO7i26yr+lBOMngzgD8FHW31Xr9piyFzjJ7bD4pNTPk0USFmJj2HfkFN+TjzLCYHnnV0HthPCJ7GAsdZnwGMWPIVp6XScCRBPJzbwEk8HlGftKnmVJzRkGRQHhapHwsHKSnlT5aBFkxlG6OmWvCle6WjG6CJHHlnioYMDwrFr0qZw5cOTMoKX8eF1oc0VefkkzhT/3HTy6BXdUib+lDklK7okX3wVT2iD2zaUTkagtkPSWlzp27zEq67qKg6DVfycLD05XmCM2I525syZfVvRTHjQxAa1zIHxBP/0GfAGjecLxj4s6WFBOhls8Op8ZsPCOpfwwx/+8EX7tV10ZjT+2c2QfWwxg2YDlYZDS3KAtY98jn28Mbqukq+Dm33Qz+zSB0bbN5TGZGRgtOUYWGa+ZlzW965//etfjX2K92qEZxPYzvf2DMqb3/zmex9HVW7kjPElrdLgMRs0W7VU5IWZg8JHP/rRoV31ETLZEKTc1j5T5aF3GI3ZpT3l2tXHK+fAVFnagTxybPmqLz1ELt45OrK/uxnt6iOwadc5vMpKf0evfuK+C+dljLoTQr7taA5t1392CboD3qA1dRQdN52EQ3CJm1kkHZ2O6VYraSkOfTogJy0/AzU062BOjjyDJ/IyCOfK0dHDk1mJON3mOOCU0w5AMtQNrrpVupQbGctwZuRo6BbbRt46svDE7uxH1jr8Y3q27ZC6j9GOpdV6jLUrnrk6RlbKYX+8dIrtKs1cuZGXukVe0smcKwuti27AV5LbPwZLdGbCu+aA+y6I9JgDYB3NoOMYdKI4L2keKHToFugWmLZAxk/+CDhzH+n8vM/7vD0mM/d6NOVexgkPdAe8hQbUcTKjzexMp9KRxvYAb6HILqJbYKcsYKwEjB3LEO66Aj5w4CFfnHTSTzruSxBbaEEdxgXcOln/zS3Uda973b28LRTVRXQL7KQF3C1WsH5eHbA1//pKcqU9yeH9tT7JNTlC3f0r5585Yf/oliPOP//8I9SsF90tcHIskDFE4zve8Y4LD7sDzgTmgN1h7hJ0B7yl1rTu62l6bqWyBOEpeGbHWyqqi+kW2FkLZKzYC1xfSbbE50yI7BneFQN0B7yllrSNxpPa/IvrSGbA9VNEWyqqi+kW2EkLxPnC7RKECvvK+Kc//emdqvupdcBZKthGa5Jli4x1qnoGsA9xWgNu17e2UWaXce4s4ABwe6B9jWHXbnnPndW2I9lY4oDNfu0gyizYGLJH2J7jXYJT64A1osZ2awMnnnDiFQ9EEz8crw3u5AEP4W5605sO4e6ABzOcmB/t+LznPW/xzGc+c689T4zyO6SoF4DigFXrrW996/Cpr1rFOl7b9GV5lfYow6duF4RG8Q+bDe7VOeYWKDTiU43YNhoHnPMXkqcDRWbSOj7+FrDf9PLLL194o80MODtajr/mu6Vh3jZVK+PQNrR2L3A7vup4zfjNJMtYl9byHKXVTs0MWCNwugGN4TLA0kAaz1UbaG6D2Xr2yU9+MuKH9d8zZ99pB1XekNB/jrUF0g+OtZKnQLnb3OY2ezshMi4zXqeqX8daxjxc08k6LnAqHHAMrvE4So5YWnBtHGF54ZnbUHkIhz7ynCtATuJzZXW6o7WA9rKbBa53SLRat18cbU1Oduk3u9nN9r0Npz28Eed5SwsZs1Ptk/yW76jjp2IJwj8gsE3s9a9//eLd7373EL/lLW+5uP/97z8s9k85SQ03lTcI+exPDiZJBzCAz3x2BjxXRpXXw0dvgbRl1WROX6j0Pby5BRxSVdeAjSlb0ewHdsLgGGgf7WZyZbKFx5kXaUu4/VMdk3NYaafCAWsUJ4U9+9nPHs57dTiOGeub3/zmYWvL4x73uL1TsNJQGWhp0GUNoqE9uMkZEGg1/Jd8yZfMct7LZPe8w7eAPpDb1vSDw9eil1hPHTRewdQMuLWWUwZNtqzjP/CBD1zc+ta3HtpUu2pf4/M4wKlwwP7xXvayly1e9KIXLR71qEctHvGIRwzO8ud//ucXL33pS4eDP/zb5tVHDeSaO/h0Dv/MFTSwhwhgrpzK38NHawHtD3rbHV07aIPzzr6QkQkTTWwNdCc7BWk3x6lecsklixe84AWLd77znYuHPOQhiwsuuGA4MvM4tempcMDWjF784hcPe3If/OAHL+5whzsMT7adZfr0pz99cdllly3uc5/7DA5TA1orhjnuNJZ/Tum5EtcR7ButXwOQ5yUMb8GZFY/d8qScyIMPAlVOwtF9rlw6hafKSJic5K+SSVYAf+wVWeSEZq7MKi9yguVtIg9Pq5u06OdW1p9p0tCugugUjH6dOtZ6kEGHyILnyIv+kRWeKifhlmYoYMUPnvBXHDb5c+pc6YRrXdndPnoTo5Tn6xjuNtH9/+3d+69dRdkH8P2+/g/+aAqRqIEmVkVQQS4tmKYqRIqilZtcQrh4IUU0GtNIFUIhBMRiaK000ERIIBBKQW5i5RK8xKQa+QVM+YEf+R/e8xn9nne63Pvsvc/Z+5zdc+ZJ1p5Zc3lm9vOs+c6sZy6r5i8e5X+rkzRWJj322GMFhDdv3tzT/k8//fQC6uGpffKvxKh4TQCwHWp20XzhC18oCo2SKMJSMR/9o9SMWDUyCpEuSo5bP2D80lkao4wQRTrTFIkPhWf3PvWp0ybNYty6ruGZMhbi161fv7Th1y8u5S6UJvkGpRkUnnxxky5uwrn9wur4+FPfhe4DuHiOIsPw6rqp01J41DzDrw4bxd/9z3We1C1uHdfPvxAv6YfVMeUM4gMY183NozBFJA0TBPMCgBUf8Ez96jKTh4nwn//8ZzFH/OUvf+lt2bKld9555/WsspCGjpM2fJbLXRMA7LthlGBnTb5DRcA5LP3o0aPlZP80NkqlkLhRqnvgWl/ijHJrE4TRb+y/1pBGudLyhx8+XZ6LVXzNJ368UjZ/yuXvR/6/vPLU/1FYwvvlS1jKSjn5v+Fb/9/kSZpu3sT3c6VNnVJP6VJuvzz9wvCRJ7y4qaP04t2nPC7dDisnPFI3bvhxh+VPmpSvTKO58BM/jORFKYub/1HXz39eLNUgmLqlvFF4pj51XcNHvfA66aSTyoRbdKQde3P1VusrM9L3K1N4+KoLvzyHDx/u/eMf/+j99a9/LSD8+c9/vmyYWoocRvmvg9KsCQC2rTQPsBnRKNMDhMyqGsVSEmUaLXvVERbFCKdA9idhwBu4AhevObYhC3cvLVA36Yfch3dc4Tlk2in/dccgblzyYDoxyv/0YNYnSSkTqcdClHpKz66tfvV/DZ9BPBLfLYec7eF3kIrZ6/qzOnWZ+Hbz9itLGm8c/is7X2a5R8mbOoavPOqlUdOBz/6YsPVWRNf79u2bB2G6lZ+7ENV6dah4nqG67GF1FS8fvXo+dfL0il/NZ1A9kgaflCWMHvDDhx4A1WKJ/PFjb631mvKG8U0dk04+bcnz8sYbb5RgOsj247St5557rrRF7YaM+pWnLWS1U+KljQzM/bz66qu9L3/5y+VyZou34eWmNQHAmVyjwPSYUYowQKoRU47LLihbUTVEow8k3AOnQcjjHuEDrDwoHmggLd5XkL/73e8Wv3zhk3Llr/2F2RJ/wq/LJnXthnfv5U/afrwS183X7z756/8pXc2jTiOuLt/9IEq6uEnXvU/4Qq48XaI/IKCu3/nOd+YBmB49P+IXon48pa//+yj5u+lrvt24cfiFT81DWO5r/yh8pUle/n78hS9EKbPOS95AM5f85P/SSy/1XnjhhWPCk08adXFf68m9KyDM1VHqYHW2P/zhDxsAE940yMcCgayRThRDSSbPKNkHLB2cIw4xH3g1MVLICCH5NMykiysPZVPooUOHSlnnnntuCZPG6CV8pK3zuZ8m9WsYw8qr80g7an0Xm08Z8o5ajvRoMeWNkofeTcwa2Zm0oTv56F7DrRv2v2sy/HeUcvtxqfONKp/kkZ6/m6+Or8scFF6nWcif/NJ0y+zmq9MmrpuHvPOW6a3UoMYEOlMiHeDh6ubDj+7YfQ2mUNLySy+/t0R2YHNB+aCo+OWkNTECtuECwDILGKHmw3453s6rhxULUaTDoD/+8Y8XPeRBSdxCymG28JDYAffggw/OJ81oKrzmI2bc4z/nAQc8o8hgsX8pZQ3Ln/rU6fqF1fH9/HWe/C9hiB537NhRTFF79uyZN3HUefrxHCWsW9YoeaRZSr78r5Q1qqyTfpi72LqFby3Xum4BWc+eJaS33XZb77rrrut95StfmQfUlB03/5Up49577y0ALCzheNrcYZfdWWedVXgZbCU+dVoud00AMEAEsnpDtkN+Sj1y5EixL1GAHjBKoKQ8CHEHKSTxXEBLufwoceGb8EG8ZjE8dSaTaVPKWqicfmn6hS3EQ1w3Dx0ljBudeUMyCqvjhvEeJT78Rklbp1lMvn55+oXV5SzGv1iedb5aD565THoK12ZRBgPy1XnF1ffRoXB+bzLW+5966qkFeAE5/srBs04vz3LQmgBg9t2vfe1rvZ07d/Z+97vflRENgT/zzDNlYuNzn/vcfy0bo0iKoZR+ihEvnBvlUTAbcf0QUKJGPIhPlNzNk/Bx3H71TP5R+PfLL1/CR+GR8pIn97UbnrVbxy/kH8R3HF41j+QTxo/ipmEmTBqdbCjpan7iuvcpQ1zy8A8i+V112vjDO/cL8ejGyZP84sKjX1g3b7/7fvn6hfXLm7A6Pdl250qkU09tyMVPLyH+3Hd51eHMjBs2bOh98Ytf7G3durUsORWvTO2Xn7vctCYAmHD1dtYAMt7bTWP21sz3N7/5zWJXiuApsbbXJpxbP8Dx93PzENUPhFFUo5WXAH2hWjd1rcTTv0uDNIGbZ0JcbJLJIy68hvFOnuYeKwFyI0NuBjP85k4SV+cQlrT0FB3UAEpP4qzO+PCHP9zbtGlTaeunnHLKvL6kr/PUZSyXf00AMGVZdrNr167ewYMHe3/+85/LZ68dxPOpT32q2H+HCRwPxO02OGF6UkBvJUT3oJDkHVZGi5++BKKLNPiU6J7+xGu45gwyMZu0dCwuk6pp/DWP8E9YcwdLIKNOMiNjA5fIWlj0gQNA1a64dZpwT5j78GJqdOzApZde2jMpHp7i6W4WaE0AMIG7jGYuvvjiopT0fFGy+FGoTsdPqS78gO9pp502v6U5/LIMLvfNXTkJ1PqjN6AqjP6iR531pz/96dJJR8dJw2XSQvwhed3jlzegxDW3vwSYFMgysk8HR5YoIEmuJ8ydCeE8BysgUNLIK592LH10aM30V7/61d62bdvKKqc6fa23wmwFf/5nrmL//rcrWInlKJqyKQv5y1GCcIqLskepi/w1D8qPfcrDIE4jFq4c5aa8Ufi3NNOTAD0h+jCSdQ8w6d/oCkW/9Ed3adzs+wCDjrs6zfPAzXNWmLWfgRJI+6CL6CFhdEOO9BJg5qY9SZd7fumjRwWmPYsLz+gMD9cs0JoYAVNAFELR/GmIlDZqg9G4QsmTBoef5UvWHiJL2ZwtYclbHqDkbe7KSYDurRHNbkdzAV5VbXnNSpg02ICqPMJstjFvkGeJXqWxK9LlmfAcMF3MSgNfOUkPLxlokpNOjXxt57eW3r2VSzrEtF3ptFXydiHypg+rmyw7o4OPfOQjZaWDXY3045IuOsMnPIfXcPop1sQIOA0pykjj8AAgygmgLiTyKF4aPMLXBg/bI52spnGLMwK+4YYbiv1Jg8wDsBD/Fjd9CVgHbguqbeLOA7BskO43btzY+/73v186TQ0d0W+eEZszfvrTn/b27t07v448td2+fXvRNeDIM5G45i4sAR2WN4u//e1vvV//+tfl5DKdoZ2otgenreLCLz19uazrv++++8pqJjyQlQ7WCltaSh/ypKOUxz13VmhNjICjRCAYEKUUDS1xwxSSfEmXhuaBcN6oRmjzxu7du8sOu5/97Ge9H/zgB2VUZSKg0WxIwGjpgQceKLsiNXibdLgPPfRQAePbb7+9PCNGSSgjWkCsI2UbtiEASaMxs0umoXuuMlFUErWfgRIgK3LVId54443l0PQkrsFTu5WWrLU7l+3/2pqlpAD361//evnAwt13311A2TGWn/jEJwq7dKhu5I3eUtZKumsCgGsBB3DHHZHKF+Xjl17Uhzgd9u6BueWWW8oIysPyve99r/fUU0/19u/fXyb96jo0/8pIgF5efvnl8sp6/fXXl1EvnX7pS18qJ2Q5vMXBTXZJeT7qV1Xb1sUzLTnYO5TnCR/+cZ+r8FmLLnmZoGaq0ynaIEG+KB1g2lnkI4/LIUx/+tOfym42a3sd/+q42bfeequMnl977bV5AE5ebvLXYSvpn52x+EpKYcSyKc+DodcOGDtHmLL1uDZ0iJeODcoec0ve8jCNWExLNiEJGL0CXbqiA7sg33nnnXL8oNGRcCMptt/PfOYzxcabswNUQePXseJDp2zA0gJjr7zRqzJQ3HLTfoZKgHxd1uYyAWW1EL2kIwsAR48J99bJju9sCOt8kbgT5lZLWK5mrT8+s05rbgS8VIWkB41yHVnpAr7IA6Nhepgc9PHKK6+UB8WkQqPlkwD90FUoAOqYQiYD+tCo6csuKaMwdl52RSR/4umTbR84s+2buDNKZo6gYyaHrJAomdvPRCWQNocpP70Y+Bj15tztxAkzGDKZp8M0MTfL1EbAY2gnoKvR5qEAvhqtuIyIsPSQGCnrlb3WNlp+CdBRdKV05iKz5nTjYhsUT0dGuUbDGnZNdOqiS6NmQGszDxOGV+b777+/5DVSbjQdCWhbdBRd0CtdOidYWNollx65Oky6nnVqI+AxNRTg7WajeI3UyFcaEzbCNHBXo+WXAD2gNFD3/Botcg+E3afhphGLA9L0qfGvX7++9/vf/77kl8ZXVmzqufXWW8vmG2BsJMw8kWegFNJ+JiKB6IpLV4jetLl0puQfoD5e2l1DhjEfDw+AK6Sx+YqCh8EoKA24btjsUo2WXwLRVfTF1ODYUQ2WvrjpJNWOecFpWXSnIdNt0rl3iaNnqyd8VQGPAwcOlHR4WNaW8tw3WroE6IBMtS0Xsr4+yzsDuuIAL53QJVPErFMD4DE01G1YFK6R2vZocicPh3Qatm9P8WvUjVZeAgDY5dXUK2z05f7dd98tB3R3G60GDXRR3mQAAqDVsYqz+UYYfkxS4bvy/3j11oBt18c5Y2Ygc22NCem9994rk+COBph1agC8CA2lgVG4BmsyxoPwr3/9qzwEGqqG6PQ1X1SQrtHySiAyp6tc69atK5NnGinApSeXT0/ZxehNxlkeIfmMprjswyGdKxuj3VdegU3E4aNMr8GNJisBck0Hh7N7JiH6Mr/y/vvvz3eO7qUVn2dgsrWZLLcGwEuUp1lXh4QA4F/84heFm51xDz/8cHllveKKK5ZYQsu+WAkATqQhuoCkztIo2HfFjs5NngJTnx+yxlfchz70oTKysrbbRJvjS4EvwDYBh6cGDrQvu+yy8hp80003zQMvG3DKXWy9W75jJUBvNmv4xuJjjz1WPj5r2aDla873/uMf/1j0azfd66+/XjZ0ZFXSsZxm765Nwo2pkzTmZGNrMjsOaH/zm9+U7a1GQb6SbHePLZEarIeo0fJKoB4BRW8api/hWvjvtCy2REDsWFJnQyPLl2zYAMJeY9l7fZzTG47RLn1q7F6BfSrHkrTo1/MQ//L+2+OvtHRU0RO58cfVObLvuveVZLvedIJZ+/uNb3yjrO2249SWZCNh+tLu6Az/8J5V6ayJsyCmLXwPCpuiTRd26FC67015nfVAeI1gIU9xAAAQD0lEQVRtr6bT1sJo/IGn5UvOgbCTyujWAS5GVNaUmmAzimU+MuqyS85CfyD9hz/8oax+AArsvw75dpCPyaA09ga+o+khqbSNAC7dPPHEE0UH2o/NTGRNtkxEzlsBvlac6OjoydyLjVBcurBLkT4zGSr/LFMD4Alpx4PkgXDYCzIx5yHw8HiwZv1BmJAYZp4NfbiMpJiK6EZjdtXEhk+fQCCrIehWHoAhva/q6mzxCPDO+oir/o+z4NdutA0yJDvydU/m7nOxtYsT7oq8DX7oRUeq86QTE6N0jJJuFv5rvzo0AO4nlTHDPDwepH6K94BkFn1Mti35FCRATxpnOkR+jZQOXcKFxc8VH/3mtViagEf4cRsAj6e0WoZkF/niEnkKj58+QvLSjSvx0krjPjpO+ll0m2FyAlpJozOq8lAgDwDyECSsBLSfFZVAdKWha6h1445fg+Y36uIPCBsR06tw+ev0dOy+0XgSIF8UWfKTLTkHbOPW7Ui8PMLoQ5q0uegv+fCcVWqTcBPSjIeB4jPazcOQ8AkV09gsUQL04UJpqHTlch+98SMNnJ/Ljp80afC5T/qSqf2MJAGyji6SgRzJVrjBC38dljzJx01aPJI3egnfWXWbCWJCmknD9TDEj3UelAkV09hMQALRT62bNOy4KUaDDtUgK12dlj+NvuabvM39bwlEfl2QlTIyjJzrsDouXIVJW6dL3Cy7DYBnWTutbseNBAImx02FZ6iikZ23jNhtE6aag/wz9BcWXZVmA1606NZeRiMVjQFx+/mlyahRfH0/isSSt+umzFF4LCWNOgMC9ny2xVEpo7JR0y+UTvmuyDh1ijy5SJrYS/lR5FZuhvxIi1fNI7yHZJ1odGRXv2EkTEGD/BOtxAoxawC8QoI/HovVEPo1Bg1ZI+YmTRqysIDDOP855ciP8It/HD7jpk39uam3sl3jAPK45aYM+br/1T1K3crNf+4TV4fxJ5wbf9Jwa71Ed4PS1vmm6Y/Op1nGrPFuADxrGpnh+gCkzDirpsPN33777d7RuU0K1s0GIOqGZLeYbb92lWnoNWnwwvCtwTXhFt4/NPetNvfia741n0n7jcRMpmbzjPKNiOsR2qTLzH9TlnLIzZGXvvqgbBS55N6h4z5eaZdY6kaWtlVfffXVZeeYfHjL60Jk/qtf/apsRrGyw2u/NNzwKQnbz9Ql0FZBTF3Eq6cAjVMjduDJk08+WRo6oBBmAbyv2W7evLnsVkqj9ukY+/d9nikgU0tEWMAXeMhX39sy/NGPfrTsVFtOcPDVC8Bm+7GdcOeee+7UwCnA6H8718CWW59ot/HAZh4nf/ngq4OdfLmDzMjddulHH320t2PHjhImP3AW/sgjj/TOOeeceVGnjAToOO0E/MlPftI7YW5XXzc+6Zo7XQm0EfB05buquAcAHV7jsCEjYMDqMmJ8+umne/fcc09p2EDCN9iAgVGWMxiA6zACIvK6PvnJT5bPB+3ateu/Rs/D+Cw2HujZcuyr1r5+/Mtf/rL8h/z3dA6L5V/nw6vL78033+w9/vjj5f+ff/75pTOzzd2Xm41aHbVINk5icx6xbbcf+9jH5uVjRxj9qC+ZA1YdW2SqfHrwpW6fWHLIvM6mjq/r2PzTlUAD4OnKd9Vx11Cdf+v112j3hhtuKNfNN99cgNio6vnnny8H2jhvwQjYqVUOR6kpIy78AALA4Lrndzlu8Kqrruq9+OKL5WyGrgmj5jcpP7BjLnGuh5PuAJ1DXtQrYDapsvALT0CsHOU5NN4ZFD637qS1b3/72wVonZNw+PDhMvp15ohR7NatW8soWZ3IjIx8GdjHQ333Tr1R6s91OaeEToC9MkPAOrpJWHOnJ4EGwNOT7arjDCTYgH1x1n58r8MOonEZiTnO0RkJRmmAzAiLbdinxrNBhVACPFyA4XzeI0eOlJGmU8a89sfmCiicx8CMMc1JsCjLf3G6nRO1LrnkklLXaQE/ICRTcuB3xCVZOBSIWUCYL/x6wzjjjDMKQDvkX+cHgJ1FrXMLYHLJO6e2AVPgypTCLmx0nxG3cxOc1OfUPpeRc+rBDc/IpbnTkUCzAU9HrquWK3MCoDA6dNJbAARIafDugTAbpRHaBz/4wTLSAp4afRq29MDhlbmvRhslO1AFbwed+2rxnXfeWcAZALF/GlUzC0ybjLq3bNlSQN/ZssjIEgFE9c7kXAlcwk8AmIsAq3Olga+vqKQsMnUpl63dcZlAc8OGDaXzk06HqNNybjGgZbfeu3dv0QFQJ1uniH3rW98qNmX8nAAn39///vfeZz/72aIrdYmOlvDXWtYRJdAAeERBtWT/BiCN3+fbjdK8KofYEb36aujr1q0rIyrHPnoNzieZ0rABBTPG/v37C5AAHHZMAL579+4y+x+QkwfQmJwCGjXlXhojPPVKGXW62m+Eq35e0WsKCDKDSFObRWIaqXnHXwNWeNRxynDvEp80CQfu/rfLxJvOzZGLOp6kAapMOzo95xYb5QJqcpE/pgdy9SahLMczkr/zi+nApKkPBji72AcEdIbkoBM1qQq0dZaIXLt1LRHtZ+ISaAA8cZGuXoYatqVRgA5IACqvrgDCZJvlZlZCMBuwZ4oDdEZZTBABTCNjn3M38rr22mt7F154YXmdBhr79u0rIEGK8gCbHIJuUs8IMCPSAJu0hw4dKmfG8i9ERrgOZA8AhweeAZ0ALj78NdWmFOHy+181nxrAasBN+oTJgz/w5ff/gKaOjWzTqR08eLAAMDMEU4/zb5FRrvpETkbCTDj+C55XXnll7+yzzy7/S9pt27YVIGZ6IAf5dKRAP8eo4itv938LbzR5CTQAnrxMVy1HwAGArX4w2rUSgrlBGHuvr0tcfvnlxWRghh54WkaFAAwCzJZZMT1cc801Jb3RLvBgN3YByIAUkDAy4wJ6o2WADiSSBuAAJqPGAGkprM+PNPnwpvypV9w+WY4Jkk5d5c0lAZOLjklY6sYfks+o05Iy9U/euEagABgPIKqDIqvYfIGvL3awhwNMvPgDtgCT+Yb9l05MjjrUPHUAxPLQk44RqROd6Qi9uYSGyTDpmrt0CTQAXroM1wwHDVYDBzAaLKAAaK4LLrigTOqYcAMAafiEAxzcy2+U++yzz/ZOPPHE8kmgmBqAta8eAAmja4CrHC7Aw0N+FzLKFAYs+C+66KISNwp41HzwkmccSn75Up4vaHgDYAZIfF1nYSYtrVrQiaDICI90PgCaLHyNA8CSE5PB2XMAahTL9JAyMynp/5MTYLVxQ56sARYnfdL6UIB7lPDopwT+Jzx1S1hzpyOBBsDTkeuq5KqBA2AftTTCMjlmBOUCpBp5Gq5G3W3YABVAWCVhmZUPYCLg5BWY/dKImW0zhEdGbMwP7qUHHlzlARnAjkfKT/6ua/TJlqrO8qPwkTdh3Xz1vXSuAJnygSbzQYBOenVFqZNOK2Ul3D0+Rr9A2GQY2bKbk7N64i9d3Px3PFJfcWRrIs8bBEBWLplLb22wulmpkrcS4ToM99KHUt/cN3d6Evh/qU+vjMZ5lUgAwDA/rJubvPHK70JpsHGFsWMazea1XBiQMtEELKxBBQ5Ayj3gsAECEOCP8BMHnKT1za+AT4CHi4dNIFYG1HUoTDo/RphGlPlqrvzh1Uk68FadUg/lKR/o6VRQ6iAdv7qnnolLmbm3IoRpwSSbN4Da1q0sxCVX4AzMTYgiMmN+sImDzNngkTICrHbMKYtZAuCKc8/MAeTZnBstvwQaAC+/zI/bEk2esXUCCWYHjTgNGTgETPxBkzwaNfAEDvxGYPwAiV8eIGWyyWYLI2OTQsAnJC07KOB0ZVQpr9Ebl4kitl3xdT3CJy4ACwCpO8Ij/yNhSd/PTRkpp86TsPDlps78ia/LU76OiWzYu2vwlacmbxrkQC5WfmzcuLHI0FvC0bkzOdAJcyYO5eCLLPPTQRld212Ih/LZnY26bfVm7kD5L6lnCWw/U5NAA+CpiXb1MTbCBMAaOBBIA6//qYbrMkqz/MlrMWAwWgbCgDn2Y5N3wMZ6WzP9TBoAMny50pokshmhBjJAAUgCNFZTZMQ5Cniknuoef8Cn/j/9/MpBRpfyyM+t/eJrvokThvIf+XVG1urqGMgt/MQlffzKNGL1NsBmjvDCn+0dEOuw8EEmLq2pZnb58Y9/XNJIj68JOWYbmzksR6sp9a3Dmn/yEmgAPHmZrlqORqhADwAD0wCeP5xGza/xWubFzGD0xbZrna88J598cgm3lAovmy5M5p09N8kknTThgY/XciM9qyvyOi2+BqaE16BWmAz4wRfVPHKfOPfqp0MAbCF5hIfIQJ644iMXfnWqZZN8KUcapgQgbPLMRB0SLo2LvyadmMlOMjSCNWpWRzK03te5EXYrykv+8u/YsaPY1tOJidPxkV13BCx9t8y6/OafnAQ+MKeYHZNj1zitZgkYzQJWDd2IKg2121gDOkwEAMDM/ZlnnlmWXxlpAWb2YHZPYGJlALssG7OZfq/JQMzluEU8du7cWUbd3bIWI+/Ue1DelAEUjcDVZ/369fNgWAO9tO4BWy6gFn/Aty4T+IWE+9/MMIDQbjWTb6jOk/TcmFCsfdZhyaNTYL5JXudwMGnQ1fbt28v2avUKT+Xdddddxd68adOm+XyJr8tr/ulJoH2SaHqyXXWcAxxxNVZ+LqrDhbEx7tmzp+dcXwfKOE4R9UufvEDPRBKyQws4OxXsRz/6UQGbGvxKogn/qEf9n7DPvXrrFCZRh0HlKE85kZH7LslrNcodd9xRRroHDhw4Zheb9AvlF//b3/62APDPf/7zMiGZpYPDypa30eQk8L+TY9U4rXYJpFH3AyBxwl1JZ2LHBgK2YFtpk4acktZIMeFcIzmTaq5X5jZrAAangtWv/dOUszq4UO0C3jqs3CzxpyvHlJ1yB7EXb7TrsCAU2fLX8oyMuy4AZ3PXIRp1G0WPWrYyGk1OAm0EPDlZrnpOGm6IX6ONG3/uk86kkBl+LiC2DhdJD9QABjKrj4AFv3iTTPxAQrgw1zQp/1E58Ssv/6uu82LrgVf41TxSnv86jKT1hsHWq5NyxgNKvcOrLidxTBNWtDAnmdCLDb2Wbe0fVpcWv3gJNABevOzWXE5gmIYNJPgDjIThPiPF3AdMNOjE8aeBByDCGxhIV/N3L13AepqCz39IvVNWXf9uXNKM6+a/c3PhoaxRykh+eWr5Cc+OvIyI40obkgeRbx0f3SRdc6cngQbA05PtquM8CEDzRzVowJF0woEB8wE3xJ/7GmwSJh0AyWg5fJcbGNTHfwkYTqr8/M/wSznuc0VWg1x5mGki24CocDzchz+3LjNp8I4/8fm/yTuo/BY+GQk0AJ6MHBuXJoEmgSaBsSUw3Ng0NsuWoUmgSaBJoElgFAk0AB5FSi1Nk0CTQJPAFCTQAHgKQm0smwSaBJoERpFAA+BRpNTSNAk0CTQJTEECDYCnINTGskmgSaBJYBQJ/B8+zduzcuujCQAAAABJRU5ErkJggg==)\n",
        "\n",
        "Credits: Natural Language Processing with Python book"
      ],
      "metadata": {
        "id": "60tsv6qOwnmD"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aNPbZrfly7xy"
      },
      "source": [
        "# Naive Bayes Classifiers\n",
        "\n",
        "- Every feature gets a say in determining which label should be assigned to a given input value. \n",
        "- To choose a label for an input value, the naive Bayes classifier begins by calculating the prior probability of each label, which is determined by checking frequency of each label in the training set.\n",
        "\n",
        "Credits: https://www.nltk.org/book/ch06.html"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "SHAOFSfsy7xy"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}