03 Prepare data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "53414860",
   "metadata": {},
   "source": [
    "# Prepare a dataset for use in a training task\n",
    "\n",
    "**Goals**: show the steps necessary to download, preprocess, and register a dataset for use with AzureML. \n",
    "\n",
    "Working with AzureML registered datasets allows you to avoid bundling the dataset with the context you send to each training node (slow) or setting up a persistent mount on your DeepSpeed Docker images (more complex and brittle). Additionally, using registered datasets offers the opportunity to log what version of a dataset a model was trained with. This means you can monitor the effects of retraining over time as your data increases or changes. \n",
    "\n",
    "A note on terminology: I'll endeavor use the term dataset to refer to the AzureML dataset object stored on AzureML and data when referencing the same information stored locally.\n",
    "\n",
    "Here we'll be working with the [CoLA](https://nyu-mll.github.io/CoLA/) dataset, on which we'll later be fine-tuning a model. We'll perform the following steps:\n",
    "\n",
    "- Download the data appropriate for the training task\n",
    "- Preprocess (tokenize) it\n",
    "- Save it locally\n",
    "- Upload the preprocessed data to the AzureML datastore where our AzureML datasets are located\n",
    "- Register the resulting files as a new AzureML dataset\n",
    "\n",
    "First, imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d880194",
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "import azureml.core\n",
    "import datasets\n",
    "import transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44b5dbc3",
   "metadata": {},
   "source": [
    "## Using an external config.yml\n",
    "\n",
    "Now we'll load a `src/config.yml` file that tells us the task name and data locations, this is also used by the training functions and so keeps everything in one place. More information about it is available in the companion `Train model` notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d02e7f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('src/config.yml', 'r') as f:\n",
    "    config = yaml.safe_load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe3f7661",
   "metadata": {},
   "source": [
    "## Load or mount data \n",
    "\n",
    "The data itself will be downloaded by the `datasets` libraries from [Huggingface](https://huggingface.co/). Loading this public dataset is quite simple but this is where you'd locally mount or otherwise access a proprietary dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fe94083",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = datasets.load_dataset(\"glue\", config['task'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f04dbc0-6507-4c6a-ba84-c33bc4cd55e9",
   "metadata": {},
   "source": [
    "## Preprocess data\n",
    "\n",
    "We will preprocess our data before we save it. In this case it means tokenizing the data to prepare it for the model specified in the `src/config.yml` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fabfa0e2-b545-4572-9611-dd3c2f86d689",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = transformers.AutoTokenizer.from_pretrained(config['model'])\n",
    "def tokenizer_function(examples):\n",
    "    return tokenizer(examples['sentence'], padding=\"max_length\", truncation=True)\n",
    "\n",
    "data = data.map(tokenizer_function, batched=True)\n",
    "data.save_to_disk(config['data_dir'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f862c5a4",
   "metadata": {},
   "source": [
    "## Connect to AzureML workspace and upload data\n",
    "\n",
    "Now we'll connect to the AzureML workspace and Azure datastore where our dataset will live. We instantiate a connection to the workspace using a configuration file that is automatically provided on the AzureML compute instances but which [we must create](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-environment#workspace) if we run this notebook on our own desktop or laptop machine. \n",
    "\n",
    "[Azure datastores](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.datastore.datastore?view=azure-ml-py) can be blob storage, file shares, datalakes, and more. Here we'll use the default, a file share, but any of the other backing services can be used after first [registering them with AzureML](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-access-data#create-and-register-datastores) either through the studio UI or Python SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28ec883b",
   "metadata": {},
   "outputs": [],
   "source": [
    "workspace = azureml.core.Workspace.from_config()\n",
    "datastore = workspace.get_default_datastore()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cad7539b",
   "metadata": {},
   "source": [
    "Upload data stored on local disk to the datastore. We are clobbering any prior versions in this upload, assign a UUID subdirectory or other versioning mechanism to retain prior versions of this data within the datastore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa983db6",
   "metadata": {},
   "outputs": [],
   "source": [
    "datastore.upload(src_dir=config['data_dir'], \n",
    "                 target_path=config['data_dir'], \n",
    "                 overwrite=True\n",
    "                )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a88cc1e6",
   "metadata": {},
   "source": [
    "## Register the data as a dataset\n",
    "\n",
    "Register the dataset with associated metadata describing the task for which it is meant and the model used to tokenize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43537570",
   "metadata": {},
   "outputs": [],
   "source": [
    "name = config['task']\n",
    "description = f\"Glue dataset for {config['task']}, tokenized for {config['model']}\"\n",
    "tags = {\"task\":config['task'], \"model\": config['model']}\n",
    "path = datastore.path(config['data_dir'])\n",
    "\n",
    "amldataset = azureml.core.Dataset.File.from_files(path)\n",
    "amldataset = amldataset.register(workspace, name, description, tags, True) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab36c826",
   "metadata": {},
   "source": [
    "And we'll need to know what version that created in order to use it going forward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f51ff26",
   "metadata": {},
   "outputs": [],
   "source": [
    "amldataset.version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6882843d",
   "metadata": {},
   "source": [
    "This shows us the steps to:\n",
    "\n",
    "- Prepare data before moving it to Azure\n",
    "- Push data to an Azure datastore accessible by AzureML\n",
    "- Register the data with AzureML as a dataset"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:aml]",
   "language": "python",
   "name": "conda-env-aml-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}