Remove unused imports and correct typos.
"if nlp_path not in sys.path:\n",
" sys.path.insert(0, nlp_path)\n",
"\n",
- "from utils_nlp.common.pytorch_utils import get_device\n",
"from utils_nlp.dataset.cnndm import CNNDMBertSumProcessedData, CNNDMSummarizationDataset\n",
"from utils_nlp.eval.evaluate_summarization import get_rouge\n",
"from utils_nlp.models.transformers.extractive_summarization import (\n",
" ExtSumProcessor,\n",
")\n",
"\n",
- "import numpy as np\n",
"import pandas as pd\n",
"import scrapbook as sb"
]
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Data Preprossing\n",
+ "### Data Preprocessing\n",
"\n",
"The dataset we used for this notebook is CNN/DM dataset which contains the documents and accompanying questions from the news articles of CNN and Daily mail. The highlights in each article are used as summary. The dataset consits of ~289K training examples, ~11K valiation examples and ~11K test examples. You can choose the [Option 1] below preprocess the data or [Option 2] to use the preprocessed version at [BERTSum published example](https://github.com/nlpyang/BertSum/). You don't need to manually download any of these two data sets as the code below will handle downloading. Functions defined specific in [cnndm.py](../../utils_nlp/dataset/cnndm.py) are unique to CNN/DM dataset that's preprocessed by harvardnlp. However, it provides a skeleton of how to preprocessing text into the format that model preprocessor takes: sentence tokenization and work tokenization. \n",
"\n",
"source": [
"### Model Evaluation\n",
"\n",
- "[ROUGE](https://en.wikipedia.org/wiki/ROUGE_(metric)), or Recall-Oriented Understudy for Gisting Evaluation has been commonly used for evaluation text summarization."
+ "[ROUGE](https://en.wikipedia.org/wiki/ROUGE_(metric)), or Recall-Oriented Understudy for Gisting Evaluation has been commonly used for evaluating text summarization."
]
},
{
