modify the notebook according to review comments
"\n",
"Large pre-trained language models like BERT and UniLM usually consists of **hundreds** of millions of parameters and it's challleging to fine-tune such large models and also serve real-life applications due to latency and capacity constraints.\n",
"\n",
- "MiniLM is a small version of UniLM, which is trained to deelply mimic UniLM with deep self-attention knowledge distillation. It only consits of **tens** of millions of parameters (33M), which is less than one third of BERT base model and only half of the size of [DistilBERT](https://arxiv.org/abs/1910.01108). Experimental results demonstrate that MiniLM retains most of the performance of UniLM on various NLP tasks with much less computation. \n"
+ "MiniLM is a small version of UniLM, which is trained to deelply mimic UniLM with deep self-attention knowledge distillation. It only consits of **tens** of millions of parameters (33M), which is less than one third of BERT base model and only half of the size of [DistilBERT](https://arxiv.org/abs/1910.01108). Experimental results demonstrate that MiniLM retains most of the performance of UniLM on various NLP tasks with much less computation. Our experimentation shows that to achieve very similar ROUGE scores, MiniLM finetuning on CNN/DailyMail dataset is 4 to 5 times faster than UniLM's. \n"
]
},
{
"metadata": {},
"outputs": [],
"source": [
+ "# example code to load xsum dataset from the UnilM repo\n",
"# train_dataset = processor.s2s_dataset_from_json_or_file(\"/dadendev/unilm/data/xsum.train.uncased_tokenized.json\", train_mode=True, top_n=TOP_N)\n",
"# test_dataset = processor.s2s_dataset_from_json_or_file(\"/dadendev/unilm/data/xsum.test.uncased_tokenized.json\", train_mode=False, top_n=TOP_N)"
]
"metadata": {},
"outputs": [],
"source": [
+ "# example code to save and reload preprocssed s2s dataset\n",
"# import torch\n",
"# torch.save(train_dataset, os.path.join(CACHE_DIR, \"cnndm_train_dataset.pt\"))\n",
"# torch.save(test_dataset, os.path.join(CACHE_DIR, \"cnndm_test_dataset.pt\"))\n",
"metadata": {},
"outputs": [],
"source": [
+ "# example code to load a saved model checkpoint\n",
"\"\"\"\n",
"abs_summarizer = S2SAbstractiveSummarizer(\n",
" model_name=MODEL_NAME,\n",
"metadata": {},
"outputs": [],
"source": [
- "#\"\"\"\n",
"abs_summarizer.fit(\n",
" train_dataset=train_dataset,\n",
" num_gpus=NUM_GPUS,\n",
" fp16=FP16,\n",
" save_model_to_dir=MODEL_DIR\n",
")\n",
- "#\"\"\"\n"
]
},
{
"metadata": {},
"outputs": [],
"source": [
+ "# save the model\n",
"# abs_summarizer.save_model(RESULT_DIR, 5000, False)"
]
},
