Update notebook after updating data preprocessing functions.
"FORBID_IGNORE_WORD = \".\"\n",
"\n",
"# mixed precision setting. To enable mixed precision training, follow instructions in SETUP.md. \n",
+ "# You will be able to increase the batch sizes with mixed precision training.\n",
"FP16 = False"
]
},
"metadata": {},
"outputs": [],
"source": [
- "train_dataset = processor.train_dataset_from_sum_ds(train_ds, load_cached_features=True)\n",
- "test_dataset = processor.test_dataset_from_sum_ds(test_ds)"
+ "train_dataset = processor.s2s_dataset_from_sum_ds(train_ds, cached_features_file=\"cached_features_for_training.pt\", train_mode=True)\n",
+ "test_dataset = processor.s2s_dataset_from_sum_ds(test_ds, cached_features_file=\"cached_features_for_testing.pt\", train_mode=False)"
]
},
{
"source": [
"## Distributed training with DistributedDataParallel (DDP)\n",
"This notebook uses DataParallel for multi-GPU training by default. In general, DistributedDataParallel(DDP) is recommended because of its better performance. See details [here](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html). \n",
- "Since DDP requires multiprocess and can not be run from the notebook, we provide a python script [abstractive_summarization_unilm_cnndm.py](./abstractive_summarization_unilm_cnndm.py) to demonstrate how to use DDP. \n",
- "**Note that the python script set `fp16=False` by default. If you have enabled mixed precision training following the instructions in SETUP.md, you can call the script with an additional argument \"--fp16 true\".** "
+ "Since DDP requires multiprocess and can not be run from the notebook, we provide a python script [abstractive_summarization_unilm_cnndm.py](./abstractive_summarization_unilm_cnndm.py) to demonstrate how to use DDP. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "First, we save the training and testing dataset to jsonlines files to be used by the python script. This avoids multiple processes repeating the initial data pre-processing. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "train_ds.save_to_jsonl(\"train_ds.jsonl\")\n",
+ "test_ds.save_to_jsonl(\"test_ds.jsonl\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Next, we can execute the Python script using torch.distributed.launch, set `--nproc_per_node` to the number of GPUs on your machine.\n",
+ "\n",
+ "```\n",
+ "python -m torch.distributed.launch --nproc_per_node=4 --nnode=1 abstractive_summarization_unilm_cnndm.py\n",
+ "```\n",
+ "\n",
+ "**Note that the python script set `fp16=False` by default. If you have enabled mixed precision training following the instructions in [SETUP.md](\"../../SETUP.md\"), you can call the script with an additional argument \"--fp16 true\". You will be able to increase the batch sizes with mixed precision training.**"
]
}
],
"metadata": {
"kernelspec": {
- "display_name": "nlp_gpu_test_new",
+ "display_name": "nlp_gpu",
"language": "python",
- "name": "nlp_gpu_test_new"
+ "name": "nlp_gpu"
},
"language_info": {
"codemirror_mode": {
