Added text explaining AzureML Estimator and analyzing results
"source": [
"### 2.2 Create a Pytorch Estimator\n",
"\n",
- "We create a Pytorch Estimator using AzureML SDK and additonally define an EstimatorStep to run it on AzureML pipelines."
+ "We create a Pytorch Estimator using AzureML SDK and additonally define an EstimatorStep to run it on AzureML pipelines.\n",
+ "\n",
+ "The Azure ML SDK's PyTorch Estimator allows us to submit PyTorch training jobs for both single-node and distributed runs. For more information on the PyTorch estimator, refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-pytorch).\n",
+ "\n",
+ "This Estimator specifies that the training script will run on 4 nodes, with 2 worker per node. In order to execute a distributed run using GPU, we must define `use_gpu` and `distributed_backend` to use MPI/Horovod. PyTorch, Horovod, and other necessary dependencies are installed automatically."
]
},
{
"source": [
"### 2.3 Submit the pipeline\n",
"\n",
- "The model is fine tuned on AML Compute and takes ~45 minutes to train."
+ "The model is fine tuned on AML Compute and takes **45 minutes** to train. The total time to run the pipeline will be around **1h 30 minutes** if you use the default value `max_epoch=1`."
]
},
{
" print(pd.DataFrame.from_dict(parsed).transpose())"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "From the above chart we can notice the performance of the model trained on a distributed setup in AzureML Compute. From our comparison to fine tuning the same model on MNLI dataset on a `STANDARD_NC12` machine [here](tc_mnli_bert.ipynb) we notice a gain of 20% in the model training time with no drop in performance for AzureML Compute. We present the comparison of weight avg of the metrics along with the training time below,\n",
+ "\n",
+ "| Training Setup | F1-Score | Precision | Recall | Training Time |\n",
+ "| --- | --- | --- | --- | --- |\n",
+ "|Standard NC12 | 0.93 |0.93 |0.93 | 58 min |\n",
+ "|AzureML Compute*|0.934| 0.934 | 0.934| 46 min |\n",
+ "\n",
+ "* AzureML Compute - The setup used 4 nodes with `STANDARD_NC12` machines.\n",
+ "\n",
+ "We also observe common tradeoffs associated with distributed training. We make use of [Horovod](https://github.com/horovod/horovod), a distributed training tool for many popular deep learning frameworks that enables parallelization of work across the nodes in the cluster. Distributed training decreases the time it takes for the model to converge in theory, but the model may also take more time in communicating with each node. Note that the communication time will eventually become negligible when training on larger and larger datasets, but being aware of this tradeoff is helpful for choosing the node configuration when training on smaller datasets. We expect the gains of using AzureML to increase with increased dataset size."
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
