Joint NGD/Adam step in example notebook
"source": [
"### Difference #3: The updated training loop\n",
"\n",
- "The training iterations now consist of two parts.\n",
- "- The first part is the NGD step (where we update the variational parameters).\n",
- "- The second part is the Adam step (where we update the other hyperparameters)."
+ "In the training loop, we have to update both optimizers (`variational_ngd_optimizer` and `hyperparameter_optimizer`). "
]
},
{
{
"data": {
"application/vnd.jupyter.widget-view+json": {
- "model_id": "cf4264ce654e49b0956fafad188850bc",
+ "model_id": "126c78ce830a4352aba355851d55ad5a",
"version_major": 2,
"version_minor": 0
},
" for x_batch, y_batch in minibatch_iter:\n",
" ### Perform NGD step to optimize variational parameters\n",
" variational_ngd_optimizer.zero_grad()\n",
+ " hyperparameter_optimizer.zero_grad()\n",
" output = model(x_batch)\n",
" loss = -mll(output, y_batch)\n",
" minibatch_iter.set_postfix(loss=loss.item())\n",
" loss.backward()\n",
" variational_ngd_optimizer.step()\n",
- " \n",
- " ### Perform Adam step to optimize hyperparameters\n",
- " hyperparameter_optimizer.zero_grad()\n",
- " output = model(x_batch)\n",
- " loss = -mll(output, y_batch)\n",
- " loss.backward()\n",
" hyperparameter_optimizer.step()"
]
},
"cell_type": "markdown",
"metadata": {},
"source": [
+ "You could also modify the optimization loop to alternate between the NGD step and the hyperparameter updates:\n",
+ "\n",
+ "```python\n",
+ " for x_batch, y_batch in minibatch_iter:\n",
+ " ### Perform NGD step to optimize variational parameters\n",
+ " variational_ngd_optimizer.zero_grad()\n",
+ " output = model(x_batch)\n",
+ " loss = -mll(output, y_batch)\n",
+ " minibatch_iter.set_postfix(loss=loss.item())\n",
+ " loss.backward()\n",
+ " variational_ngd_optimizer.step()\n",
+ " \n",
+ " ### Perform Adam step to optimize hyperparameters\n",
+ " hyperparameter_optimizer.zero_grad()\n",
+ " output = model(x_batch)\n",
+ " loss = -mll(output, y_batch)\n",
+ " loss.backward()\n",
+ " hyperparameter_optimizer.step()\n",
+ "```\n",
+ "\n",
+ "### Evaluation\n",
+ "\n",
"That's it! This model should converge must faster/better than the standard SVGP model - and it can often get better performance (especially when using more inducing points)."
]
},
"name": "stdout",
"output_type": "stream",
"text": [
- "Test MAE: 0.07682487368583679\n"
+ "Test MAE: 0.07514254748821259\n"
]
}
],
