How to choose right optimizer according to the DL problem?

how we will get to know which optimization to be used?

  1. Based on the task but how do we understand that particular optimizer is not working and which scenario
    or conditions we need to look into so that optimizer is not working and should go for another optimizer.
  2. I have checked code multiple times still this scenario is happening

I think visualizing the loss is a better approach, and please follow the learnings from Optimization Algorithms Hands-on chapter, and try to understand what are the pros and cons of each of them.

1 Like

I have gone through the pros and cons of optimization. But the point is even if I plot it loss that loss is intern depend on optimization.
Adam is the most popular optimizer. But in some papers people use SGD over Adam.
My question is will it take only loss and accuracy as criteria or any other thing?

Widely there are two parameters to determine the performance of an optimization algorithm:

  1. Accuracy
  2. Minimum loss value, and timesteps taken to reach it.

There are several other statistical methods which get discussed in various papers, but for now, you can stick to these two i guess. As optimization algorithms are not just used in Deep Learning but also in various fields, there are many applied mathematical tests.

For a broad look, you can refer to this discussion thread

1 Like