Regarding "cosine" lr scheduler - especially lr_cycle_limit paramater

[amaarora]

Hey Ross! Hope you're well.

I have a quick question regarding the LR schedulers especially the "cosine" scheduler.

From what I can see in the args defaults in the training script, we set:

parser.add_argument('--lr-cycle-limit', type=int, default=1, metavar='N',
                    help='learning rate cycle limit')

That is lr_cycle_limit is set to 1. Now as per my understanding, in SGDR we restart the learning rate after some epochs so that the LR schedule looks something like:

However, since the defaults in training script set lr_cycle_limit to 1, we never actually restart the LR schedule. Therefore, based on the implementation, the LR schedule ends up looking something like:

I am wondering if we should update the defaults or would it make sense to have a Callback that tracks the learning rate similar to fastai and plots the LR for more visibility of the various schedulers.

I have shared my implementation in this nb as gist - https://gist.github.com/amaarora/8f328bb19d2d262e5d172601d00fa330.

Keen to hear your thoughts and understand whether my understanding of schedulers in timm is correct? Thanks! :)

[amaarora]

FYI - I am in the process of documenting the various defaults and available schedulers as part of timmdocs.

Recently, I just added some more documentation for the available Optimizers too - https://fastai.github.io/timmdocs/Optimizers. :)

[amaarora]

Also, further reading and looking into the source code suggests that the annealing is different from Pytorch's implementation where we anneal the learning rate at the start of the next epoch instead of per mini-batch?

[linhduongtuan]

Hi @ross, it is great news to see timmdocs. The documents in both timmdocs and fastai are explained concisely and precisely. I love reading them. Best regards Linh

…

On 7 Mar 2021, at 07:36, Aman Arora ***@***.***> wrote:  FYI - I'll am in the process of documenting the various defaults and available schedulers as part of timmdocs. Recently, I just added some more documentation for the available Optimizers too - https://fastai.github.io/timmdocs/Optimizers. :) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[rwightman]

Thank @amaarora, the timmdocs are his effort.

RE l0c, I think I can do better, not super happy with end result likely will toss, don't have numbers handy but you can try.