Add wandb project name argument and allow change wandb run name

[caojiaolong]

Currently, it is not possible to specify a custom name for a run in WandB; the run name is generated randomly. The args.experiment argument only determines the project name in WandB. Could we introduce a new argument, args.wandb_project, to control the project name and repurpose args.experiment to define the run name? This would ensure the run name aligns with the output directory.

[rwightman]

@caojiaolong so I don't use wandb enough myself to have formed any preferences ... but others are using it with timm and have not brought this up. To not break old behaviour, wondering if the project_name arg should be added as you have, default to None. If it is None, then behave as now and use experiment as project name, otherwise if it's set do as you have and set project=args.project_name and name=args.experiment?

[caojiaolong]

@rwightman Thank you for your quick reply. While it’s easy to modify the code to maintain backward compatibility (like below), I don’t think it’s necessary, as the old behavior doesn’t make much sense.

            wandb.init(
                project=args.wandb_project if args.wandb_project else args.experiment,
                name=args.experiment if args.wandb_project else None,
                config=args,
                tags=args.wandb_tags,
                resume="must" if args.wandb_resume_id else None,
                id=args.wandb_resume_id if args.wandb_resume_id else None,
            )

In my understanding, a single WandB project can contain multiple experiments, and the WandB run name should correspond to the experiment name (which is likely why the output directory is based on args.experiment). Therefore, different experiments should have distinct run names rather than separate project names. However, in the current implementation of timm, changing args.experiment creates a new project in WandB, which is not logical in this context. It might be time to correct this behavior.

[caojiaolong]

Additionally, in this comment (#550 (comment)), I noticed that you suggested using args.experiment as the project name. However, doesn’t args.experiment represent more of a single experiment rather than an entire project?

[rwightman]

@caojiaolong yeah re that original PR, I wasn't very familiar with wandb, the author was working at wandb at the time I believe, I assumed best practices re name vs project, etc where known

I'll take a look and actually try some runs... I used wandb more with open_clip (that was after it was added here) and that was working decently, I'll take a look at what we did there too and add anything that may have been different in addition to these changes.

[caojiaolong]

@rwightman Thank you for your reply. I’m not very familiar with open_clip, so I quickly reviewed its code and noticed some differences:

https://github.com/mlfoundations/open_clip/blob/63fbfd857c83af619e6a0a8344635ebb4a151a96/src/open_clip_train/main.py#L440C7-L448C10

As you can see, in open_clip, different experiments are distinguished using args.name, and the WandB project name is explicitly specified via the args.wandb_project_name argument. This allows you to simply modify args.name to run different experiments within the same project. Unfortunately, this flexibility is not currently available in timm.

[rwightman]

@caojiaolong thanks, I did that merge already and made one small change after testing, pushed the wandb init into the previous is_primary block and use the resolved exp_name. Appears to work well and yes, better than before. Though do need to add step logging... will work that into some other changes