Skip to content

Commit 8331fac

Browse files
mrT23mrT23
authored
Merge pull request #2 from rwightman/master
merge original
2 parents 8a63c1a + 70ae7f0 commit 8331fac

File tree

156 files changed

+14925
-4925
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

156 files changed

+14925
-4925
lines changed

‎.github/workflows/tests.yml

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: Python tests
2+
3+
on:
4+
push:
5+
branches: [ master ]
6+
pull_request:
7+
branches: [ master ]
8+
9+
env:
10+
OMP_NUM_THREADS: 2
11+
MKL_NUM_THREADS: 2
12+
13+
jobs:
14+
test:
15+
name: Run tests on ${{ matrix.os }} with Python ${{ matrix.python }}
16+
strategy:
17+
matrix:
18+
os: [ubuntu-latest, macOS-latest]
19+
python: ['3.8']
20+
torch: ['1.5.0']
21+
torchvision: ['0.6.0']
22+
runs-on: ${{ matrix.os }}
23+
24+
steps:
25+
- uses: actions/checkout@v2
26+
- name: Set up Python ${{ matrix.python }}
27+
uses: actions/setup-python@v1
28+
with:
29+
python-version: ${{ matrix.python }}
30+
- name: Install testing dependencies
31+
run: |
32+
python -m pip install --upgrade pip
33+
pip install pytest pytest-timeout
34+
- name: Install torch on mac
35+
if: startsWith(matrix.os, 'macOS')
36+
run: pip install --no-cache-dir torch==${{ matrix.torch }} torchvision==${{ matrix.torchvision }}
37+
- name: Install torch on ubuntu
38+
if: startsWith(matrix.os, 'ubuntu')
39+
run: pip install --no-cache-dir torch==${{ matrix.torch }}+cpu torchvision==${{ matrix.torchvision }}+cpu -f https://download.pytorch.org/whl/torch_stable.html
40+
- name: Install requirements
41+
run: |
42+
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
43+
pip install --no-cache-dir git+https://github.com/mapillary/[email protected]
44+
- name: Run tests
45+
run: |
46+
pytest -vv --durations=0 ./tests

‎MANIFEST.in

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
include timm/models/pruned/*.txt
2+

‎README.md

Lines changed: 175 additions & 418 deletions
Large diffs are not rendered by default.

‎avg_checkpoints.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
EMA (exponential moving average) of the model weights or performing SWA (stochastic
1010
weight averaging), but post-training.
1111
12-
Hacked together by Ross Wightman (https://github.com/rwightman)
12+
Hacked together by / Copyright 2020 Ross Wightman (https://github.com/rwightman)
1313
"""
1414
import torch
1515
import argparse
@@ -103,7 +103,11 @@ def main():
103103
v = v.clamp(float32_info.min, float32_info.max)
104104
final_state_dict[k] = v.to(dtype=torch.float32)
105105

106-
torch.save(final_state_dict, args.output)
106+
try:
107+
torch.save(final_state_dict, args.output, _use_new_zipfile_serialization=False)
108+
except:
109+
torch.save(final_state_dict, args.output)
110+
107111
with open(args.output, 'rb') as f:
108112
sha_hash = hashlib.sha256(f.read()).hexdigest()
109113
print("=> Saved state_dict to '{}, SHA256: {}'".format(args.output, sha_hash))

‎clean_checkpoint.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
and outputs a CPU tensor checkpoint with only the `state_dict` along with SHA256
66
calculation for model zoo compatibility.
77
8-
Hacked together by Ross Wightman (https://github.com/rwightman)
8+
Hacked together by / Copyright 2020 Ross Wightman (https://github.com/rwightman)
99
"""
1010
import torch
1111
import argparse
@@ -57,7 +57,11 @@ def main():
5757
new_state_dict[name] = v
5858
print("=> Loaded state_dict from '{}'".format(args.checkpoint))
5959

60-
torch.save(new_state_dict, _TEMP_NAME)
60+
try:
61+
torch.save(new_state_dict, _TEMP_NAME, _use_new_zipfile_serialization=False)
62+
except:
63+
torch.save(new_state_dict, _TEMP_NAME)
64+
6165
with open(_TEMP_NAME, 'rb') as f:
6266
sha_hash = hashlib.sha256(f.read()).hexdigest()
6367

‎docs/archived_changes.md

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Archived Changes
2+
3+
### Feb 29, 2020
4+
* New MobileNet-V3 Large weights trained from stratch with this code to 75.77% top-1
5+
* IMPORTANT CHANGE - default weight init changed for all MobilenetV3 / EfficientNet / related models
6+
* overall results similar to a bit better training from scratch on a few smaller models tried
7+
* performance early in training seems consistently improved but less difference by end
8+
* set `fix_group_fanout=False` in `_init_weight_goog` fn if you need to reproducte past behaviour
9+
* Experimental LR noise feature added applies a random perturbation to LR each epoch in specified range of training
10+
11+
### Feb 18, 2020
12+
* Big refactor of model layers and addition of several attention mechanisms. Several additions motivated by 'Compounding the Performance Improvements...' (https://arxiv.org/abs/2001.06268):
13+
* Move layer/module impl into `layers` subfolder/module of `models` and organize in a more granular fashion
14+
* ResNet downsample paths now properly support dilation (output stride != 32) for avg_pool ('D' variant) and 3x3 (SENets) networks
15+
* Add Selective Kernel Nets on top of ResNet base, pretrained weights
16+
* skresnet18 - 73% top-1
17+
* skresnet34 - 76.9% top-1
18+
* skresnext50_32x4d (equiv to SKNet50) - 80.2% top-1
19+
* ECA and CECA (circular padding) attention layer contributed by [Chris Ha](https://github.com/VRandme)
20+
* CBAM attention experiment (not the best results so far, may remove)
21+
* Attention factory to allow dynamically selecting one of SE, ECA, CBAM in the `.se` position for all ResNets
22+
* Add DropBlock and DropPath (formerly DropConnect for EfficientNet/MobileNetv3) support to all ResNet variants
23+
* Full dataset results updated that incl NoisyStudent weights and 2 of the 3 SK weights
24+
25+
### Feb 12, 2020
26+
* Add EfficientNet-L2 and B0-B7 NoisyStudent weights ported from [Tensorflow TPU](https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet)
27+
28+
### Feb 6, 2020
29+
* Add RandAugment trained EfficientNet-ES (EdgeTPU-Small) weights with 78.1 top-1. Trained by [Andrew Lavin](https://github.com/andravin) (see Training section for hparams)
30+
31+
### Feb 1/2, 2020
32+
* Port new EfficientNet-B8 (RandAugment) weights, these are different than the B8 AdvProp, different input normalization.
33+
* Update results csv files on all models for ImageNet validation and three other test sets
34+
* Push PyPi package update
35+
36+
### Jan 31, 2020
37+
* Update ResNet50 weights with a new 79.038 result from further JSD / AugMix experiments. Full command line for reproduction in training section below.
38+
39+
### Jan 11/12, 2020
40+
* Master may be a bit unstable wrt to training, these changes have been tested but not all combos
41+
* Implementations of AugMix added to existing RA and AA. Including numerous supporting pieces like JSD loss (Jensen-Shannon divergence + CE), and AugMixDataset
42+
* SplitBatchNorm adaptation layer added for implementing Auxiliary BN as per AdvProp paper
43+
* ResNet-50 AugMix trained model w/ 79% top-1 added
44+
* `seresnext26tn_32x4d` - 77.99 top-1, 93.75 top-5 added to tiered experiment, higher img/s than 't' and 'd'
45+
46+
### Jan 3, 2020
47+
* Add RandAugment trained EfficientNet-B0 weight with 77.7 top-1. Trained by [Michael Klachko](https://github.com/michaelklachko) with this code and recent hparams (see Training section)
48+
* Add `avg_checkpoints.py` script for post training weight averaging and update all scripts with header docstrings and shebangs.
49+
50+
### Dec 30, 2019
51+
* Merge [Dushyant Mehta's](https://github.com/mehtadushy) PR for SelecSLS (Selective Short and Long Range Skip Connections) networks. Good GPU memory consumption and throughput. Original: https://github.com/mehtadushy/SelecSLS-Pytorch
52+
53+
### Dec 28, 2019
54+
* Add new model weights and training hparams (see Training Hparams section)
55+
* `efficientnet_b3` - 81.5 top-1, 95.7 top-5 at default res/crop, 81.9, 95.8 at 320x320 1.0 crop-pct
56+
* trained with RandAugment, ended up with an interesting but less than perfect result (see training section)
57+
* `seresnext26d_32x4d`- 77.6 top-1, 93.6 top-5
58+
* deep stem (32, 32, 64), avgpool downsample
59+
* stem/dowsample from bag-of-tricks paper
60+
* `seresnext26t_32x4d`- 78.0 top-1, 93.7 top-5
61+
* deep tiered stem (24, 48, 64), avgpool downsample (a modified 'D' variant)
62+
* stem sizing mods from Jeremy Howard and fastai devs discussing ResNet architecture experiments
63+
64+
### Dec 23, 2019
65+
* Add RandAugment trained MixNet-XL weights with 80.48 top-1.
66+
* `--dist-bn` argument added to train.py, will distribute BN stats between nodes after each train epoch, before eval
67+
68+
### Dec 4, 2019
69+
* Added weights from the first training from scratch of an EfficientNet (B2) with my new RandAugment implementation. Much better than my previous B2 and very close to the official AdvProp ones (80.4 top-1, 95.08 top-5).
70+
71+
### Nov 29, 2019
72+
* Brought EfficientNet and MobileNetV3 up to date with my https://github.com/rwightman/gen-efficientnet-pytorch code. Torchscript and ONNX export compat excluded.
73+
* AdvProp weights added
74+
* Official TF MobileNetv3 weights added
75+
* EfficientNet and MobileNetV3 hook based 'feature extraction' classes added. Will serve as basis for using models as backbones in obj detection/segmentation tasks. Lots more to be done here...
76+
* HRNet classification models and weights added from https://github.com/HRNet/HRNet-Image-Classification
77+
* Consistency in global pooling, `reset_classifer`, and `forward_features` across models
78+
* `forward_features` always returns unpooled feature maps now
79+
* Reasonable chance I broke something... let me know
80+
81+
### Nov 22, 2019
82+
* Add ImageNet training RandAugment implementation alongside AutoAugment. PyTorch Transform compatible format, using PIL. Currently training two EfficientNet models from scratch with promising results... will update.
83+
* `drop-connect` cmd line arg finally added to `train.py`, no need to hack model fns. Works for efficientnet/mobilenetv3 based models, ignored otherwise.

‎docs/changes.md

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# Recent Changes
2+
3+
### Aug 12, 2020
4+
* New/updated weights from training experiments
5+
* EfficientNet-B3 - 82.1 top-1 (vs 81.6 for official with AA and 81.9 for AdvProp)
6+
* RegNetY-3.2GF - 82.0 top-1 (78.9 from official ver)
7+
* CSPResNet50 - 79.6 top-1 (76.6 from official ver)
8+
* Add CutMix integrated w/ Mixup. See [pull request](https://github.com/rwightman/pytorch-image-models/pull/218) for some usage examples
9+
* Some fixes for using pretrained weights with `in_chans` != 3 on several models.
10+
11+
### Aug 5, 2020
12+
Universal feature extraction, new models, new weights, new test sets.
13+
14+
* All models support the `features_only=True` argument for `create_model` call to return a network that extracts features from the deepest layer at each stride.
15+
* New models
16+
* CSPResNet, CSPResNeXt, CSPDarkNet, DarkNet
17+
* ReXNet
18+
* (Modified Aligned) Xception41/65/71 (a proper port of TF models)
19+
* New trained weights
20+
* SEResNet50 - 80.3 top-1
21+
* CSPDarkNet53 - 80.1 top-1
22+
* CSPResNeXt50 - 80.0 top-1
23+
* DPN68b - 79.2 top-1
24+
* EfficientNet-Lite0 (non-TF ver) - 75.5 (submitted by [@hal-314](https://github.com/hal-314))
25+
* Add 'real' labels for ImageNet and ImageNet-Renditions test set, see [`results/README.md`](results/README.md)
26+
* Test set ranking/top-n diff script by [@KushajveerSingh](https://github.com/KushajveerSingh)
27+
* Train script and loader/transform tweaks to punch through more aug arguments
28+
* README and documentation overhaul. See initial (WIP) documentation at https://rwightman.github.io/pytorch-image-models/
29+
* adamp and sgdp optimizers added by [@hellbell](https://github.com/hellbell)
30+
31+
### June 11, 2020
32+
Bunch of changes:
33+
34+
* DenseNet models updated with memory efficient addition from torchvision (fixed a bug), blur pooling and deep stem additions
35+
* VoVNet V1 and V2 models added, 39 V2 variant (ese_vovnet_39b) trained to 79.3 top-1
36+
* Activation factory added along with new activations:
37+
* select act at model creation time for more flexibility in using activations compatible with scripting or tracing (ONNX export)
38+
* hard_mish (experimental) added with memory-efficient grad, along with ME hard_swish
39+
* context mgr for setting exportable/scriptable/no_jit states
40+
* Norm + Activation combo layers added with initial trial support in DenseNet and VoVNet along with impl of EvoNorm and InplaceAbn wrapper that fit the interface
41+
* Torchscript works for all but two of the model types as long as using Pytorch 1.5+, tests added for this
42+
* Some import cleanup and classifier reset changes, all models will have classifier reset to nn.Identity on reset_classifer(0) call
43+
* Prep for 0.1.28 pip release
44+
45+
### May 12, 2020
46+
* Add ResNeSt models (code adapted from https://github.com/zhanghang1989/ResNeSt, paper https://arxiv.org/abs/2004.08955))
47+
48+
### May 3, 2020
49+
* Pruned EfficientNet B1, B2, and B3 (https://arxiv.org/abs/2002.08258) contributed by [Yonathan Aflalo](https://github.com/yoniaflalo)
50+
51+
### May 1, 2020
52+
* Merged a number of execellent contributions in the ResNet model family over the past month
53+
* BlurPool2D and resnetblur models initiated by [Chris Ha](https://github.com/VRandme), I trained resnetblur50 to 79.3.
54+
* TResNet models and SpaceToDepth, AntiAliasDownsampleLayer layers by [mrT23](https://github.com/mrT23)
55+
* ecaresnet (50d, 101d, light) models and two pruned variants using pruning as per (https://arxiv.org/abs/2002.08258) by [Yonathan Aflalo](https://github.com/yoniaflalo)
56+
* 200 pretrained models in total now with updated results csv in results folder
57+
58+
### April 5, 2020
59+
* Add some newly trained MobileNet-V2 models trained with latest h-params, rand augment. They compare quite favourably to EfficientNet-Lite
60+
* 3.5M param MobileNet-V2 100 @ 73%
61+
* 4.5M param MobileNet-V2 110d @ 75%
62+
* 6.1M param MobileNet-V2 140 @ 76.5%
63+
* 5.8M param MobileNet-V2 120d @ 77.3%
64+
65+
### March 18, 2020
66+
* Add EfficientNet-Lite models w/ weights ported from [Tensorflow TPU](https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite)
67+
* Add RandAugment trained ResNeXt-50 32x4d weights with 79.8 top-1. Trained by [Andrew Lavin](https://github.com/andravin) (see Training section for hparams)

0 commit comments

Comments
 (0)