Docs/update 20 12 (#1018)

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

* +

Author: Scitator

CHANGELOG.md

@@ -9,16 +9,24 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ### Added
 
+- CVS Logger ([#1005](https://github.com/catalyst-team/catalyst/pull/1005))
 - ([#1002](https://github.com/catalyst-team/catalyst/pull/1002))
-    - a few docs 
+    - a few docs
+- ([#998](https://github.com/catalyst-team/catalyst/pull/998))
+    - ``reciprocal_rank`` metric 
+    - unified recsys metrics preprocessing 
+-  ([#1018](https://github.com/catalyst-team/catalyst/pull/1014))
+    - readme examples for all supported metrics under ``catalyst.metrics``
+    - ``wrap_metric_fn_with_activation`` for model outputs wrapping with activation
+    -  extra tests for metrics
 
 ### Changed
 
-
+- docs update ([#1000](https://github.com/catalyst-team/catalyst/pull/1000))
+- ``AMPOptimizerCallback`` and ``OptimizerCallback`` were merged ([#1007](https://github.com/catalyst-team/catalyst/pull/1007))
 - ([#1017](https://github.com/catalyst-team/catalyst/pull/1017))
     - fixed bug in `SchedulerCallback`
     - Log LRs and momentums for all param groups, not only for the first one
-
 - ([#1002](https://github.com/catalyst-team/catalyst/pull/1002))
     - ``tensorboard, ipython, matplotlib, pandas, scikit-learn`` moved to optional requirements
     - ``PerplexityMetricCallback`` moved to ``catalyst.callbacks`` from ``catalyst.contrib.callbacks``
@@ -29,6 +37,12 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
     - ``catalyst.utils``, ``catalyst.data.utils`` and ``catalyst.contrib.utils`` restructured
     - ``ReaderSpec`` renamed to ``IReader``
     - ``SupervisedExperiment`` renamed to ``AutoCallbackExperiment``
+- gain functions renamed for ``dcg``/``ndcg`` metrics ([#998](https://github.com/catalyst-team/catalyst/pull/998))
+- ([#1014](https://github.com/catalyst-team/catalyst/pull/1014))
+    - requirements respecification: ``catalyst[cv]``, ``catalyst[dev]``, ``catalyst[log]``, ``catalyst[ml]``, ``catalyst[nlp]``,``catalyst[tune]``
+    - settings respecification
+    - extra tests for settings
+    - contrib refactoring
 
 ### Removed
 
@@ -38,11 +52,17 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
     - ``catalyst.data.utils``
     - unnecessary ``catalyst.tools.meters``
     - todos for unnecessary docs
+- ([#1014](https://github.com/catalyst-team/catalyst/pull/1014))
+    - transformers-based contrib (too unstable)
+- ([#1018](https://github.com/catalyst-team/catalyst/pull/1014))
+    - ClasswiseIouCallback/ClasswiseJaccardCallback as deprecated on (should be refactored in future releases)
+
 
 
 ### Fixed
 
 - prevented modifying config during the experiment and runner initialization ([#1004](https://github.com/catalyst-team/catalyst/pull/1004))
+- a few test for RecSys MAP computation ([#1018](https://github.com/catalyst-team/catalyst/pull/1014))
 
 
 
@@ -63,6 +83,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 - update Catalyst version to `20.10.1` for tutorials ([#967](https://github.com/catalyst-team/catalyst/pull/967))
 - added link to dl-course ([#967](https://github.com/catalyst-team/catalyst/pull/967))
+- ``IRunner`` -> simplified ``IRunner`` ([#984](https://github.com/catalyst-team/catalyst/pull/984))
 - docs were restructured ([#985](https://github.com/catalyst-team/catalyst/pull/985))
 - `set_global_seed` moved from `utils.seed` to `utils.misc` ([#986](https://github.com/catalyst-team/catalyst/pull/986))
 
@@ -177,7 +198,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - kornia augmentations `BatchTransformCallback` ([#862](https://github.com/catalyst-team/catalyst/issues/862))
 - `average_precision` and `mean_average_precision` metrics ([#883](https://github.com/catalyst-team/catalyst/pull/883))
 - `MultiLabelAccuracyCallback`, `AveragePrecisionCallback` and `MeanAveragePrecisionCallback` callbacks ([#883](https://github.com/catalyst-team/catalyst/pull/883))
-- minimal examples for multi-class and milti-label classification ([#883](https://github.com/catalyst-team/catalyst/pull/883))
+- minimal examples for multiclass and multilabel classification ([#883](https://github.com/catalyst-team/catalyst/pull/883))
 - experimental TPU support ([#893](https://github.com/catalyst-team/catalyst/pull/893))
 - add `Imagenette`, `Imagewoof`, and `Imagewang` datasets ([#902](https://github.com/catalyst-team/catalyst/pull/902))
 - `IMetricCallback`, `IBatchMetricCallback`, `ILoaderMetricCallback`, `BatchMetricCallback`, `LoaderMetricCallback` abstractions ([#897](https://github.com/catalyst-team/catalyst/pull/897))

LICENSE

@@ -1,4 +1,4 @@
-Copyright 2018-2020 Sergey Kolesnikov
+Copyright 2018- Sergey Kolesnikov
 
                                  Apache License
                            Version 2.0, January 2004

README.md

@@ -216,7 +216,7 @@ runner.train(
 
 
 <details>
-<summary>ML - multi-class classification</summary>
+<summary>ML - multiclass classification</summary>
 <p>
 
 ```python
@@ -258,7 +258,7 @@ runner.train(
 
 
 <details>
-<summary>ML - multi-label classification</summary>
+<summary>ML - multilabel classification</summary>
 <p>
 
 ```python
@@ -759,7 +759,7 @@ runner.train(
 </details>
 
 <details>
-<summary>ML - multi-class classification (fp16 training version)</summary>
+<summary>ML - multiclass classification (fp16 training version)</summary>
 <p>
 
 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1q8BPg1XpQn2J5vWV9OYKSBo-k9wA2jYS?usp=sharing)
@@ -804,7 +804,7 @@ runner.train(
 </details>
 
 <details>
-<summary>ML - multi-class classification (advanced fp16 training version)</summary>
+<summary>ML - multiclass classification (advanced fp16 training version)</summary>
 <p>
 
 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1q8BPg1XpQn2J5vWV9OYKSBo-k9wA2jYS?usp=sharing)
@@ -976,7 +976,7 @@ utils.distributed_cmd_run(train)
 </details>
 
 <details>
-<summary>ML - multi-class classification (TPU version)</summary>
+<summary>ML - multiclass classification (TPU version)</summary>
 <p>
 
 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1AhvNzTRb3gd3AYhzUfm3dzw8TddlsfhD?usp=sharing)

catalyst/__version__.py

@@ -1 +1 @@
-__version__ = "20.11.1"
+__version__ = "20.12"

catalyst/callbacks/meter.py

@@ -44,7 +44,7 @@ def __init__(
                 If None, defaults to indices for each class, starting from 0.
             num_classes: Number of classes; must be > 1
             activation: An torch.nn activation applied to the logits.
-                Must be one of ['none', 'Sigmoid', 'Softmax2d']
+                Must be one of ['none', 'Sigmoid', 'Softmax']
         """
         super().__init__(CallbackOrder.metric)
         self.metric_names = metric_names

catalyst/callbacks/metric.py

@@ -243,6 +243,9 @@ def __init__(
             **metrics_kwargs: extra metric params
                 to pass for metric computation
         """
+        # @TODO:
+        # - add ~(transform: Callable = lambda x: x) for runner
+        # - add ~(transform: Callable = lambda x: x) for metrics output
         super().__init__(
             prefix=prefix,
             input_key=input_key,

catalyst/callbacks/metrics/__init__.py

@@ -6,22 +6,22 @@
 )
 from catalyst.callbacks.metrics.auc import AUCCallback
 from catalyst.callbacks.metrics.cmc_score import CMCScoreCallback
+
 from catalyst.callbacks.metrics.dice import (
     DiceCallback,
     MultiClassDiceMetricCallback,
     MulticlassDiceMetricCallback,
 )
+
 from catalyst.callbacks.metrics.f1_score import F1ScoreCallback
 from catalyst.callbacks.metrics.iou import (
     IouCallback,
     JaccardCallback,
-    ClasswiseIouCallback,
-    ClasswiseJaccardCallback,
 )
 from catalyst.callbacks.metrics.mrr import MRRCallback
 from catalyst.callbacks.metrics.perplexity import (
-    PerplexityCallback,
     PerplexityMetricCallback,
+    PerplexityCallback,
 )
 from catalyst.callbacks.metrics.ppv_tpr_f1 import (
     PrecisionRecallF1ScoreCallback,

catalyst/callbacks/metrics/accuracy.py

@@ -1,20 +1,21 @@
 from typing import List
 
 from catalyst.callbacks.metric import BatchMetricCallback
-from catalyst.metrics.accuracy import accuracy, multi_label_accuracy
+from catalyst.metrics.accuracy import accuracy, multilabel_accuracy
 from catalyst.metrics.functional import (
     get_default_topk_args,
+    wrap_metric_fn_with_activation,
     wrap_topk_metric2dict,
 )
 
 
 class AccuracyCallback(BatchMetricCallback):
     """Accuracy metric callback.
 
-    Computes multi-class accuracy@topk for the specified values of `topk`.
+    Computes multiclass accuracy@topk for the specified values of `topk`.
 
     .. note::
-        For multi-label accuracy please use
+        For multilabel accuracy please use
         `catalyst.callbacks.metrics.MultiLabelAccuracyCallback`
     """
 
@@ -23,7 +24,6 @@ def __init__(
         input_key: str = "targets",
         output_key: str = "logits",
         prefix: str = "accuracy",
-        multiplier: float = 1.0,
         topk_args: List[int] = None,
         num_classes: int = None,
         accuracy_args: List[int] = None,
@@ -42,8 +42,6 @@ def __init__(
                 [1, 3, 5] - accuracy at 1, 3 and 5
             num_classes: number of classes to calculate ``topk_args``
                 if ``accuracy_args`` is None
-            activation: An torch.nn activation applied to the outputs.
-                Must be one of ``"none"``, ``"Sigmoid"``, or ``"Softmax"``
         """
         topk_args = (
             topk_args or accuracy_args or get_default_topk_args(num_classes)
@@ -54,27 +52,26 @@ def __init__(
             metric_fn=wrap_topk_metric2dict(accuracy, topk_args=topk_args),
             input_key=input_key,
             output_key=output_key,
-            multiplier=multiplier,
             **kwargs,
         )
 
 
 class MultiLabelAccuracyCallback(BatchMetricCallback):
     """Accuracy metric callback.
-    Computes multi-class accuracy@topk for the specified values of `topk`.
+    Computes multiclass accuracy@topk for the specified values of `topk`.
 
     .. note::
-        For multi-label accuracy please use
+        For multilabel accuracy please use
         `catalyst.callbacks.metrics.MultiLabelAccuracyCallback`
     """
 
     def __init__(
         self,
         input_key: str = "targets",
         output_key: str = "logits",
-        prefix: str = "multi_label_accuracy",
-        threshold: float = None,
+        prefix: str = "multilabel_accuracy",
         activation: str = "Sigmoid",
+        threshold: float = None,
     ):
         """
         Args:
@@ -89,11 +86,12 @@ def __init__(
         """
         super().__init__(
             prefix=prefix,
-            metric_fn=multi_label_accuracy,
+            metric_fn=wrap_metric_fn_with_activation(
+                metric_fn=multilabel_accuracy, activation=activation
+            ),
             input_key=input_key,
             output_key=output_key,
             threshold=threshold,
-            activation=activation,
         )
 
 

catalyst/callbacks/metrics/auc.py

@@ -2,22 +2,25 @@
 
 from catalyst.callbacks.metric import LoaderMetricCallback
 from catalyst.metrics.auc import auc
-from catalyst.metrics.functional import wrap_class_metric2dict
+from catalyst.metrics.functional import (
+    wrap_class_metric2dict,
+    wrap_metric_fn_with_activation,
+)
 
 
 class AUCCallback(LoaderMetricCallback):
     """Calculates the AUC  per class for each loader.
 
     .. note::
-        Currently, supports binary and multi-label cases.
+        Currently, supports binary and multilabel cases.
     """
 
     def __init__(
         self,
         input_key: str = "targets",
         output_key: str = "logits",
         prefix: str = "auc",
-        multiplier: float = 1.0,
+        activation: str = "Sigmoid",
         class_args: List[str] = None,
         **kwargs,
     ):
@@ -27,17 +30,22 @@ def __init__(
                 specifies our ``y_true``.
             output_key: output key to use for auc calculation;
                 specifies our ``y_pred``.
-            prefix: metric's name.
+            prefix: key for the metric's name
             multiplier: scale factor for the metric.
+            activation: An torch.nn activation applied to the outputs.
+                Must be one of ``'none'``, ``'Sigmoid'``, or ``'Softmax'``
             class_args: class names to display in the logs.
                 If None, defaults to indices for each class, starting from 0
         """
+        metric_fn = wrap_metric_fn_with_activation(
+            metric_fn=auc, activation=activation
+        )
+        metric_fn = wrap_class_metric2dict(metric_fn, class_args=class_args)
         super().__init__(
             prefix=prefix,
-            metric_fn=wrap_class_metric2dict(auc, class_args=class_args),
+            metric_fn=metric_fn,
             input_key=input_key,
             output_key=output_key,
-            multiplier=multiplier,
             **kwargs,
         )
 

catalyst/callbacks/metrics/dice.py

@@ -9,6 +9,7 @@
 )
 from catalyst.core.callback import Callback, CallbackOrder
 from catalyst.metrics.dice import calculate_dice, dice
+from catalyst.metrics.functional import wrap_metric_fn_with_activation
 
 if TYPE_CHECKING:
     from catalyst.core.runner import IRunner
@@ -26,17 +27,16 @@ class DiceCallback(BatchMetricCallback):
         eps: epsilon to avoid zero division
         threshold: threshold for outputs binarization
         activation: An torch.nn activation applied to the outputs.
-            Must be one of ``'none'``, ``'Sigmoid'``, ``'Softmax2d'``
+            Must be one of ``'none'``, ``'Sigmoid'``, ``'Softmax'``
     """
 
     def __init__(
         self,
         input_key: str = "targets",
         output_key: str = "logits",
         prefix: str = "dice",
-        eps: float = 1e-7,
-        threshold: float = None,
         activation: str = "Sigmoid",
+        **kwargs,
     ):
         """
         Args:
@@ -45,25 +45,25 @@ def __init__(
             output_key: output key to use for iou calculation;
                 specifies our ``y_pred``
             prefix: key to store in logs
+            activation: An torch.nn activation applied to the model outputs.
+                Must be one of ``'none'``, ``'Sigmoid'``, ``'Softmax'``
             eps: epsilon to avoid zero division
             threshold: threshold for outputs binarization
-            activation: An torch.nn activation applied to the outputs.
-                Must be one of ``'none'``, ``'Sigmoid'``, ``'Softmax2d'``
         """
         super().__init__(
             prefix=prefix,
-            metric_fn=dice,
+            metric_fn=wrap_metric_fn_with_activation(
+                metric_fn=dice, activation=activation
+            ),
             input_key=input_key,
             output_key=output_key,
-            eps=eps,
-            threshold=threshold,
-            activation=activation,
+            **kwargs,
         )
 
 
 class MultiClassDiceMetricCallback(Callback):
     """
-    Global Multi-Class Dice Metric Callback: calculates the exact
+    Global multiclass Dice Metric Callback: calculates the exact
     dice score across multiple batches. This callback is good for getting
     the dice score with small batch sizes where the batchwise dice is noisier.
     """