Skip to content

Factory

danling.metric.factory

binary_metrics

Python
binary_metrics(ignore_index: int | None = -100, **kwargs)

Create a pre-configured Metrics instance for binary classification tasks.

This factory function returns a Metrics object with a standard set of binary classification metrics, including: - AUROC (Area Under ROC Curve) - AUPRC (Area Under Precision-Recall Curve) - Accuracy - MCC (Matthews Correlation Coefficient) - F1 Score

The returned Metrics instance is ready to use with model predictions/logits and binary labels (0/1 or False/True).

Parameters:

Name Type Description Default

ignore_index

int | None

Value in the target to ignore (e.g., padding).

 -100

**kwargs

Additional metric functions to include or override default metrics

 {}

Returns:

Name Type Description
Metrics

A configured Metrics instance for binary classification
Source code in danling/metric/factory.py
Python
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
def binary_metrics(ignore_index: int | None = -100, **kwargs):
    """
    Create a pre-configured Metrics instance for binary classification tasks.

    This factory function returns a Metrics object with a standard set of binary
    classification metrics, including:
    - AUROC (Area Under ROC Curve)
    - AUPRC (Area Under Precision-Recall Curve)
    - Accuracy
    - MCC (Matthews Correlation Coefficient)
    - F1 Score

    The returned Metrics instance is ready to use with model predictions/logits
    and binary labels (0/1 or False/True).

    Args:
        ignore_index: Value in the target to ignore (e.g., padding).
        **kwargs: Additional metric functions to include or override default metrics

    Returns:
        Metrics: A configured Metrics instance for binary classification
    """
    lazy_import.check()
    return Metrics(
        auroc=partial(auroc, task="binary", preprocess=False),
        auprc=partial(auprc, task="binary", preprocess=False),
        acc=partial(accuracy, task="binary", preprocess=False),
        mcc=partial(mcc, task="binary", preprocess=False),
        f1=partial(f1_score, task="binary", preprocess=False),
        preprocess=partial(preprocess_binary, ignore_index=ignore_index),
        **kwargs,
    )

multiclass_metrics

Python
multiclass_metrics(
    num_classes: int,
    ignore_index: int | None = -100,
    **kwargs
)

Create a pre-configured Metrics instance for multiclass classification tasks.

This factory function returns a Metrics object with a standard set of multiclass classification metrics, including: - AUROC (per class and macro-averaged) - AUPRC (per class and macro-averaged) - Accuracy - MCC (Matthews Correlation Coefficient) - F1 Score (macro-averaged)

The returned Metrics instance is ready to use with model logits (shape [batch_size, num_classes]) and class labels (shape [batch_size]).

Parameters:

Name Type Description Default

num_classes

int

Number of classes in the classification task

 required

ignore_index

int | None

Value in the target to ignore (e.g., padding).

 -100

**kwargs

Additional metric functions to include or override default metrics

 {}

Returns:

Name Type Description
Metrics

A configured Metrics instance for multiclass classification
Source code in danling/metric/factory.py
Python
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
def multiclass_metrics(num_classes: int, ignore_index: int | None = -100, **kwargs):
    """
    Create a pre-configured Metrics instance for multiclass classification tasks.

    This factory function returns a Metrics object with a standard set of multiclass
    classification metrics, including:
    - AUROC (per class and macro-averaged)
    - AUPRC (per class and macro-averaged)
    - Accuracy
    - MCC (Matthews Correlation Coefficient)
    - F1 Score (macro-averaged)

    The returned Metrics instance is ready to use with model logits (shape [batch_size, num_classes])
    and class labels (shape [batch_size]).

    Args:
        num_classes: Number of classes in the classification task
        ignore_index: Value in the target to ignore (e.g., padding).
        **kwargs: Additional metric functions to include or override default metrics

    Returns:
        Metrics: A configured Metrics instance for multiclass classification
    """
    lazy_import.check()
    return Metrics(
        auroc=partial(auroc, task="multiclass", num_classes=num_classes, preprocess=False),
        auprc=partial(auprc, task="multiclass", num_classes=num_classes, preprocess=False),
        acc=partial(accuracy, task="multiclass", num_classes=num_classes, preprocess=False),
        mcc=partial(mcc, task="multiclass", num_classes=num_classes, preprocess=False),
        f1=partial(f1_score, task="multiclass", num_classes=num_classes, preprocess=False),
        preprocess=partial(preprocess_multiclass, num_classes=num_classes, ignore_index=ignore_index),
        **kwargs,
    )

multiclass_metric_meters

Python
multiclass_metric_meters(
    num_classes: int,
    ignore_index: int | None = -100,
    **kwargs
)

Create a pre-configured MetricMeters instance for multiclass classification tasks.

Similar to multiclass_metrics(), but returns a MetricMeters object that is more memory efficient by only tracking running averages instead of storing all predictions and labels. This is suitable for metrics that can be meaningfully averaged across batches.

The returned MetricMeters includes: - Accuracy - F1 Score (macro-averaged)

Note: AUROC and AUPRC are not included as they cannot be meaningfully averaged batch-by-batch. Use multiclass_metrics() if you need those metrics.

Parameters:

Name Type Description Default

num_classes

int

Number of classes in the classification task

 required

ignore_index

int | None

Value in the target to ignore (e.g., padding).

 -100

**kwargs

Additional metric functions to include or override default metrics

 {}

Returns:

Name Type Description
MetricMeters

A configured MetricMeters instance for multiclass classification
Source code in danling/metric/factory.py
Python
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
def multiclass_metric_meters(num_classes: int, ignore_index: int | None = -100, **kwargs):
    """
    Create a pre-configured MetricMeters instance for multiclass classification tasks.

    Similar to multiclass_metrics(), but returns a MetricMeters object that is more memory
    efficient by only tracking running averages instead of storing all predictions and labels.
    This is suitable for metrics that can be meaningfully averaged across batches.

    The returned MetricMeters includes:
    - Accuracy
    - F1 Score (macro-averaged)

    Note: AUROC and AUPRC are not included as they cannot be meaningfully averaged batch-by-batch.
    Use multiclass_metrics() if you need those metrics.

    Args:
        num_classes: Number of classes in the classification task
        ignore_index: Value in the target to ignore (e.g., padding).
        **kwargs: Additional metric functions to include or override default metrics

    Returns:
        MetricMeters: A configured MetricMeters instance for multiclass classification
    """
    lazy_import.check()
    return MetricMeters(
        acc=partial(accuracy, task="multiclass", num_classes=num_classes, preprocess=False),
        f1=partial(f1_score, task="multiclass", num_classes=num_classes, preprocess=False),
        preprocess=partial(preprocess_multiclass, num_classes=num_classes, ignore_index=ignore_index),
        **kwargs,
    )

multilabel_metrics

Python
multilabel_metrics(
    num_labels: int,
    ignore_index: int | None = -100,
    **kwargs
)

Create a pre-configured Metrics instance for multi-label classification tasks.

In multi-label classification, each sample can belong to multiple classes simultaneously. This factory returns a Metrics object with metrics appropriate for multi-label tasks: - AUROC (per label and macro-averaged) - AUPRC (per label and macro-averaged) - Accuracy (exact match or subset accuracy) - MCC (Matthews Correlation Coefficient, per label) - F1 Score (macro-averaged)

The returned Metrics instance expects model outputs with shape [batch_size, num_labels] and binary labels with the same shape.

Parameters:

Name Type Description Default

num_labels

int

Number of possible labels in the multi-label task

 required

ignore_index

int | None

Value in the target to ignore.

 -100

**kwargs

Additional metric functions to include or override default metrics

 {}

Returns:

Name Type Description
Metrics

A configured Metrics instance for multi-label classification
Source code in danling/metric/factory.py
Python
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
def multilabel_metrics(num_labels: int, ignore_index: int | None = -100, **kwargs):
    """
    Create a pre-configured Metrics instance for multi-label classification tasks.

    In multi-label classification, each sample can belong to multiple classes simultaneously.
    This factory returns a Metrics object with metrics appropriate for multi-label tasks:
    - AUROC (per label and macro-averaged)
    - AUPRC (per label and macro-averaged)
    - Accuracy (exact match or subset accuracy)
    - MCC (Matthews Correlation Coefficient, per label)
    - F1 Score (macro-averaged)

    The returned Metrics instance expects model outputs with shape [batch_size, num_labels]
    and binary labels with the same shape.

    Args:
        num_labels: Number of possible labels in the multi-label task
        ignore_index: Value in the target to ignore.
        **kwargs: Additional metric functions to include or override default metrics

    Returns:
        Metrics: A configured Metrics instance for multi-label classification
    """
    lazy_import.check()
    return Metrics(
        auroc=partial(auroc, task="multilabel", num_labels=num_labels, preprocess=False),
        auprc=partial(auprc, task="multilabel", num_labels=num_labels, preprocess=False),
        acc=partial(accuracy, task="multilabel", num_labels=num_labels, preprocess=False),
        mcc=partial(mcc, task="multilabel", num_labels=num_labels, preprocess=False),
        f1=partial(f1_score, task="multilabel", num_labels=num_labels, preprocess=False),
        preprocess=partial(preprocess_multilabel, num_labels=num_labels, ignore_index=ignore_index),
        **kwargs,
    )

multilabel_metric_meters

Python
multilabel_metric_meters(
    num_labels: int,
    ignore_index: int | None = -100,
    **kwargs
)

Create a pre-configured MetricMeters instance for multi-label classification tasks.

Similar to multilabel_metrics(), but returns a MetricMeters object that is more memory efficient by only tracking running averages instead of storing all predictions and labels.

The returned MetricMeters includes: - Accuracy (exact match or subset accuracy) - F1 Score (macro-averaged)

Note: AUROC and AUPRC are not included as they cannot be meaningfully averaged batch-by-batch. Use multilabel_metrics() if you need those metrics.

Parameters:

Name Type Description Default

num_labels

int

Number of possible labels in the multi-label task

 required

ignore_index

int | None

Value in the target to ignore.

 -100

**kwargs

Additional metric functions to include or override default metrics

 {}

Returns:

Name Type Description
MetricMeters

A configured MetricMeters instance for multi-label classification
Source code in danling/metric/factory.py
Python
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
def multilabel_metric_meters(num_labels: int, ignore_index: int | None = -100, **kwargs):
    """
    Create a pre-configured MetricMeters instance for multi-label classification tasks.

    Similar to multilabel_metrics(), but returns a MetricMeters object that is more memory
    efficient by only tracking running averages instead of storing all predictions and labels.

    The returned MetricMeters includes:
    - Accuracy (exact match or subset accuracy)
    - F1 Score (macro-averaged)

    Note: AUROC and AUPRC are not included as they cannot be meaningfully averaged batch-by-batch.
    Use multilabel_metrics() if you need those metrics.

    Args:
        num_labels: Number of possible labels in the multi-label task
        ignore_index: Value in the target to ignore.
        **kwargs: Additional metric functions to include or override default metrics

    Returns:
        MetricMeters: A configured MetricMeters instance for multi-label classification
    """
    lazy_import.check()
    return MetricMeters(
        acc=partial(accuracy, task="multilabel", num_labels=num_labels, preprocess=False),
        f1=partial(f1_score, task="multilabel", num_labels=num_labels, preprocess=False),
        preprocess=partial(preprocess_multilabel, num_labels=num_labels, ignore_index=ignore_index),
        **kwargs,
    )

regression_metrics

Python
regression_metrics(
    num_outputs: int = 1, ignore_nan: bool = True, **kwargs
)

Create a pre-configured Metrics instance for regression tasks.

This factory function returns a Metrics object with standard regression metrics: - R² Score (Coefficient of Determination) - Pearson Correlation Coefficient - Spearman Correlation Coefficient - RMSE (Root Mean Squared Error)

The metrics can handle both single-output regression (e.g., predicting a single value) and multi-output regression (e.g., predicting multiple values per sample).

Parameters:

Name Type Description Default

num_outputs

int

Number of regression outputs per sample.

 1

ignore_nan

bool

Whether to ignore NaN values in inputs/targets.

 True

**kwargs

Additional metric functions to include or override default metrics

 {}

Returns:

Name Type Description
Metrics

A configured Metrics instance for regression tasks
Source code in danling/metric/factory.py
Python
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
def regression_metrics(num_outputs: int = 1, ignore_nan: bool = True, **kwargs):
    """
    Create a pre-configured Metrics instance for regression tasks.

    This factory function returns a Metrics object with standard regression metrics:
    - R² Score (Coefficient of Determination)
    - Pearson Correlation Coefficient
    - Spearman Correlation Coefficient
    - RMSE (Root Mean Squared Error)

    The metrics can handle both single-output regression (e.g., predicting a single value)
    and multi-output regression (e.g., predicting multiple values per sample).

    Args:
        num_outputs: Number of regression outputs per sample.
        ignore_nan: Whether to ignore NaN values in inputs/targets.
        **kwargs: Additional metric functions to include or override default metrics

    Returns:
        Metrics: A configured Metrics instance for regression tasks
    """
    lazy_import.check()
    return Metrics(
        pearson=partial(pearson, num_outputs=num_outputs, preprocess=False),
        spearman=partial(spearman, num_outputs=num_outputs, preprocess=False),
        r2=partial(r2_score, num_outputs=num_outputs, preprocess=False),
        rmse=partial(rmse, num_outputs=num_outputs, preprocess=False),
        preprocess=partial(preprocess_regression, num_outputs=num_outputs, ignore_nan=ignore_nan),
        **kwargs,
    )