Processes¶

Processes describe how a (or several) neural networks (trainables) are trained with given data. In a large number of cases, simply leveraging StandardClassification will be sufficient for many cases, as it provides many idiosynchratic options, as listed below.

See the processes guide for an overview on how these work.

Classes

`BaseProcess`([lr, metrics, …])	Initialization of the Base Trainable object.
`LDAMLoss`(cls_num_list[, max_m, weight, s])
`StandardClassification`(classifier[, …])

Functions

get_label_balance(dataset)

Given a dataset, return the proportion of each target class and the counts of each class type

class dn3.trainable.processes.BaseProcess(lr=0.001, metrics=None, evaluation_only_metrics=None, l2_weight_decay=0.01, cuda=None, **kwargs)¶

Initialization of the Base Trainable object. Any learning procedure that leverages DN3atasets should subclass this base class.

By default uses the SGD with momentum optimization.

Methods

`build_network`(**kwargs)	This method is used to add trainable modules to the process.
`calculate_loss`(inputs, outputs)	Given the inputs to and outputs from underlying modules, calculate the loss.
`calculate_metrics`(inputs, outputs)	Given the inputs to and outputs from the underlying module.
`evaluate`(dataset, **loader_kwargs)	Calculate and return metrics for a dataset
`fit`(training_dataset[, epochs, …])	sklearn/keras-like convenience method to simply proceed with training across multiple epochs of the provided
`forward`(*inputs)	Given a batch of inputs, return the outputs produced by the trainable module.
`load_best`(best)	Load the parameters as saved by `save_best()`.
`parameters`()	All the trainable parameters in the Trainable.
`predict`(dataset, **loader_kwargs)	Determine the outputs for all loaded data from the dataset
`save_best`()	Create a snapshot of what is being currently trained for re-laoding with the `load_best()` method.
`set_scheduler`(scheduler[, step_every_batch])	This allow the addition of a learning rate schedule to the process.

build_network(**kwargs)¶

This method is used to add trainable modules to the process. Rather than placing objects for training in the __init__ method, they should be placed here.

By default any arguments that propagate unused from __init__ are included here.

calculate_loss(inputs, outputs)¶

Given the inputs to and outputs from underlying modules, calculate the loss.

Returns: Single loss quantity to be minimized.
Return type: Loss

calculate_metrics(inputs, outputs)¶

Given the inputs to and outputs from the underlying module. Return tracked metrics.

Parameters

inputs – Input tensors.
outputs – Output tensors.

Returns

metrics – Dictionary of metric quantities.

Return type

OrderedDict, None

evaluate(dataset, **loader_kwargs)¶

Calculate and return metrics for a dataset

Parameters

dataset (DN3ataset, DataLoader) – The dataset that will be used for evaluation, if not a DataLoader, one will be constructed
loader_kwargs (dict) – Args that will be passed to the dataloader, but shuffle and drop_last will be both be forced to False

Returns

metrics – Metric scores for the entire

Return type

OrderedDict

fit(training_dataset, epochs=1, validation_dataset=None, step_callback=None, resume_epoch=None, resume_iteration=None, log_callback=None, epoch_callback=None, batch_size=8, warmup_frac=0.2, retain_best='loss', validation_interval=None, train_log_interval=None, **loader_kwargs)¶

sklearn/keras-like convenience method to simply proceed with training across multiple epochs of the provided dataset

Parameters

training_dataset (DN3ataset, DataLoader) –
validation_dataset (DN3ataset, DataLoader) –
epochs (int) – Total number of epochs to fit
resume_epoch (int) – The starting epoch to train from. This will likely only be used to resume training at a certain point.
resume_iteration (int) – Similar to start epoch but specified in batches. This can either be used alone, or in conjunction with start_epoch. If used alone, the start epoch is the floor of start_iteration divided by batches per epoch. In other words this specifies cumulative batches if start_epoch is not specified, and relative to the current epoch otherwise.
step_callback (callable) – Function to run after every training step that has signature: fn(train_metrics) -> None
log_callback (callable) – Function to run after every log interval that has signature: fn(train_metrics) -> None
epoch_callback (callable) – Function to run after every epoch that has signature: fn(validation_metrics) -> None
batch_size (int) – The batch_size to be used for the training and validation datasets. This is ignored if they are provided as DataLoader.
warmup_frac (float) – The fraction of iterations that will be spent increasing the learning rate under the default 1cycle policy (with cosine annealing). Value will be automatically clamped values between [0, 0.5]
retain_best ((str, None)) – If `validation_dataset` is provided, which model weights to retain. If ‘loss’ (default), will retain the model at the epoch with the lowest validation loss. If another string, will assume that is the metric to monitor for the highest score. If None, the final model is used.
validation_interval (int, None) – The number of batches between checking the validation dataset
train_log_interval (int, None) – The number of batches between persistent logging of training metrics, if None (default) happens at the end of every epoch.
loader_kwargs – Any remaining keyword arguments will be passed as such to any DataLoaders that are automatically constructed. If both training and validation datasets are provided as DataLoaders, this will be ignored.

Notes

If the datasets above are provided as DN3atasets, automatic optimizations are performed to speed up loading. These include setting the number of workers = to the number of CPUs/system threads - 1, and pinning memory for rapid CUDA transfer if leveraging the GPU. Unless you are very comfortable with PyTorch, it’s probably better to not provide your own DataLoader, and let this be done automatically.

Returns

train_log (Dataframe) – Metrics after each iteration of training as a pandas dataframe
validation_log (Dataframe) – Validation metrics after each epoch of training as a pandas dataframe

forward(*inputs)¶

Given a batch of inputs, return the outputs produced by the trainable module.

Parameters: inputs – Tensors needed for underlying module.
Returns: Outputs of module
Return type: outputs

load_best(best)¶

Load the parameters as saved by save_best().

Parameters: best (Any) –

parameters()¶

All the trainable parameters in the Trainable. This includes any architecture parameters and meta-parameters.

Returns: An iterator of parameters
Return type: params

predict(dataset, **loader_kwargs)¶

Determine the outputs for all loaded data from the dataset

Parameters

dataset (DN3ataset, DataLoader) – The dataset that will be used for evaluation, if not a DataLoader, one will be constructed
loader_kwargs (dict) – Args that will be passed to the dataloader, but shuffle and drop_last will be both be forced to False

Returns

inputs (Tensor) – The exact inputs used to calculate the outputs (in case they were stochastic and need saving)
outputs (Tensor) – The outputs from each run of :function:`forward`

save_best()¶

Create a snapshot of what is being currently trained for re-laoding with the load_best() method.

Returns: best – Whatever format is needed for load_best(), will be the argument provided to it.
Return type: Any

set_scheduler(scheduler, step_every_batch=False)¶

This allow the addition of a learning rate schedule to the process. By default, a linear warmup with cosine decay will be used. Any scheduler that is an instance of Scheduler (pytorch’s schedulers, or extensions thereof) can be set here. Additionally, a string keywords can be used including:

“constant”

Parameters

scheduler (str, Scheduler) –
step_every_batch (bool) – Whether to call step after every batch (if True), or after every epoch (False)

class dn3.trainable.processes.LDAMLoss(cls_num_list, max_m=0.5, weight=None, s=30)¶

Methods

forward(x, target)

Defines the computation performed at every call.

forward(x, target)¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class dn3.trainable.processes.StandardClassification(classifier: torch.nn.modules.module.Module, loss_fn=None, cuda=None, metrics=None, learning_rate=0.01, label_smoothing=None, **kwargs)¶

Methods

`calculate_loss`(inputs, outputs)	Given the inputs to and outputs from underlying modules, calculate the loss.
`fit`(training_dataset[, epochs, …])	sklearn/keras-like convenience method to simply proceed with training across multiple epochs of the provided
`forward`(*inputs)	Given a batch of inputs, return the outputs produced by the trainable module.

calculate_loss(inputs, outputs)¶

Given the inputs to and outputs from underlying modules, calculate the loss.

Returns: Single loss quantity to be minimized.
Return type: Loss

fit(training_dataset, epochs=1, validation_dataset=None, step_callback=None, epoch_callback=None, batch_size=8, warmup_frac=0.2, retain_best='loss', balance_method=None, **loader_kwargs)¶

sklearn/keras-like convenience method to simply proceed with training across multiple epochs of the provided dataset

Parameters

training_dataset (DN3ataset, DataLoader) –
validation_dataset (DN3ataset, DataLoader) –
epochs (int) –
step_callback (callable) – Function to run after every training step that has signature: fn(train_metrics) -> None
epoch_callback (callable) – Function to run after every epoch that has signature: fn(validation_metrics) -> None
batch_size (int) – The batch_size to be used for the training and validation datasets. This is ignored if they are provided as DataLoader.
warmup_frac (float) – The fraction of iterations that will be spent increasing the learning rate under the default 1cycle policy (with cosine annealing). Value will be automatically clamped values between [0, 0.5]
retain_best ((str, None)) – If `validation_dataset` is provided, which model weights to retain. If ‘loss’ (default), will retain the model at the epoch with the lowest validation loss. If another string, will assume that is the metric to monitor for the highest score. If None, the final model is used.
balance_method ((None, str)) – If and how to balance training samples when training. None (default) will simply randomly sample all training samples equally. ‘undersample’ will sample each class N_min times where N_min is equal to the number of examples in the minority class. ‘oversample’ will sample each class N_max times, where N_max is the number of the majority class.
loader_kwargs – Any remaining keyword arguments will be passed as such to any DataLoaders that are automatically constructed. If both training and validation datasets are provided as DataLoaders, this will be ignored.

Notes

If the datasets above are provided as DN3atasets, automatic optimizations are performed to speed up loading. These include setting the number of workers = to the number of CPUs/system threads - 1, and pinning memory for rapid CUDA transfer if leveraging the GPU. Unless you are very comfortable with PyTorch, it’s probably better to not provide your own DataLoader, and let this be done automatically.

Returns

train_log (Dataframe) – Metrics after each iteration of training as a pandas dataframe
validation_log (Dataframe) – Validation metrics after each epoch of training as a pandas dataframe

forward(*inputs)¶

Given a batch of inputs, return the outputs produced by the trainable module.

Parameters: inputs – Tensors needed for underlying module.
Returns: Outputs of module
Return type: outputs

dn3.trainable.processes.get_label_balance(dataset)¶

Given a dataset, return the proportion of each target class and the counts of each class type

Parameters: dataset –
Returns
Return type: sample_weights, counts