autogl.module.nas

NAS algorithms

class autogl.module.nas.algorithm.BaseNAS(device='auto')[source]

Base NAS algorithm class

Parameters:device (str or torch.device) – The device of the whole process
search(space, dataset, estimator) → autogl.module.model.pyg.base.BaseAutoModel[source]

The search process of NAS.

Parameters:
Returns:

model – The searched model.
Return type:

autogl.module.model.BaseModel
to(device)[source]

Change the device of the whole NAS search process

Parameters:device (str or torch.device) –
class autogl.module.nas.algorithm.Darts(num_epochs=5, workers=4, gradient_clip=5.0, model_lr=0.001, model_wd=0.0005, arch_lr=0.0003, arch_wd=0.001, device='auto')[source]

DARTS trainer.

Parameters:
  • num_epochs (int) – Number of epochs planned for training.
  • workers (int) – Workers for data loading.
  • gradient_clip (float) – Gradient clipping. Set to 0 to disable. Default: 5.
  • model_lr (float) – Learning rate to optimize the model.
  • model_wd (float) – Weight decay to optimize the model.
  • arch_lr (float) – Learning rate to optimize the architecture.
  • arch_wd (float) – Weight decay to optimize the architecture.
  • device (str or torch.device) – The device of the whole process
search(space: autogl.module.nas.space.base.BaseSpace, dataset, estimator)[source]

The search process of NAS.

Parameters:
Returns:

model – The searched model.
Return type:

autogl.module.model.BaseModel
class autogl.module.nas.algorithm.Enas(num_epochs=5, n_warmup=100, log_frequency=None, grad_clip=5.0, entropy_weight=0.0001, skip_weight=0.8, baseline_decay=0.999, ctrl_lr=0.00035, ctrl_steps_aggregate=20, ctrl_kwargs=None, model_lr=0.005, model_wd=0.0005, disable_progress=True, device='auto')[source]

ENAS trainer.

Parameters:
  • num_epochs (int) – Number of epochs planned for training.
  • n_warmup (int) – Number of epochs for training super network.
  • log_frequency (int) – Step count per logging.
  • grad_clip (float) – Gradient clipping. Set to 0 to disable. Default: 5.
  • entropy_weight (float) – Weight of sample entropy loss.
  • skip_weight (float) – Weight of skip penalty loss.
  • baseline_decay (float) – Decay factor of baseline. New baseline will be equal to baseline_decay * baseline_old + reward * (1 - baseline_decay).
  • ctrl_lr (float) – Learning rate for RL controller.
  • ctrl_steps_aggregate (int) – Number of steps that will be aggregated into one mini-batch for RL controller.
  • ctrl_kwargs (dict) – Optional kwargs that will be passed to ReinforceController.
  • model_lr (float) – Learning rate for super network.
  • model_wd (float) – Weight decay for super network.
  • disable_progeress (boolean) – Control whether show the progress bar.
  • device (str or torch.device) – The device of the whole process, e.g. “cuda”, torch.device(“cpu”)
search(space: autogl.module.nas.space.base.BaseSpace, dset, estimator)[source]

The search process of NAS.

Parameters:
Returns:

model – The searched model.
Return type:

autogl.module.model.BaseModel
class autogl.module.nas.algorithm.RandomSearch(device='auto', num_epochs=400, disable_progress=False, hardware_metric_limit=None)[source]

Uniformly random architecture search

Parameters:
  • device (str or torch.device) – The device of the whole process, e.g. “cuda”, torch.device(“cpu”)
  • num_epochs (int) – Number of epochs planned for training.
  • disable_progeress (boolean) – Control whether show the progress bar.
search(space: autogl.module.nas.space.base.BaseSpace, dset, estimator)[source]

The search process of NAS.

Parameters:
Returns:

model – The searched model.
Return type:

autogl.module.model.BaseModel
class autogl.module.nas.algorithm.RL(num_epochs=5, device='auto', log_frequency=None, grad_clip=5.0, entropy_weight=0.0001, skip_weight=0.8, baseline_decay=0.999, ctrl_lr=0.00035, ctrl_steps_aggregate=20, ctrl_kwargs=None, n_warmup=100, model_lr=0.005, model_wd=0.0005, disable_progress=False)[source]

RL in GraphNas.

Parameters:
  • num_epochs (int) – Number of epochs planned for training.
  • device (torch.device) – torch.device("cpu") or torch.device("cuda").
  • log_frequency (int) – Step count per logging.
  • grad_clip (float) – Gradient clipping. Set to 0 to disable. Default: 5.
  • entropy_weight (float) – Weight of sample entropy loss.
  • skip_weight (float) – Weight of skip penalty loss.
  • baseline_decay (float) – Decay factor of baseline. New baseline will be equal to baseline_decay * baseline_old + reward * (1 - baseline_decay).
  • ctrl_lr (float) – Learning rate for RL controller.
  • ctrl_steps_aggregate (int) – Number of steps that will be aggregated into one mini-batch for RL controller.
  • ctrl_steps (int) – Number of mini-batches for each epoch of RL controller learning.
  • ctrl_kwargs (dict) – Optional kwargs that will be passed to ReinforceController.
  • n_warmup (int) – Number of epochs for training super network.
  • model_lr (float) – Learning rate for super network.
  • model_wd (float) – Weight decay for super network.
  • disable_progress (boolean) – Control whether show the progress bar.
search(space: autogl.module.nas.space.base.BaseSpace, dset, estimator)[source]

The search process of NAS.

Parameters:
Returns:

model – The searched model.
Return type:

autogl.module.model.BaseModel
class autogl.module.nas.algorithm.GraphNasRL(device='auto', num_epochs=10, log_frequency=None, grad_clip=5.0, entropy_weight=0.0001, skip_weight=0, baseline_decay=0.95, ctrl_lr=0.00035, ctrl_steps_aggregate=100, ctrl_kwargs=None, n_warmup=100, model_lr=0.005, model_wd=0.0005, topk=5, disable_progress=False, hardware_metric_limit=None)[source]

RL in GraphNas.

Parameters:
  • device (torch.device) – torch.device("cpu") or torch.device("cuda").
  • num_epochs (int) – Number of epochs planned for training.
  • log_frequency (int) – Step count per logging.
  • grad_clip (float) – Gradient clipping. Set to 0 to disable. Default: 5.
  • entropy_weight (float) – Weight of sample entropy loss.
  • skip_weight (float) – Weight of skip penalty loss.
  • baseline_decay (float) – Decay factor of baseline. New baseline will be equal to baseline_decay * baseline_old + reward * (1 - baseline_decay).
  • ctrl_lr (float) – Learning rate for RL controller.
  • ctrl_steps_aggregate (int) – Number of steps that will be aggregated into one mini-batch for RL controller.
  • ctrl_steps (int) – Number of mini-batches for each epoch of RL controller learning.
  • ctrl_kwargs (dict) – Optional kwargs that will be passed to ReinforceController.
  • n_warmup (int) – Number of epochs for training super network.
  • model_lr (float) – Learning rate for super network.
  • model_wd (float) – Weight decay for super network.
  • topk (int) – Number of architectures kept in training process.
  • disable_progeress (boolean) – Control whether show the progress bar.
search(space: autogl.module.nas.space.base.BaseSpace, dset, estimator)[source]

The search process of NAS.

Parameters:
Returns:

model – The searched model.
Return type:

autogl.module.model.BaseModel
class autogl.module.nas.algorithm.Spos(n_warmup=1000, grad_clip=5.0, disable_progress=False, optimize_mode='maximize', population_size=100, sample_size=25, cycles=20000, mutation_prob=0.05, device='cuda')[source]

SPOS trainer.

Parameters:
  • n_warmup (int) – Number of epochs for training super network.
  • model_lr (float) – Learning rate for super network.
  • model_wd (float) – Weight decay for super network.
  • parameters see Evolution (Other) –
search(space: autogl.module.nas.space.base.BaseSpace, dset, estimator)[source]

The search process of NAS.

Parameters:
Returns:

model – The searched model.
Return type:

autogl.module.model.BaseModel
class autogl.module.nas.algorithm.Gasso(num_epochs=250, warmup_epochs=10, model_lr=0.01, model_wd=0.0001, arch_lr=0.03, stru_lr=0.04, lamb=0.6, device='auto')[source]

GASSO trainer.

Parameters:
  • num_epochs (int) – Number of epochs planned for training.
  • warmup_epochs (int) – Number of epochs planned for warming up.
  • workers (int) – Workers for data loading.
  • model_lr (float) – Learning rate to optimize the model.
  • model_wd (float) – Weight decay to optimize the model.
  • arch_lr (float) – Learning rate to optimize the architecture.
  • stru_lr (float) – Learning rate to optimize the structure.
  • lamb (float) – The parameter to control the influence of hidden feature smoothness
  • device (str or torch.device) – The device of the whole process
prepare(dset)[source]

Train Pro-GNN.

search(space: autogl.module.nas.space.base.BaseSpace, dataset, estimator)[source]

The search process of NAS.

Parameters:
Returns:

model – The searched model.
Return type:

autogl.module.model.BaseModel
class autogl.module.nas.algorithm.GRNA(n_warmup=1000, grad_clip=5.0, disable_progress=False, optimize_mode='maximize', population_size=100, sample_size=25, cycles=20000, mutation_prob=0.05, device='cuda')[source]

GRNA trainer.

Parameters:
  • n_warmup (int) – Number of epochs for training super network.
  • model_lr (float) – Learning rate for super network.
  • model_wd (float) – Weight decay for super network.
  • parameters see Evolution (Other) –
class autogl.module.nas.space.BaseSpace[source]

Base space class of NAS module. Defining space containing all models. Please use mutables to define your whole space. Refer to https://nni.readthedocs.io/en/stable/NAS/WriteSearchSpace.html for detailed information.

Parameters:init (bool) – Whether to initialize the whole space. Default: False
forward(*args, **kwargs)[source]

Define the forward pass of space model

instantiate()[source]

Instantiate the space, reset default key for the mutables here/

parse_model(selection: dict, device) → autogl.module.model.pyg.base.BaseAutoModel[source]

Export the searched model from space.

Parameters:
  • selection (Dict) – The dictionary containing all the choices of nni.
  • device (str or torch.device) – The device to put model on.
Returns:

model – model to be exported.
Return type:

autogl.module.model.BaseModel
setInputChoice(order, n_candidates=None, choose_from=None, n_chosen=None, reduction='sum', return_mask=False, key=None)[source]

Give a unique key if not given

setLayerChoice(order, op_candidates, reduction='sum', return_mask=False, key=None)[source]

Give a unique key if not given

wrap()[source]

Return a BoxModel which wrap self as a model Used to pass to trainer To use this function, must contain input_dim and output_dim

class autogl.module.nas.space.GraphNasMacroNodeClassificationSpace(hidden_dim: Optional[int] = 64, layer_number: Optional[int] = 2, dropout: Optional[float] = 0.6, input_dim: Optional[int] = None, output_dim: Optional[int] = None, ops: Tuple = None, search_act_con=False)[source]
instantiate(hidden_dim: Optional[int] = None, layer_number: Optional[int] = None, input_dim: Optional[int] = None, output_dim: Optional[int] = None, ops: Tuple = None, dropout=None)[source]

Instantiate the space, reset default key for the mutables here/

parse_model(selection, device) → autogl.module.model.pyg.base.BaseAutoModel[source]

Export the searched model from space.

Parameters:
  • selection (Dict) – The dictionary containing all the choices of nni.
  • device (str or torch.device) – The device to put model on.
Returns:

model – model to be exported.
Return type:

autogl.module.model.BaseModel
class autogl.module.nas.space.GraphNasNodeClassificationSpace(hidden_dim: Optional[int] = 64, layer_number: Optional[int] = 2, dropout: Optional[float] = 0.9, input_dim: Optional[int] = None, output_dim: Optional[int] = None, gnn_ops: Sequence[Union[str, Any]] = ['gat_8', 'gat_6', 'gat_4', 'gat_2', 'gat_1', 'gcn', 'cheb', 'sage', 'arma', 'sg', 'linear', 'zero'], act_ops: Sequence[Union[str, Any]] = ['sigmoid', 'tanh', 'relu', 'linear', 'elu'], con_ops: Sequence[Union[str, Any]] = ['add', 'product', 'concat'])[source]
forward(data)[source]

Define the forward pass of space model

instantiate(hidden_dim: Optional[int] = None, layer_number: Optional[int] = None, dropout: Optional[float] = None, input_dim: Optional[int] = None, output_dim: Optional[int] = None, gnn_ops: Sequence[Union[str, Any]] = None, act_ops: Sequence[Union[str, Any]] = None, con_ops: Sequence[Union[str, Any]] = None)[source]

Instantiate the space, reset default key for the mutables here/

parse_model(selection, device) → autogl.module.model.pyg.base.BaseAutoModel[source]

Export the searched model from space.

Parameters:
  • selection (Dict) – The dictionary containing all the choices of nni.
  • device (str or torch.device) – The device to put model on.
Returns:

model – model to be exported.
Return type:

autogl.module.model.BaseModel
class autogl.module.nas.space.SinglePathNodeClassificationSpace(hidden_dim: Optional[int] = 64, layer_number: Optional[int] = 2, dropout: Optional[float] = 0.2, input_dim: Optional[int] = None, output_dim: Optional[int] = None, ops: Tuple = ['gcn', 'gat_8'])[source]
forward(data)[source]

Define the forward pass of space model

instantiate(hidden_dim: Optional[int] = None, layer_number: Optional[int] = None, input_dim: Optional[int] = None, output_dim: Optional[int] = None, ops: Tuple = None, dropout=None)[source]

Instantiate the space, reset default key for the mutables here/

parse_model(selection, device) → autogl.module.model.pyg.base.BaseAutoModel[source]

Export the searched model from space.

Parameters:
  • selection (Dict) – The dictionary containing all the choices of nni.
  • device (str or torch.device) – The device to put model on.
Returns:

model – model to be exported.
Return type:

autogl.module.model.BaseModel
class autogl.module.nas.space.GassoSpace(hidden_dim: Optional[int] = 64, layer_number: Optional[int] = 2, dropout: Optional[float] = 0.8, input_dim: Optional[int] = None, output_dim: Optional[int] = None, ops: Tuple = ['gat', 'gcn', 'gin', 'sage', 'linear'])[source]
forward(data)[source]

Define the forward pass of space model

instantiate(hidden_dim: Optional[int] = 64, layer_number: Optional[int] = 2, dropout: Optional[float] = 0.8, input_dim: Optional[int] = None, output_dim: Optional[int] = None, ops: Tuple = ['gat', 'gcn', 'gin', 'sage', 'linear'])[source]

Instantiate the space, reset default key for the mutables here/

parse_model(selection, device) → autogl.module.model.pyg.base.BaseAutoModel[source]

Export the searched model from space.

Parameters:
  • selection (Dict) – The dictionary containing all the choices of nni.
  • device (str or torch.device) – The device to put model on.
Returns:

model – model to be exported.
Return type:

autogl.module.model.BaseModel
class autogl.module.nas.space.GRNASpace(hidden_dim: Optional[int] = 64, layer_number: Optional[int] = 2, dropout: Optional[float] = 0.6, input_dim: Optional[int] = None, output_dim: Optional[int] = None, ops: Tuple = ['gcn', 'gat_2'], rob_ops: Tuple = ['identity', 'svd', 'jaccard', 'gnnguard'], act_ops: Sequence[Union[str, Any]] = ['sigmoid', 'tanh', 'relu', 'linear', 'elu'])[source]
forward(data)[source]

Define the forward pass of space model

instantiate(hidden_dim: Optional[int] = None, layer_number: Optional[int] = None, input_dim: Optional[int] = None, output_dim: Optional[int] = None, ops: Tuple = None, rob_ops: Tuple = None, act_ops: Tuple = None, dropout=None)[source]

Instantiate the space, reset default key for the mutables here/

parse_model(selection, device) → autogl.module.model.pyg.base.BaseAutoModel[source]

Export the searched model from space.

Parameters:
  • selection (Dict) – The dictionary containing all the choices of nni.
  • device (str or torch.device) – The device to put model on.
Returns:

model – model to be exported.
Return type:

autogl.module.model.BaseModel
class autogl.module.nas.space.AutoAttendNodeClassificationSpace(hidden_dim: Optional[int] = 64, layer_number: Optional[int] = 2, dropout: Optional[float] = 0.9, input_dim: Optional[int] = None, output_dim: Optional[int] = None, ops_type=0, gnn_ops: Sequence[Union[str, Any]] = None, act_op='tanh', head=8, agg_ops=['add', 'attn'])[source]

AutoAttend Search Space , please refer to http://proceedings.mlr.press/v139/guan21a.html for details. The current implementation is nc (no context weight sharing), we will in future add other types of partial weight sharing proposed in the paper.

Parameters:
  • ops_type (int) – 0 or 1 , choosing from two sets of ops with index ops_type
  • gnn_ops (list of str) – op names for searching, which descripts the compostion of operation pool
  • act_op (str) – determine used activation function
  • agg_ops (list of str) – agg op names for searching. Only [‘add’,’attn’] are options, as mentioned in the paper.
forward(data)[source]

Define the forward pass of space model

instantiate(hidden_dim: Optional[int] = None, layer_number: Optional[int] = None, dropout: Optional[float] = None, input_dim: Optional[int] = None, output_dim: Optional[int] = None, ops_type=None, gnn_ops: Sequence[Union[str, Any]] = None, act_op=None, head=None, agg_ops=None)[source]

Instantiate the space, reset default key for the mutables here/

parse_model(selection, device)[source]

Export the searched model from space.

Parameters:
  • selection (Dict) – The dictionary containing all the choices of nni.
  • device (str or torch.device) – The device to put model on.
Returns:

model – model to be exported.
Return type:

autogl.module.model.BaseModel
class autogl.module.nas.estimator.BaseEstimator(loss_f: str = 'nll_loss', evaluation=[<autogl.module.train.evaluation.Acc object>])[source]

The estimator of NAS model.

Parameters:
  • loss_f (callable) – Default loss function for evaluation
  • evaluation (list of autogl.module.train.evaluation.Evaluation) – Default evaluation metric
infer(model: autogl.module.nas.space.base.BaseSpace, dataset, mask='train') → Tuple[torch.Tensor, torch.Tensor][source]

Calculate the loss and metrics of given model on given dataset using specified masks.

Parameters:
  • model (autogl.module.nas.space.BaseSpace) – The model in space.
  • dataset (autogl.dataset) – The dataset to perform infer
  • mask (str) – The mask to evalute on dataset
Returns:

  • metrics (list of float) – the metrics on given datasets.
  • loss (torch.Tensor) – the loss on given datasets. Note that loss should be differentiable.
class autogl.module.nas.estimator.OneShotEstimator(loss_f='nll_loss', evaluation=[<autogl.module.train.evaluation.Acc object>])[source]

One shot estimator.

Use model directly to get estimations.

Parameters:
  • loss_f (str) – The name of loss funciton in PyTorch
  • evaluation (list of Evaluation) – The evaluation metrics in module/train/evaluation
infer(model: autogl.module.nas.space.base.BaseSpace, dataset, mask='train')[source]

Calculate the loss and metrics of given model on given dataset using specified masks.

Parameters:
  • model (autogl.module.nas.space.BaseSpace) – The model in space.
  • dataset (autogl.dataset) – The dataset to perform infer
  • mask (str) – The mask to evalute on dataset
Returns:

  • metrics (list of float) – the metrics on given datasets.
  • loss (torch.Tensor) – the loss on given datasets. Note that loss should be differentiable.
class autogl.module.nas.estimator.TrainEstimator(loss_f='nll_loss', evaluation=[<autogl.module.train.evaluation.Acc object>])[source]

An estimator which trans from scratch

Parameters:
  • loss_f (str) – The name of loss funciton in PyTorch
  • evaluation (list of Evaluation) – The evaluation metrics in module/train/evaluation
infer(model: autogl.module.nas.space.base.BaseSpace, dataset, mask='train')[source]

Calculate the loss and metrics of given model on given dataset using specified masks.

Parameters:
  • model (autogl.module.nas.space.BaseSpace) – The model in space.
  • dataset (autogl.dataset) – The dataset to perform infer
  • mask (str) – The mask to evalute on dataset
Returns:

  • metrics (list of float) – the metrics on given datasets.
  • loss (torch.Tensor) – the loss on given datasets. Note that loss should be differentiable.
class autogl.module.nas.estimator.OneShotEstimator_HardwareAware(loss_f='nll_loss', evaluation=[<autogl.module.train.evaluation.Acc object>], hardware_evaluation='parameter', hardware_metric_weight=0)[source]

One shot hardware-aware estimator.

Use model directly to get estimations with some hardware-aware metrics.

Parameters:
  • loss_f (str) – The name of loss funciton in PyTorch
  • evaluation (list of Evaluation) – The evaluation metrics in module/train/evaluation
  • hardware_evaluation (str or runable) – The hardware-aware metrics. Can be ‘parameter’ or ‘latency’. Or you can define a special metric by a runable function
  • hardware_metric_weight (float) – The weight of hardware-aware metric, which will be a bias added to metrics
infer(model: autogl.module.nas.space.base.BaseSpace, dataset, mask='train')[source]

Calculate the loss and metrics of given model on given dataset using specified masks.

Parameters:
  • model (autogl.module.nas.space.BaseSpace) – The model in space.
  • dataset (autogl.dataset) – The dataset to perform infer
  • mask (str) – The mask to evalute on dataset
Returns:

  • metrics (list of float) – the metrics on given datasets.
  • loss (torch.Tensor) – the loss on given datasets. Note that loss should be differentiable.
class autogl.module.nas.estimator.TrainEstimator_HardwareAware(loss_f='nll_loss', evaluation=[<autogl.module.train.evaluation.Acc object>], hardware_evaluation='parameter', hardware_metric_weight=0)[source]

An hardware-aware estimator which trans from scratch

class autogl.module.nas.estimator.GRNAEstimator(loss_f='nll_loss', evaluation=[<autogl.module.train.evaluation.Acc object>], lambda_=0.05, perturb_type='random', adv_sample_num=10, dis_type='ce', ptbr=0.05)[source]

Graph robust neural architecture estimator under adversarial attack.

Use model directly to get estimations.

Parameters:
  • loss_f (str) – The name of loss funciton in PyTorch
  • evaluation (list of Evaluation) – The evaluation metrics in module/train/evaluation GRNA_metric = acc_metric+ robustness_metric
  • lambda (float) – The hyper-parameter to balance the accuracy metric and robustness metric to perform ultimate evaluation
  • perturb_type (str) – Perturbation methods to simulate the adversarial attack process
  • adv_sample_num (int) – Adversarial sample number used in measure architecture robustness.
infer(model: autogl.module.nas.space.base.BaseSpace, dataset, mask='train')[source]

Calculate the loss and metrics of given model on given dataset using specified masks.

Parameters:
  • model (autogl.module.nas.space.BaseSpace) – The model in space.
  • dataset (autogl.dataset) – The dataset to perform infer
  • mask (str) – The mask to evalute on dataset
Returns:

  • metrics (list of float) – the metrics on given datasets.
  • loss (torch.Tensor) – the loss on given datasets. Note that loss should be differentiable.