.. _nas_cn锛�
绁炵粡鏋舵瀯鎼滅储
============================
鎴戜滑鍦ㄤ笉鍚岀殑鎼滅储绌洪棿涓敮鎸佷笉鍚岀殑绁炵粡鏋舵瀯鎼滅储绠楁硶銆�
绁炵粡鏋舵瀯鎼滅储閫氬父鐢变笁涓ā鍧楁瀯鎴愶細鎼滅储绌洪棿銆佹悳绱㈢瓥鐣ュ拰璇勪及绛栫暐銆�
鎼滅储绌洪棿鎻忚堪浜嗘墍鏈夊彲鑳借鎼滅储鐨勬灦鏋勩€傜┖闂翠富瑕佺敱涓ら儴鍒嗙粍鎴愶細鎿嶄綔锛堝GCNconv銆丟ATconv锛夊拰杈撳叆杈撳嚭鍏崇郴銆�
澶х┖闂村彲鑳芥湁鏇村ソ鐨勬渶浼樻灦鏋勶紝浣嗛渶瑕佹洿澶氱殑绠楀姏鏉ユ帰绱€€�
浜虹被鐭ヨ瘑鍙互甯姪璁捐鍚堢悊鐨勬悳绱㈢┖闂达紝鍑忓皯鎼滅储绛栫暐鐨勫紑閿€銆�
鎼滅储绛栫暐鍒欐帶鍒剁潃濡備綍鏉ユ帰绱㈡悳绱㈢┖闂淬€�
瀹冨寘鍚簡缁忓吀鐨勬帰绱�-鍒╃敤鍥板銆�
涓€鏂归潰锛屼汉浠笇鏈涘揩閫熸壘鍒版€ц兘鑹ソ鐨勬灦鏋勶紝
鑰屽彟涓€鏂归潰锛屼篃闇€瑕侀伩鍏嶈繃鏃╂敹鏁涘埌娆′紭鏋舵瀯鍖哄煙銆�
璇勪及绛栫暐鍦ㄦ帰绱㈡椂缁欏嚭鏋舵瀯鐨勬€ц兘銆�
鏈€绠€鍗曠殑鏂规硶鏄鏁版嵁鎵ц鏍囧噯鐨勬灦鏋勮缁冨拰楠岃瘉銆�
浣嗙敱浜庡湪鎼滅储杩囩▼涓湁寰堝鏋舵瀯闇€瑕佽瘎浼帮紝鍥犳闇€瑕侀潪甯告湁鏁堢殑璇勪及绛栫暐鏉ヨ妭鐪佽绠楄祫婧愩€�
.. image:: ../../../resources/nas.svg
:align: center
涓轰簡鏇村姞鐏垫椿鏄撶敤锛屾垜浠皢绁炵粡鏋舵瀯鎼滅储杩囩▼鍒嗚В涓轰笁閮ㄥ垎锛氱畻娉曘€佺┖闂村拰璇勪及鍣紝鍒嗗埆瀵瑰簲浜庝笁涓ā鍧楋細鎼滅储绌洪棿銆佹悳绱㈢瓥鐣ュ拰璇勪及绛栫暐銆�
涓嶅悓閮ㄥ垎鐨勪笉鍚屾ā鍨嬪彲浠ュ湪婊¤冻涓€瀹氭潯浠剁殑鏃跺€欑粍鍚堣捣鏉ャ€�
濡傛灉鎮ㄦ兂璁捐鑷繁鐨勭缁忔灦鏋勬悳绱㈡祦绋嬶紝鎮ㄥ彲浠ユ牴鎹渶瑕佹洿鏀瑰叾涓殑浠绘剰閮ㄥ垎銆�
鐢ㄦ硶
-----
鎮ㄥ彲浠ラ€氳繃灏嗙畻娉曘€佺┖闂翠笌璇勪及鍣ㄧ洿鎺ヤ紶閫掔粰姹傝В鍣ㄦ潵鍦ㄨ妭鐐瑰垎绫讳换鍔′笂鍚敤鏋舵瀯鎼滅储銆�
涓嬮潰鏄竴涓緥瀛愶細
.. code-block:: python
# 鍦╟ora鏁版嵁闆嗕笂浣跨敤鍥剧缁忔灦鏋勬悳绱�
from autogl.datasets import build_dataset_from_name
from autogl.solver import AutoNodeClassifier
solver = AutoNodeClassifier(
feature = 'PYGNormalizeFeatures',
graph_models = (),
hpo = 'tpe',
ensemble = None,
nas_algorithms=['rl'],
nas_spaces='graphnasmacro',
nas_estimators=['scratch']
)
cora = build_dataset_from_name('cora')
solver.fit(cora)
涓婇潰鐨勪唬鐮佸皢鍏堢敤 ``rl`` 鎼滅储绠楁硶鍦ㄧ┖闂� ``GraphnaSmaro`` 涓壘鍒版渶浼樻灦鏋勩€�
鐒跺悗閫氳繃瓒呭弬鏁颁紭鍖栨柟娉� ``tpe`` 杩涗竴姝ヤ紭鍖栨悳绱㈠埌鐨勬灦鏋勩€�
.. note:: ``graph_models`` 鍙傛暟涓庣缁忔灦鏋勬悳绱㈡ā鍧椾笉鍐茬獊銆傛偍鍙互灏� ``graph_models`` 璁剧疆涓�
绁炵粡鏋舵瀯鎼滅储鍙戠幇鐨勬ā鍨嬪鐨勶紝鍏朵粬鎵嬪伐璁捐鐨勬ā鍨嬨€傝€岀缁忔灦鏋勬悳绱㈡ā鍧楁淳鐢熷嚭鏉ョ殑鏋舵瀯
鐨勪綔鐢ㄤ篃涓庣洿鎺ラ€氳繃鍥炬ā鍨嬫ā鍧椾紶閫掔殑鎵嬪伐璁捐鐨勬ā鍨嬬浉鍚屻€�
鎼滅储绌洪棿
------------
绌洪棿瀹氫箟鍩轰簬寮€婧愬伐鍏峰寘NNI涓娇鐢ㄧ殑鍙彉鏂瑰紡锛屽畾涔変负缁ф壙鍩虹绌洪棿鐨勬ā鍨嬨€�
瀹氫箟鎼滅储绌洪棿鐨勬柟娉曚富瑕佹湁涓ょ锛屼竴绉嶆敮鎸佸崟鏍锋湰鏂规硶锛岃€屽彟涓€绉嶅垯涓嶆敮鎸併€�
鐩墠锛屾垜浠敮鎸佸涓嬫悳绱㈢┖闂达細
+------------------------+-----------------------------------------------------------------+
| 绌洪棿 | 鎻忚堪 |
+========================+=================================================================+
| ``singlepath`` [4]_ | 澶氫釜杩炵画灞傛瀯鎴愮殑鏋舵瀯锛屽叾涓瘡灞� |
| | 鍙兘鏈変竴涓€夋嫨 |
+------------------------+-----------------------------------------------------------------+
| ``graphnas`` [1]_ | 涓虹洃鐫ｈ妭鐐瑰垎绫婚棶棰樼殑妯″瀷璁捐鐨� |
| | 鍥剧缁忔灦鏋勬悳绱㈠井鎼滅储绌洪棿 |
+------------------------+-----------------------------------------------------------------+
| ``graphnasmacro`` [1]_ | 涓哄崐鐩戠潱鑺傜偣鍒嗙被闂鐨勬ā鍨嬭璁＄殑 |
| | 鍥剧缁忔灦鏋勬悳绱㈠畯鎼滅储绌洪棿 |
+------------------------+-----------------------------------------------------------------+
鎮ㄤ篃鍙互瀹氫箟鑷繁鐨勭缁忔灦鏋勬悳绱㈢┖闂淬€�
濡傛灉闇€瑕佹敮鎸佸崟鏍锋湰鏂瑰紡锛屽彲浠ヤ娇鐢ㄥ嚱鏁� ``setLayerChoice`` 涓� ``setInputChoice`` 鏉ユ瀯寤鸿秴缃戠粶銆�
涓嬮潰鏄竴涓緥瀛愩€�
.. code-block:: python
# 鍒涘缓涓€涓缁忔灦鏋勬悳绱㈢┖闂寸殑渚嬪瓙
from autogl.module.nas.space.base import BaseSpace
from autogl.module.nas.space.operation import gnn_map
class YourOneShotSpace(BaseSpace):
# 鍦ㄥ垵濮嬪寲鏃惰幏鍙栧熀鏈弬鏁�
def __init__(self, input_dim = None, output_dim = None):
super().__init__()
# 蹇呴』鍦ㄧ┖闂翠腑璁惧畾杈撳叆缁村害涓庤緭鍑虹淮搴︼紝涔熷彲浠ュ湪鍑芥暟 `instantiate` 涓垵濮嬪寲杩欎袱涓弬鏁癭
self.input_dim = input_dim
self.output_dim = output_dim
# 瀹炰緥鍖栬秴缃戠粶
def instantiate(self, input_dim = None, output_dim = None):
# 蹇呴』鍦ㄥ嚱鏁颁腑璋冪敤鐖剁被鐨勫疄渚嬪寲
super().instantiate()
self.input_dim = input_dim or self.input_dim
self.output_dim = output_dim or self.output_dim
# 鎸夌収椤哄簭瀹氫箟涓ゅ眰缃戠粶
setattr(self, 'layer0', self.setLayerChoice(0, [gnn_map(op,self.input_dim,self.output_dim)for op in ['gcn', 'gat']], key = 'layer0')
setattr(self, 'layer1', self.setLayerChoice(1, [gnn_map(op,self.input_dim,self.output_dim)for op in ['gcn', 'gat']], key = 'layer1')
# 瀹氫箟涓€涓粠涓ゅ眰鐨勭粨鏋滀腑閫夋嫨鐨勮緭鍏ラ€夐」
setattr(self, 'input_layer', self.setInputChoice(2, choose_from = ['layer0', 'layer1'], n_chosen = 1, returen_mask = False, key = 'input_layer'))
self._initialized = True
# 瀹氫箟鍓嶅悜浼犳挱杩囩▼
def forward(self, data):
x, edges = data.x, data.edge_index
x_0 = self.layer0(x, edges)
x_1 = self.layer1(x, edges)
y = self.input_layer([x_0, x_1])
y = F.log_fostmax(y, dim = 1)
return y
# 瀵逛簬鍗曟牱鏈寖寮忥紝鎮ㄥ彲浠ヤ娇鐢ㄥ ``parse_model`` 鍑芥暟涓殑鏂规硶
def parse_model(self, selection, device) -> BaseModel:
return self.wrap().fix(selection)
鎮ㄤ篃鍙互浣跨敤涓嶆敮鎸佸崟鏍锋湰鑼冨紡鐨勬柟寮忋€�
杩欐牱鐨勮瘽锛屾偍鍙互鐩存帴澶嶅埗妯″瀷锛屽苟杩涜灏戦噺鏇存敼銆�
浣嗙浉搴旂殑锛屾偍涔熷彧鑳戒娇鐢ㄥ熀浜庢牱鏈殑鎼滅储绛栫暐銆�
.. code-block:: python
# 鍒涘缓涓€涓缁忔灦鏋勬悳绱㈢┖闂寸殑渚嬪瓙
from autogl.module.nas.space.base import BaseSpace, map_nn
from autogl.module.nas.space.operation import gnn_map
# 鍦ㄨ繖閲岋紝鎴戜滑浠� `head` 浣滀负鍙傛暟锛屽湪涓夌鍥惧嵎绉笂杩涜鎼滅储
# 鍦ㄦ悳绱� `heads` 鏃讹紝鎴戜滑鍦ㄦ悳绱㈠浘鍗风Н
from torch_geometric.nn import GATConv, FeaStConv, TransformerConv
class YourNonOneShotSpace(BaseSpace):
# 鍦ㄥ垵濮嬪寲鏃惰幏鍙栧熀鏈弬鏁�
def __init__(self, input_dim = None, output_dim = None):
super().__init__()
# 蹇呴』鍦ㄧ┖闂翠腑璁惧畾杈撳叆缁村害涓庤緭鍑虹淮搴︼紝涔熷彲浠ュ湪鍑芥暟 `instantiate` 涓垵濮嬪寲杩欎袱涓弬鏁癭
self.input_dim = input_dim
self.output_dim = output_dim
# 瀹炰緥鍖栬秴缃戠粶
def instantiate(self, input_dim, output_dim):
# 蹇呴』鍦ㄥ嚱鏁颁腑璋冪敤鐖剁被鐨勫疄渚嬪寲
super().instantiate()
self.input_dim = input_dim or self.input_dim
self.output_dim = output_dim or self.output_dim
# 璁剧疆浣犳瘡涓€灞傜殑閫夋嫨
self.choice0 = self.setLayerChoice(0, map_nn(["gat", "feast", "transformer"]), key="conv")
self.choice1 = self.setLayerChoice(1, map_nn([1, 2, 4, 8]), key="head")
# 涓嶈蹇樿鍦ㄨ繖閲屽畾涔夊墠鍚戜紶鎾繃绋�
# 瀵逛簬闈炲崟鏍锋湰鑼冨紡锛屾偍鍙互鐩存帴杩斿洖閫夋嫨涓嬬殑妯″瀷
# ``YourModel`` 涔熷氨鏄偍鐨勬ā鍨嬪繀椤荤户鎵垮熀纭€绌洪棿
def parse_model(self, selection, device) -> BaseModel:
model = YourModel(selection, self.input_dim, self.output_dim).wrap()
return model
.. code-block:: python
# ``YourModel`` 涔熷氨鏄偍鐨勬ā鍨嬪畾涔夊涓�
class YourModel(BaseSpace):
def __init__(self, selection, input_dim, output_dim):
self.input_dim = input_dim
self.output_dim = output_dim
if selection["conv"] == "gat":
conv = GATConv
elif selection["conv"] == "feast":
conv = FeaStConv
elif selection["conv"] == "transformer":
conv = TransformerConv
self.layer = conv(input_dim, output_dim, selection["head"])
def forward(self, data):
x, edges = data.x, data.edge_index
y = self.layer(x, edges)
return y
鎬ц兘璇勪及鍣�
---------------------
鎬ц兘璇勪及鍣ㄧ敤浜庤瘎浼颁竴涓灦鏋勭殑浼樺姡. 鐩墠鎴戜滑鏀寔濡備笅涓€浜涜瘎浼板櫒:
+-------------------------+-------------------------------------------------------+
| 璇勪及鍣� | 鎻忚堪 |
+=========================+=======================================================+
| ``oneshot`` | 瀵逛簬缁欏畾鐨勫瓙鏋舵瀯锛屾棤闇€璁粌鍦扮洿鎺ヨ瘎浼板叾鏁堟灉 |
+-------------------------+-------------------------------------------------------+
| ``scratch`` | 瀵逛簬缁欏畾鐨勫瓙鏋舵瀯锛屼粠澶村紑濮嬭缁冪洿鍒版敹鏁涗箣鍚庡啀璇勪及鍏舵晥鏋� |
+-------------------------+-------------------------------------------------------+
鎮ㄤ篃鍙互鑷繁瀹氫箟涓€涓瘎浼板櫒. 涓嬮潰鏄竴涓棤闇€璁粌鍗冲彲璇勪及鏋舵瀯鏁堟灉鐨勮瘎浼板櫒鐨勪緥瀛� (閫氬父搴旂敤浜巓ne-shot space).
.. code-block:: python
# 渚嬪锛屾偍涔熷彲浠ヨ嚜宸卞畾涔変竴涓猠stimator
from autogl.module.nas.estimator.base import BaseEstimator
class YourOneShotEstimator(BaseEstimator):
# 鎮ㄦ墍闇€瑕佸仛鐨勫彧鏄畾涔塦`infer``杩欎釜鏂规硶
def infer(self, model: BaseSpace, dataset, mask="train"):
device = next(model.parameters()).device
dset = dataset[0].to(device)
# 瀵规灦鏋勭洿鎺ヨ繘琛屽墠鍚戜紶鎾�
pred = model(dset)[getattr(dset, f"{mask}_mask")]
y = dset.y[getattr(dset, f'{mask}_mask')]
# 浣跨敤榛樿鐨勬崯澶卞嚱鏁板拰璇勪环鎸囨爣鏉ヨ瘎浼版灦鏋勬晥鏋滐紝褰撶劧锛屽湪杩欓噷鎮ㄤ篃鍙互閫夋嫨鍏朵粬鐨勬崯澶卞嚱鏁板拰璇勪环鎸囨爣
loss = getattr(F, self.loss_f)(pred, y)
probs = F.softmax(pred, dim = 1)
metrics = [eva.evaluate(probs, y) for eva in self.evaluation]
return metrics, loss
鎼滅储绛栫暐
---------------
鎼滅储绛栫暐瀹氫箟浜嗗浣曞幓鎼滅储涓€涓ソ鐨勫瓙鏋舵瀯. 鐩墠鎴戜滑鏀寔濡備笅涓€浜涙悳绱㈢瓥鐣�:
+-------------------------+-------------------------------------------------------+
| 绛栫暐 | 鎻忚堪 |
+=========================+=======================================================+
| ``random`` | 閫氳繃鍧囧寑閲囨牱杩涜闅忔満鎼滅储 |
+-------------------------+-------------------------------------------------------+
| ``rl`` [1]_ | 閫氳繃寮哄寲瀛︿範鏂规硶鏉ヨ繘琛屾灦鏋勬悳绱� |
+-------------------------+-------------------------------------------------------+
| ``enas`` [2]_ | 閫氳繃鍏变韩鍙傛暟绛夋柟娉曪紝鏇撮珮鏁堝湴杩涜鏋舵瀯鎼滅储 |
+-------------------------+-------------------------------------------------------+
| ``darts`` [3]_ | 閫氳繃鍙井鏂规硶鏉ヨ繘琛屾灦鏋勬悳绱� |
+-------------------------+-------------------------------------------------------+
鍩轰簬閲囨牱鐨勯潪鍏变韩鏉冮噸鐨勬悳绱㈢瓥鐣ュ湪瀹炵幇涓婃洿鍔犵畝鍗�
鎺ヤ笅鏉ワ紝鎴戜滑灏嗗悜鎮ㄥ睍绀哄浣曡嚜瀹氫箟涓€涓熀浜嶥FS鐨勬悳绱㈢瓥鐣ユ潵浣滀负涓€涓緥瀛�
濡傛灉鎮ㄦ兂瑕佽嚜瀹氫箟鏇村澶嶆潅鐨勬悳绱㈢瓥鐣ワ紝鎮ㄥ彲浠ュ幓鍙傝€僋NI涓璂arts銆丒nas鎴栬€呭叾浠栨悳绱㈢瓥鐣ョ殑瀹炵幇
.. code-block:: python
from autogl.module.nas.algorithm.base import BaseNAS
class RandomSearch(BaseNAS):
# 鎺ユ敹闇€瑕侀噰鏍风殑鏁伴噺浣滀负鍒濆鍖�
def __init__(self, n_sample):
super().__init__()
self.n_sample = n_sample
# NAS绠楁硶娴佺▼涓殑鍏抽敭姝ラ锛岃繖涓柟娉曚細鏍规嵁缁欏畾鐨剆earch space銆乨ataset鍜宔stimator鍘绘悳绱竴涓悎鐞嗙殑鏋舵瀯
def search(self, space: BaseSpace, dset, estimator):
self.estimator=estimator
self.dataset=dset
self.space=space
self.nas_modules = []
k2o = get_module_order(self.space)
# 瀵绘壘骞跺瓨鍌╯earch space涓墍鏈夌殑mutables锛岃繖浜沵utables灏辨槸鎮ㄥ湪search space涓畾涔夌殑鍙悳绱㈢殑閮ㄥ垎
replace_layer_choice(self.space, PathSamplingLayerChoice, self.nas_modules)
replace_input_choice(self.space, PathSamplingInputChoice, self.nas_modules)
# 鏍规嵁缁欏畾鐨刼rders瀵筸utables杩涜鎺掑簭
self.nas_modules = sort_replaced_module(k2o, self.nas_modules)
# 寰楀埌鍖呭惈鎵€鏈夊彲鑳介€夋嫨鐨勪竴涓瓧鍏�
selection_range={}
for k,v in self.nas_modules:
selection_range[k]=len(v)
self.selection_dict=selection_range
arch_perfs=[]
# 瀹氫箟DFS鐨勬祦绋�
self.selection = {}
last_k = list(self.selection_dict.keys())[-1]
def dfs():
for k,v in self.selection_dict.items():
if not k in self.selection:
for i in range(v):
self.selection[k] = i
if k == last_k:
# 璇勪及涓€涓灦鏋勭殑鏁堟灉
self.arch=space.parse_model(self.selection,self.device)
metric,loss=self._infer(mask='val')
arch_perfs.append([metric, self.selection.copy()])
else:
dfs()
del self.selection[k]
break
dfs()
# 寰楀埌鍦ㄦ悳绱㈣繃绋嬩腑鎷ユ湁鏈€濂芥晥鏋滅殑鏋舵瀯
selection=arch_perfs[np.argmax([x[0] for x in arch_perfs])][1]
arch=space.parse_model(selection,self.device)
return arch
涓嶅悓鐨勬悳绱㈢瓥鐣ラ渶瑕佷笌鐗瑰畾鐨勬悳绱㈢┖闂翠笌璇勪及鍣ㄦ惌閰嶄娇鐢�
杩欎笌瀹冧滑鐨勫疄鐜扮浉鍏筹紝濡傞潪one-shot鐨勬悳绱㈢┖闂翠笉鑳戒笌one-shot鐨勬悳绱㈢瓥鐣ユ惌閰嶄娇鐢�
涓嬮潰鐨勮〃鏍间腑缁欏嚭浜嗘垜浠洰鍓嶆墍鏀寔鐨勬惌閰嶇粍鍚�
+----------------+-------------+-------------+------------------+
| Space | single path | GraphNAS[1] | GraphNAS-macro[1]|
+================+=============+=============+==================+
| Random | 鉁� | 鉁� | 鉁� |
+----------------+-------------+-------------+------------------+
| RL | 鉁� | 鉁� | 鉁� |
+----------------+-------------+-------------+------------------+
| GraphNAS [1]_ | 鉁� | 鉁� | 鉁� |
+----------------+-------------+-------------+------------------+
| ENAS [2]_ | 鉁� | | |
+----------------+-------------+-------------+------------------+
| DARTS [3]_ | 鉁� | | |
+----------------+-------------+-------------+------------------+
+----------------+-------------+-------------+
| Estimator | one-shot | Train |
+================+=============+=============+
| Random | | 鉁� |
+----------------+-------------+-------------+
| RL | | 鉁� |
+----------------+-------------+-------------+
| GraphNAS [1]_ | | 鉁� |
+----------------+-------------+-------------+
| ENAS [2]_ | 鉁� | |
+----------------+-------------+-------------+
| DARTS [3]_ | 鉁� | |
+----------------+-------------+-------------+
.. [1] Gao, Yang, et al. "Graph neural architecture search." IJCAI. Vol. 20. 2020.
.. [2] Pham, Hieu, et al. "Efficient neural architecture search via parameters sharing." International Conference on Machine Learning. PMLR, 2018.
.. [3] Liu, Hanxiao, Karen Simonyan, and Yiming Yang. "DARTS: Differentiable Architecture Search." International Conference on Learning Representations. 2018.
.. [4] Guo, Zichao, et al. 鈥淪ingle Path One-Shot Neural Architecture Search with Uniform Sampling.鈥� European Conference on Computer Vision, 2019, pp. 544鈥�560.