yjJddlmZddlZddlmZddlmZddlmZm Z m Z ddl Z ddl Z ddl mZmZddlmZmZddlmZdd lmZmZmZdd lmZd d lmZer dd l mZddlmZddl m!Z!Gdde Z"GddeZ#dZ$GddZ%Gdde%Z&dS)) annotationsN) defaultdict)Enum) TYPE_CHECKINGAny TypedDict)_C_ops _legacy_C_ops)core unique_name) check_type)Operator_dygraph_tracer in_pir_mode)in_dynamic_mode)amp_global_state)Tensor)OptimizerWithMixedPrecision) Optimizerc`eZdZUded<ded<ded<ded<ded<ded <ded <d ed <d S)_ScaleStateDictrscalefloat incr_ratio decr_ratiointincr_every_n_stepsdecr_every_n_nan_or_inf incr_count decr_countbooluse_dynamic_loss_scalingN)__name__ __module__ __qualname____annotations__f/lsinfo/ai/hellotax_ai/data_center/backend/venv/lib/python3.11/site-packages/paddle/amp/grad_scaler.pyrr)si $$$$&&&&&&r)rceZdZdZdZdZdS)OptimizerStaterrN)r$r%r&INITUNSCALEDSTEPPEDr(r)r*r,r,4s DHGGGr)r,cdtjiS)Nstate)r,r.r(r)r*_refresh_optimizer_stater3:s ^( ))r)ceZdZdZ d7d8dZd9dZd:dZdZd Zd;d!Z d;d"Z dd(Zd>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) T@@?renabler"init_loss_scalingrrrrrrr#returnNonectrt}|std|rj|jsQ|js8|jstjd|jdd}||_ d|_ d|_ d|_ |j r|dks Jd|dks Jd||_ ||_ ||_||_||_d|_d|_||_ t'rjt(jjd d gt1jd t(jj|j  |_ dSt)jt=jdg t<j!|_"t)jt=jdg t<j!|_#t)jt=jdg t<j!|_$t)jt=jdg t<j!|_%t)jt=jdg t<j!|_&t)jt=j|j g t<j'|_ d|_(tStT|_+dSdS)Nz;current_tracer is None, maybe it is not in imperative mode.zWAmpScaler can only be enabled on CUDAPlace, XPUPlace and CustomPlace, current place is z, so it makes no effect.F?zThe incr_ratio must be > 1.0.zThe decr_ratio must be < 1.0.rfloat32r loss_scaling)value)dtypeshapename initializer),rr ValueError_expected_place is_gpu_place is_xpu_placeis_custom_placewarningswarn_enable_use_dynamic_loss_scaling_init_loss_scaling_scale _incr_ratio _decr_ratio_incr_every_n_steps_decr_every_n_nan_or_inf _incr_count _decr_countrpaddlepirr create_persistable_valuer generatennrFConstantInitializer to_tensornparrayastypebool_ _found_inf_temp_found_inf_value_false_temp_found_inf_fp16_temp_found_inf_bf16_temp_found_inf_fp32r@_cache_found_infrr3_optimizer_states) selfr:r;rrrrr#tracers r*__init__zAmpScaler.__init__os=    $&&F  Q &3355 )6688 )99;;   _ntoE___ ).&"% <* O###%D######%D###&7D #)D )D '9D $,CD ) D  D -ED *}} O$joFF##$-n== & 5 I I"5!J!! G #)"2HaSMM((22##4:3CHaSMM((22440-3,<HaSMM((22--)-3,<HaSMM((22--)-3,<HaSMM((22--)%.Hd5677>>rzJJ )-%)45M)N)N&&&U* O* Or)varrct|dtjtjjfd|jr\t jdkrE|jr>d|_d|_d|_ tj dt jdtr|j tjjkr|d}|js|Stj||j}|}|}|jrz|jrstjjj|jj|j|j|jj|_|S|jr|s|S||jzS) al Multiplies a Tensor by the scale factor and returns scaled outputs. If this instance of :class:`AmpScaler` is not enabled, output are returned unmodified. Args: var (Tensor): The Tensor to scale. Returns: The scaled Tensor or original Tensor. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) rmzAmpScaler.scale()float16Fr?z6It is not recommended to use dynamic loss scaling for z&, so GradScaler is disable by default.r@)r rXrrYValuerNr amp_dtyperOrPrLrMrrCr DataTypeFLOAT32rar multiplyrQget_defining_op dist_attrbase libpaddlecreate_op_dist_attribute process_meshoperandsresultschunk_id_is_initialized)rjrm scale_out multiply_op src_var_ops r*rzAmpScaler.scales<    ]FJ, -      L  "", 99.:!DL-2D *&)D # MNIYI[I[IeNNN    == yDM111jj++1   ..sDK@@I#3355K,,..J$ )= K)-FF#-:#-6688#-5577",5 % | 3#6#6#8#8 JT[  r) optimizer'Optimizer | OptimizerWithMixedPrecisionargsrkwargs2tuple[list[Operator], list[tuple[Tensor, Tensor]]]ctrt|tjjjjsJ|j|_|j|_|j |_ |d|_ |jr>|j |_ |j |_ |j|_|j|_d|_d|_|j|i|S|js |j|i|S|jt+|}|dt,jur||d\}}t3|drF|d|j|j|i|\}}|d|_n&|jrd|_n|j|i|\}}d|_|jr|t?t@|_||fS) a This function is similar as `Optimizer.minimize()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Finally, the loss scaling ratio is updated. Args: optimizer(Optimizer): The optimizer used to update parameters. args: Arguments, which will be forward to `Optimizer.minimize()`. kwargs: Keyword arguments, which will be forward to `Optimizer.minimize()`. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, ... parameters=model.parameters() ... ) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) rNr2)NN_set_auxiliary_var found_infTF)!r isinstancerXstaticamp decoratorrrOrPrQ _loss_scaling _scaled_lossrTrUrRrS_num_good_steps_num_bad_stepsminimizerNriidr,r._unscalehasattrrrc_get_auxiliary_varrh_updaterr3)rjrrroptimizer_state optimize_ops params_gradss r*rzAmpScaler.minimizesT == 7 !+G  372PI /+/+BI (&*kI #%)!WI "- 0040H -12)-(8 %(,(8 %,0 )+/ (%9%t6v66 6| 7%9%t6v66 60I? 7 #~': : : MM) $ $ $%1" l 92 3 3 .  ( (do F F F););T)LV)L)L &L,$-$@$@$M$MD ! ! .(,%%-?Y-?-P-P-P* l(-%  )  LLNNN!,-E!F!F\))r)c|jsdS|jt|}|dtjurt d|dtjurt dt|ddrCt|j dtr"g}g}g}g}|j D]}|dD]}| | | | j tjkr(| | | j tjkr(| | | | nlt#r)t$j|j\}}}n5d|jD}d |D}d |D}d |D}|j|_t1|rEt3j||j||jt;j|j|j|_t1|rEt3j||j||jt;j|j|j|_t1|rEt3j||j||j t;j|j|j |_tj|d<dS) a Unscale the gradients of parameters, multiplies the gradients of parameters by 1/(loss scaling ratio). If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: optimizer(Optimizer): The optimizer used to update parameters. Returns: The unscaled parameters or original parameters. Nr2zMunscale_() has already been called on this optimizer since the last update().z(unscale_() is being called after step(). _param_groupsrparamsc^g|]*}||+SN) _grad_ivar.0params r* z&AmpScaler._unscale..s@''))5$$&&555r)c<g|]}|jtjk|Sr()rCrXrors r*rz&AmpScaler._unscale..1$$${fn44444r)c<g|]}|jtjk|Sr()rCrXbfloat16rs r*rz&AmpScaler._unscale..s1$$${fo55555r)c<g|]}|jtjk|Sr()rCrXr@rs r*rz&AmpScaler._unscale..rr))!rNrirr,r/ RuntimeErrorr0getattrrrdictrappendrCrXrorrr eagerget_grads_lists_parameter_listrdrclenr check_finite_and_unscalerQrer bitwise_orrfrg) rjrr param_gradsparam_grads_fp16param_grads_bf16param_grads_fp32grouprs r*rzAmpScaler._unscalebs|  F0I? 7 #~'> > >_ W %)? ? ?IJJ J 9ot 4 40   #A &: : 0 K! ! ! "0 H H"8_HHE''))5#**5+;+;+=+=>>> ++--3v~EE,33E4D4D4F4FGGGG"--//5HH,33E4D4D4F4FGGGG,33E4D4D4F4FGGGH H   J..y/HII $$$$ !*!: $$!,$$$ $$!,$$$ $$!,$$$ :    2   )     %/!:DO    2   )     %/!:DO    2   )     %/!:DO$2#:   r)c |jsdS|jrd|_|jdz|_|j|jkrit dt |jdt |jdt |j|j|jz|_d|_nAd|_|jdz|_|j|j kr|j|j z|_d|_dS)z+ Updates the loss_scaling. Nrrz$Found inf or nan, current scale is: z, decrease to: *) rNrhrVrWrUprintrrQrSrTrRrjs r*rzAmpScaler._updates|  F   % D #/!3D 4#@@@M5;M;MMM^cdhdo^p^pMMsxy}zJtKtKMM#kD,<< #$  D #/!3D 4#;;;"kD,<< #$ r)c|jS)z Enable loss scaling or not. Returns: bool: enable loss scaling return True else return False. )rNrs r* is_enablezAmpScaler.is_enables |r)c|jS)z Whether to use dynamic loss scaling. Returns: bool: if fixed loss_scaling is used return False, if the loss scaling is updated dynamically return true. )rOrs r*is_use_dynamic_loss_scalingz%AmpScaler.is_use_dynamic_loss_scalings --r)c|jS)z Return the initial loss scaling factor. Returns: float: the initial loss scaling factor. )rPrs r*get_init_loss_scalingzAmpScaler.get_init_loss_scalings &&r)new_init_loss_scalingc||_tjtj|jgtj|_dS)z Set the initial loss scaling factor by `new_init_loss_scaling`. Args: new_init_loss_scaling(int): The new_init_loss_scaling used to update initial loss scaling factor.s N)rPrXr^r_r`rar@rQ)rjrs r*set_init_loss_scalingzAmpScaler.set_init_loss_scalingsG#8& Hd-. / / 6 6rz B B   r)c|jS)z Return the multiplier to use when increasing the loss scaling. Returns: float: the multiplier to use when increasing the loss scaling. rRrs r*get_incr_ratiozAmpScaler.get_incr_ratio  r)new_incr_ratioc4|dks Jd||_dS)a  Set the multiplier to use when increasing the loss scaling by `new_incr_ratio`, `new_incr_ratio` should > 1.0. Args: new_incr_ratio(float): The new_incr_ratio used to update the multiplier to use when increasing the loss scaling. r?z!The new_incr_ratio must be > 1.0.Nr)rjrs r*set_incr_ratiozAmpScaler.set_incr_ratio,###%H###)r)c|jS)z Get the less-than-one-multiplier to use when decreasing the loss scaling. Returns: float: the less-than-one-multiplier to use when decreasing the loss scaling. rSrs r*get_decr_ratiozAmpScaler.get_decr_ratiorr)new_decr_ratioc4|dks Jd||_dS)a) Set the less-than-one-multiplier to use when decreasing the loss scaling by `new_incr_ratio`, `new_decr_ratio` should < 1.0. Args: new_decr_ratio(float): The new_decr_ratio used to update the less-than-one-multiplier to use when decreasing the loss scaling. r?z!The new_decr_ratio must be < 1.0.Nr)rjrs r*set_decr_ratiozAmpScaler.set_decr_ratio&rr)c|jS)a Return the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Returns: int: the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. rTrs r*get_incr_every_n_stepsz AmpScaler.get_incr_every_n_steps0s ''r)new_incr_every_n_stepsc||_dS)a^ Set the num `n` by `new_incr_every_n_steps`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Args: new_incr_every_n_steps(int): The new_incr_every_n_steps used to update the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Nr)rjrs r*set_incr_every_n_stepsz AmpScaler.set_incr_every_n_steps9s$:   r)c|jS)a Return the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Returns: int: the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. rUrs r*get_decr_every_n_nan_or_infz%AmpScaler.get_decr_every_n_nan_or_infBs ,,r)new_decr_every_n_nan_or_infc||_dS)au Set the num `n` by `new_decr_every_n_nan_or_inf`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Args: new_decr_every_n_nan_or_inf(int): The new_decr_every_n_nan_or_inf used to update the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Nr)rjrs r*set_decr_every_n_nan_or_infz%AmpScaler.set_decr_every_n_nan_or_infKs)D%%%r)rc |jrE|j|j|j|j|j|j|j|j dniS)a Returns the state of the scaler as a `dict`, If this instance is not enabled, returns an empty dict. Returns: A dict of scaler includes: scale (tensor): The loss scaling factor. incr_ratio(float): The multiplier to use when increasing the loss scaling. decr_ratio(float): The less-than-one-multiplier to use when decreasing the loss scaling. incr_every_n_steps(int): Increases loss scaling every n consecutive steps with finite gradients. decr_every_n_nan_or_inf(int): Decreases loss scaling every n accumulated steps with nan or inf gradients. incr_count(int): The number of recent consecutive unskipped steps. decr_count(int): The number of recent consecutive skipped steps. use_dynamic_loss_scaling(bool): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamically. Default is True. )rrrrrr r!r#) rNrQnumpyrRrSrTrUrVrWrOrs r* state_dictzAmpScaler.state_dictVsb4| **,,".".&*&>+/+H".".,0,J    r)rc|jsdSt|dkrtd|dd|_t jt j|jgt j |_ |d|_ |d|_ |d|_ |d|_|d |_|d |_|d |_dS) z Loads the scaler state. Args: state_dict(dict): scaler state. Should be an object returned from a call to `AmpScaler.state_dict()`. NrzdThe input state dict is empty, possibly because it was saved from a disabled instance of GradScaler.rrrrrr r!r#)rNrrrPrXr^r_r`rar@rQrRrSrTrUrVrWrO)rjrs r*load_state_dictzAmpScaler.load_state_dictts|  F z??a  :  #-W"5a"8& Hd-. / / 6 6rz B B   &l3%l3#-.B#C (23L(M%%l3%l3)34N)O&&&r)N)Tr6r7r8r9rTr:r"r;rrrrrrrrrr#r"r<r=rmrr<rrrrrrrr<rr<r"r<rrrr<r=rrr<r=rrr<r=r<rrrr<r=rrr<r=r<rrrr<r=)r$r%r&__doc__rlrrrrrrrrrrrrrrrrrrr(r)r*r5r5>s..d#*"&'()-JOJOJOJOJOXH!H!H!H!T[*[*[*[*zf;f;f;P2....''''         ****    ****((((::::---- D D D D    <PPPPPPr)r5ceZdZdZ d8d9fd Zd:fd Zd;fd Zdfd# Z d>fd$ Z d?fd% Z d@fd' Z d?fd( ZdAfd* Zd?fd+ ZdBfd- ZdCfd. ZdDfd0 ZdCfd1 ZdEfd3 ZdFfd5 ZdGfd7 ZxZS)H GradScaleram GradScaler is used for Auto-Mixed-Precision training in dynamic graph mode. It controls the scaling of loss, helps avoiding numerical overflow. The object of this class has nineteen methods `scale()`, `unscale_()`, `minimize()`, `step()`, `update()` and `get`/`set` api of parameters. `scale()` is used to multiply the loss by a scale ratio. `unscale_()` is used to unscale the gradients of parameters, multiplies the gradients of parameters by 1/(scale ratio) `minimize()` is similar as `optimizer.minimize()`, performs parameters updating, and it will update the loss_scaling, it equal to `step()` + `update()`. `step()` is similar as `optimizer.step()`, which performs parameters updating. `update` is used to update the loss_scaling. Commonly, it is used together with `paddle.amp.auto_cast` to achieve Auto-Mixed-Precision in dynamic graph mode. Args: enable(bool, optional): Enable loss scaling or not. Default is True. init_loss_scaling (float, optional): The initial loss scaling factor. Default is 65536.0. incr_ratio(float, optional): The multiplier to use when increasing the loss scaling. Default is 2.0. decr_ratio(float, optional): The less-than-one-multiplier to use when decreasing the loss scaling. Default is 0.5. incr_every_n_steps(int, optional): Increases loss scaling every n consecutive steps with finite gradients. Default is 2000. decr_every_n_nan_or_inf(int, optional): Decreases loss scaling every n accumulated steps with nan or inf gradients. Default is 1. use_dynamic_loss_scaling(bool, optional): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamically. Default is True. Returns: An GradScaler object. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() T@r7r8rr:r"r;rrrrrrr#r<r=c Vt|||||||dSr)superrl) rjr:r;rrrrr# __class__s r*rlzGradScaler.__init__s@       # $     r)rmrcFt|S)aJ Multiplies a Tensor by the scale factor and returns scaled outputs. If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: var (Tensor): The tensor to scale. Returns: The scaled tensor or original tensor. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() )rr)rjrmrs r*rzGradScaler.scales<ww}}S!!!r)rrrrrrc>tj|g|Ri|S)a This function is similar as `optimizer.minimize()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Finally, the loss scaling ratio is updated. Args: optimizer(Optimizer): The optimizer used to update parameters. args: Arguments, which will be forward to `optimizer.minimize()`. kwargs: Keyword arguments, which will be forward to `optimizer.minimize()`. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() )rr)rjrrrrs r*rzGradScaler.minimizes-N uww ;D;;;F;;;r)rcr|js|S|jt|}|dtjurt d|dtjur||t|drJ| d|j || d|_ n*|j rd|_ n|d|_ tj|d<|jstt |_dSdS)at This function is similar as `optimizer.step()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Args: optimizer(Optimizer): The optimizer used to update parameters. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.step(optimizer) # update parameters >>> scaler.update() # update the loss scaling ratio >>> optimizer.clear_grad() r2z7step() has already been called since the last update().rrTFN)rNsteprirr,r0rr.rrrrcrrhrOrr3)rjrrs r*rzGradScaler.step"s=>| $>>## #0I? 7 #~'= = =I  7 #~': : : MM) $ $ $ 92 3 3 .  ( (do F F F NN   $-$@$@$M$MD ! ! .(,%%   (-%#1#9 - K%01I%J%JD " " " K Kr)c|jsdS|jr-|tt|_dS)a Updates the loss_scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.step(optimizer) # update parameters >>> scaler.update() # update the loss scaling ratio >>> optimizer.clear_grad() N)rNrOrrr3rirs r*updatezGradScaler.update^sB2|  F  ) K LLNNN%01I%J%JD "r)cFt|S)aE Unscale the gradients of parameters, multiplies the gradients of parameters by 1/(loss scaling ratio). If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: optimizer(Optimizer): The optimizer used to update parameters. Returns: The unscaled parameters or original parameters. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.unscale_(optimizer) # unscale the parameter >>> scaler.step(optimizer) >>> scaler.update() >>> optimizer.clear_grad() )rr)rjrrs r*unscale_zGradScaler.unscale_~sBww ***r)cDtS)a Enable loss scaling or not. Returns: bool: enable loss scaling return True else return False. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> enable = scaler.is_enable() >>> print(enable) True )rrrjrs r*rzGradScaler.is_enables2ww  """r)cDtS)ax Whether to use dynamic loss scaling. Returns: bool: if fixed loss_scaling is used return False, if the loss scaling is updated dynamically return True. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> use_dynamic_loss_scaling = scaler.is_use_dynamic_loss_scaling() >>> print(use_dynamic_loss_scaling) True )rrrs r*rz&GradScaler.is_use_dynamic_loss_scaling2ww22444r)cDtS)a& Return the initial loss scaling factor. Returns: float: the initial loss scaling factor. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> init_loss_scaling = scaler.get_init_loss_scaling() >>> print(init_loss_scaling) 1024 )rrrs r*rz GradScaler.get_init_loss_scalings2ww,,...r)rcJt|dS)a Set the initial loss scaling factor by `new_init_loss_scaling`. Args: new_init_loss_scaling(float): The new_init_loss_scaling used to update initial loss scaling factor. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_init_loss_scaling()) 1024 >>> new_init_loss_scaling = 1000 >>> scaler.set_init_loss_scaling(new_init_loss_scaling) >>> print(scaler.get_init_loss_scaling()) 1000 N)rr)rjrrs r*rz GradScaler.set_init_loss_scalings$8 %%&;<<<< Return the multiplier to use when increasing the loss scaling. Returns: float: the multiplier to use when increasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> incr_ratio = scaler.get_incr_ratio() >>> print(incr_ratio) 2.0 )rrrs r*rzGradScaler.get_incr_ratio2ww%%'''r)rcJt|dS)a Set the multiplier to use when increasing the loss scaling by `new_incr_ratio`, `new_incr_ratio` should > 1.0. Args: new_incr_ratio(float): The new_incr_ratio used to update the multiplier to use when increasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_incr_ratio()) 2.0 >>> new_incr_ratio = 3.0 >>> scaler.set_incr_ratio(new_incr_ratio) >>> print(scaler.get_incr_ratio()) 3.0 N)rr)rjrrs r*rzGradScaler.set_incr_ratio+#8 ~.....r)cDtS)aW Get the less-than-one-multiplier to use when decreasing the loss scaling. Returns: float: the less-than-one-multiplier to use when decreasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> decr_ratio = scaler.get_decr_ratio() >>> print(decr_ratio) 0.5 )rrrs r*rzGradScaler.get_decr_ratioIr r)rcJt|dS)a7 Set the less-than-one-multiplier to use when decreasing the loss scaling by `new_incr_ratio`, `new_decr_ratio` should < 1.0. Args: new_decr_ratio(float): The new_decr_ratio used to update the less-than-one-multiplier to use when decreasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_decr_ratio()) 0.5 >>> new_decr_ratio = 0.1 >>> scaler.set_decr_ratio(new_decr_ratio) >>> print(scaler.get_decr_ratio()) 0.1 N)rr)rjrrs r*rzGradScaler.set_decr_ratiodr r)cDtS)a Return the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Returns: int: the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> incr_every_n_steps = scaler.get_incr_every_n_steps() >>> print(incr_every_n_steps) 1000 )rrrs r*rz!GradScaler.get_incr_every_n_stepss2ww--///r)rcJt|dS)a Set the num `n` by `new_incr_every_n_steps`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Args: new_incr_every_n_steps(int): The new_incr_every_n_steps used to update the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_incr_every_n_steps()) 1000 >>> new_incr_every_n_steps = 2000 >>> scaler.set_incr_every_n_steps(new_incr_every_n_steps) >>> print(scaler.get_incr_every_n_steps()) 2000 N)rr)rjrrs r*rz!GradScaler.set_incr_every_n_stepss$8 &&'=>>>>>r)cDtS)a Return the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Returns: int: the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> decr_every_n_nan_or_inf = scaler.get_decr_every_n_nan_or_inf() >>> print(decr_every_n_nan_or_inf) 2 )rrrs r*rz&GradScaler.get_decr_every_n_nan_or_infrr)rcJt|dS)a Set the num `n` by `new_decr_every_n_nan_or_inf`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Args: new_decr_every_n_nan_or_inf(int): The new_decr_every_n_nan_or_inf used to update the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_decr_every_n_nan_or_inf()) 2 >>> new_decr_every_n_nan_or_inf = 3 >>> scaler.set_decr_every_n_nan_or_inf(new_decr_every_n_nan_or_inf) >>> print(scaler.get_decr_every_n_nan_or_inf()) 3 N)rr)rjrrs r*rz&GradScaler.set_decr_every_n_nan_or_infs$< ++,GHHHHHr)rcDtS)a0 Returns the state of the scaler as a `dict`, If this instance is not enabled, returns an empty dict. Returns: A dict of scaler includes: scale (tensor): The loss scaling factor. incr_ratio(float): The multiplier to use when increasing the loss scaling. decr_ratio(float): The less-than-one-multiplier to use when decreasing the loss scaling. incr_every_n_steps(int): Increases loss scaling every n consecutive steps with finite gradients. decr_every_n_nan_or_inf(int): Decreases loss scaling every n accumulated steps with nan or inf gradients. incr_count(int): The number of recent consecutive unskipped steps. decr_count(int): The number of recent consecutive skipped steps. use_dynamic_loss_scaling(bool): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamically. Default is True. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> scaler_state = scaler.state_dict() )rrrs r*rzGradScaler.state_dictsDww!!###r)rcJt|dS)a: Loads the scaler state. Args: state_dict(dict): scaler state. Should be an object returned from a call to `GradScaler.state_dict()`. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> scaler_state = scaler.state_dict() >>> scaler.load_state_dict(scaler_state) N)rr)rjrrs r*rzGradScaler.load_state_dicts#4  +++++r))Trr7r8rrTrrr)rrr<r=)r<r=rrrrrrrrrr)r$r%r&rrlrrrrrrrrrrrrrrrrrrr __classcell__)rs@r*rrs11j#*"&'()-       (""""""@'<'<'<'<'<'r s+#"""""######  (((((((())))))))......HHHHHHHHHH,,,,,,'''''' 'GGGGGG;;;;;;''''')'''T ***P PP PP PP PP PP PP PP Pfc ,c ,c ,c ,c ,c ,c ,c ,c ,c ,r)