|j[9ddlmZddlZddlmZddlZddlmZddlmZm Z m Z ddl m Z m Z mZdd lmZmZerdd lmZgZGd d eZGd deZGddeZdS)) annotationsN) TYPE_CHECKING)_C_ops)core framework unique_name)_current_expected_placein_dygraph_mode in_pir_mode) Initializercalculate_gain) _NonLinearityc>eZdZdZ ddfd Z dddZxZS)MSRAInitializera]Implements the MSRA initializer a.k.a. Kaiming Initializer This class implements the weight initialization from the paper `Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification `_ by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. This is a robust initialization method that particularly considers the rectifier nonlinearities. In case of Uniform distribution, the range is [-x, x], where .. math:: x = gain \times \sqrt{\frac{3}{fan\_in}} In case of Normal distribution, the mean is 0 and the standard deviation is .. math:: \frac{gain}{\sqrt{{fan\_in}}} Args: uniform (bool, optional): whether to use uniform or normal distribution. Default is True. fan_in (float32|None, optional): fan_in (in_features) of trainable Tensor, If None, it will be inferred automatically. If you don't want to use in_features of the Tensor, you can set the value of 'fan_in' smartly by yourself. Default is None. seed (int32, optional): random seed. Default is 0. negative_slope (float, optional): negative_slope (only used with leaky_relu). Default is 0.0. nonlinearity(str, optional): the non-linear function. Default is relu. mode(str, optional): the mode of initialization, can be 'fan_in' or 'fan_out'. When set to 'fan_in', the fan_in parameter is used for initialization. When set to 'fan_out', the out_features of trainable Tensor will be used. Default is 'fan_in'. Note: It is recommended to set fan_in to None for most cases. TNrrelufan_inuniformbool float | Noneseedintnegative_slopefloat nonlinearityrmodestrreturnNonec4|J|Jt||_||_||_||_||_||_|jdvrtd|jd|jdkr|jtddSdS)zConstructor for MSRAInitializerN)rfan_outzWThe mode of KaimingNormal/KaimingUniform should be 'fan_in' or 'fan_out', but received .r"zdThe mode of KaimingNormal/KaimingUniform is 'fan_out', but fan_in is set. Please set fan_in to None.) super__init___uniform_fan_in_seed_negative_slope _nonlinearity_mode ValueError)selfrrrrrr __class__s m/lsinfo/ai/hellotax_ai/data_center/backend/venv/lib/python3.11/site-packages/paddle/nn/initializer/kaiming.pyr%zMSRAInitializer.__init__Hs"""    -) :2 2 2. $ ...  : " "t|'?@  # "'?'?var paddle.Tensorblockpaddle.pir.Block | Nonepaddle.Tensor | Nonec  t|tjr|r Jd||}t|tjt jjt jj j fsJt|tj t jj fsJ| |\}}|j dkr|j|n|j}|j dkr|}|jdkr|jj|_|j}|tjjjks!|tjjjkr~|jswtjjj}|t5jdd|jdg|j|tjjjd }nC|tj j!tj j"fvr|jstj j#}|}n|}|}tIr|jrtK|j&|j'} | tQj)d tU|z z} tWj,|j|| | |j|j-.r|j-n t_}ntK|j&|j'} | tQj)tU|z } |j-.r|j-n t_} tWj0|jd | |j|| }|tjjjks@|tjjjtj j!tj j"fvr2|js+tWj1||} | 2|n|2|dStgr*|jrrtK|j&|j'} | tQj)d tU|z z} tWj,|j|| | |jt_}notK|j&|j'} | tQj)tU|z } t_} tWj0|jd | |j|| }|tj j!tj j"fvr|jstWj1||S|S|jr|tK|j&|j'} | tQj)d tU|z z} |4d id|i|jtk|| | |jdd}nvtK|j&|j'} | tQj)tU|z } |4dd|i|jtk|d | |jdd}|tjjjks!|tjjjkr,|js%|4dd|id|i|j|d||_6|S)a\Initialize the input tensor with MSRA initialization. Args: var(Tensor): Tensor that needs to be initialized. block(Block|None, optional): The block in which initialization ops should be added. Used in static graph only, default None. Returns: The initialization op. zDCurrently, kaiming initializer not support lazy init for dist param.rNr"rr# masra_inittmpF)nameshapedtypetype persistableg@uniform_randomOut)r:r;minmaxrT)r<inputsoutputsattrs stop_gradientgaussian_random)r:r;meanstdr)r<rDrErFcastX)in_dtype out_dtype)r<rCrDrE)7 isinstancerEagerParamBaseis_dist _check_blockVariablepaddlepirValuer ParameterMetaBlock _compute_fansr+r'r(program random_seedr;VarDescVarTypeFP16BF16r&FP32 create_varr generatejoinr9r: DENSE_TENSORDataTypeFLOAT16BFLOAT16FLOAT32r rr*r)mathsqrtrrrplace_typer gaussianrJ_share_underline_tensor_tor append_oprop)r-r1r3f_inf_outr origin_dtyperMout_vargainlimitrIrjvar_tmpros r/forwardzMSRAInitializer.forwardfsE sI4 5 5 :=++--   S   !!%(( "   -      %)/6:3C!DEEEEE((-- e : ! !!\1TTt|F : " "F :??2DJy 4</4 4 4 DL05 5 5dm 5 ,1I&& )HHlCHe<==i\)6!'GG T]2DM4JK K KM L -IGG$IG   t } %d&8$:NOOtyuV}})<=== .IFJy((3CII022  &d&8$:NOOTYuV}}555y((3CII022 !/M3TZEt|3888L(-M)M*  !+g|<<22377772237774 ]]K } %d&8$:NOOtyuV}})<=== .IFJ+-- &d&8$:NOOTYuV}}555/11 /M3TZE !6 8N OOO P{7L999N} %d&8$:NOOtyuV}})<===__)"G,!(!$Y %v$ $ #'%  &d&8$:NOOTYuV}}555__*"G,!(!$Y #" $ #'%  t|3888 4 999$-9>"CL$+M%1  CFIr0)TNrrrr)rrrrrrrrrrrrrr )N)r1r2r3r4rr5)__name__ __module__ __qualname____doc__r%rw __classcell__r.s@r/rr&sF# !&,>DHtttttttttr0rc.eZdZdZ ddfd ZxZS) KaimingNormala_Implements the Kaiming Normal initializer This class implements the weight initialization from the paper `Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification `_ by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. This is a robust initialization method that particularly considers the rectifier nonlinearities. In case of Normal distribution, the mean is 0 and the standard deviation is .. math:: \frac{gain}{\sqrt{{fan\_in}}} Args: fan_in (float32|None, optional): fan_in (in_features) of trainable Tensor, If None, it will be inferred automatically. If you don't want to use in_features of the Tensor, you can set the value of 'fan_in' smartly by yourself. Default is None. negative_slope (float, optional): negative_slope (only used with leaky_relu). Default is 0.0. nonlinearity(str, optional): the non-linear function. Default is relu. mode(str, optional): the mode of initialization, can be 'fan_in' or 'fan_out'. When set to 'fan_in', the fan_in parameter is used for initialization. When set to 'fan_out', the out_features of trainable Tensor will be used. Default is 'fan_in'. Note: It is recommended to set fan_in to None for most cases. Examples: .. code-block:: python >>> import paddle >>> import paddle.nn as nn >>> linear = nn.Linear(2, 4, weight_attr=nn.initializer.KaimingNormal()) >>> data = paddle.rand([30, 10, 2], dtype='float32') >>> res = linear(data) Nr>rrrrrrrrrr cVtd|d|||dS)NFrrrrrrrr$r%r-rrrrr.s r/r%zKaimingNormal.__init__CsB )%      r0Nr>rr rrrrrrrrrr rxryrzr{r%r|r}s@r/rrs[##N $ #"            r0rc.eZdZdZ ddfd ZxZS)KaimingUniformaYImplements the Kaiming Uniform initializer This class implements the weight initialization from the paper `Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification `_ by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. This is a robust initialization method that particularly considers the rectifier nonlinearities. In case of Uniform distribution, the range is [-x, x], where .. math:: x = gain \times \sqrt{\frac{3}{fan\_in}} Args: fan_in (float32|None, optional): fan_in (in_features) of trainable Tensor, If None, it will be inferred automatically. If you don't want to use in_features of the Tensor, you can set the value of 'fan_in' smartly by yourself. Default is None. negative_slope (float, optional): negative_slope (only used with leaky_relu). Default is 0.0. nonlinearity(str, optional): the non-linear function. Default is relu. mode(str, optional): the mode of initialization, can be 'fan_in' or 'fan_out'. When set to 'fan_in', the fan_in parameter is used for initialization. When set to 'fan_out', the out_features of trainable Tensor will be used. Default is 'fan_in'. Note: It is recommended to set fan_in to None for most cases. Examples: .. code-block:: python >>> import paddle >>> import paddle.nn as nn >>> linear = nn.Linear(2, 4, weight_attr=nn.initializer.KaimingUniform()) >>> data = paddle.rand([30, 10, 2], dtype='float32') >>> res = linear(data) Nr>rrrrrrrrrr cVtd|d|||dS)NTrrrrs r/r%zKaimingUniform.__init__ysB )%      r0rrrr}s@r/rrTs[""L $ #"            r0r) __future__rrhtypingrrSrbaserrr base.frameworkr r r initializerrrr__all__rrrr0r/rsl#""""" 0000000000 54444444+****** tttttktttn4 4 4 4 4 O4 4 4 n3 3 3 3 3 _3 3 3 3 3 r0