|j9<ddlmZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z m Z mZdd lmZgZGd d eZGd d eZGddeZdS)) annotationsN)_C_ops)core framework unique_name)check_variable_and_dtype)_current_expected_placein_dygraph_mode in_pir_mode) Initializerc<eZdZdZ ddfd Z dddZxZS)XavierInitializera This class implements the Xavier weight initializer from the paper `Understanding the difficulty of training deep feedforward neural networks `_ by Xavier Glorot and Yoshua Bengio. This initializer is designed to keep the scale of the gradients approximately same in all the layers. In case of Uniform distribution, the range is [-x, x], where .. math:: x = gain \times \sqrt{\\frac{6.0}{fan\_in + fan\_out}} In case of Normal distribution, the mean is 0 and the standard deviation is .. math:: gain \times \sqrt{\\frac{2.0}{fan\_in + fan\_out}} Args: uniform (bool, optional): whether to use uniform ,if False use normal distribution. Default is True. fan_in (float|None, optional): fan_in for Xavier initialization. If None, it is inferred from the variable. Default is None. fan_out (float|None, optional): fan_out for Xavier initialization. If None, it is inferred from the variable. Default is None. seed (int, optional): Random seed. Default is 0. gain (float, optional): Scaling Tensor. Default is 1.0. Note: It is recommended to set fan_in and fan_out to None for most cases. TNr?uniformboolfan_in float | Nonefan_outseedintgainfloatreturnNonec|J|Jt||_||_||_||_||_dSN)super__init___uniform_fan_in_fan_out_seed_gain)selfrrrrr __class__s l/lsinfo/ai/hellotax_ai/data_center/backend/venv/lib/python3.11/site-packages/paddle/nn/initializer/xavier.pyr zXavierInitializer.__init__GsZ"""      var paddle.Tensorblockpaddle.pir.Block | Nonepaddle.Tensor | Nonec t ||}t|tjtjjfsJt|tjjjst|dgdd| |\}}|j |n|j }|j |n|j }|j dkr|j j|_ t|tjr|r|jn|j}|j}|t jjjks!|t jjjkry|jsrt jjj} |t7jdd|jdg|| t jjjd } nC|t j j!t j j"fvr|jst j j#} |} n|} |} tIr|jrdd||fvrd } n/|j%tMj'd tQ||zz z} tSj*|| | | |j tW} ndd||fvrd } n/|j%tMj'd tQ||zz z} tW} tSj,|d | |j | | } |t jjjks@|t jjjt j j!t j j"fvr|jstSj-| |} t|tjrI|r5tj.j/j01| |j2|j3} | 4|dStkr$|jrtd||fvrd } n/|j%tMj'd tQ||zz z} tj6*| j| | | |j tW} ngd||fvrd } n/|j%tMj'd tQ||zz z} tSj,| jd | |j | tW} |t j j!t j j"fvr|jstSj-| |S| S|jrfd||fvrd } n/|j%tMj'd tQ||zz z} |7d id| i| j| | | |j dd}nhd||fvrd } n/|j%tMj'd tQ||zz z} |7dd| i| j| jd | |j dd}|t jjjks!|t jjjkr,|js%|7dd| id|i| j|d||_8|S)a]Initialize the input tensor with Xavier initialization. Args: var(Tensor): Tensor that needs to be initialized. block(Block|None, optional): The block in which initialization ops should be added. Used in static graph only, default None. Returns: The initialization op Out)uint16float16float32float64 xavier_initNr.tmpF)nameshapedtypetype persistablegg@g@uniform_random)r9r:minmaxrT)r;inputsoutputsattrs stop_gradientgaussian_random)r9r:meanstdr)r;rArBrCcastX)in_dtype out_dtype)r;r@rArB)9 _check_block isinstancerBlockpaddlepirr ParameterMetar _compute_fansr"r#r$program random_seedEagerParamBaseis_dist _local_shaper9r:VarDescVarTypeFP16BF16r!FP32 create_varrgeneratejoinr8 DENSE_TENSORDataTypeFLOAT16BFLOAT16FLOAT32r r%mathsqrtrrrr gaussianrG distributed auto_parallelapidtensor_from_local process_mesh placements_share_underline_tensor_tor _pir_ops append_opop)r&r*r,f_inf_outrr out_var_shape origin_dtyperJout_varlimitrFplacerps r(forwardzXavierInitializer.forwardXs!!%((%)/6:3C!DEEEEE#vz<==  $;;;    ((-- e-4<=0%%dm :??2DJ3 899 >Akkmm C    y 4</4 4 4 DL05 5 5dm 5 ,1I&& )HHmSXu=>>$\)6!'GG T]2DM4JK K KM L -IGG$IG   M } )))EE JeFW$4555**E!.!FJ+-- )))CC*tyuVg=M7N7N1N'O'OOC/11 /!J t|3888L(-M)M*  !+g|<<#y788 S[[]] &48KK!13>  . .s 3 3 34 ]]Z } )))EE JeFW$4555**E!/11MFJ+-- )))CC*tyuVg=M7N7N1N'O'OOC /MJ+-- !6 8N OOO P{7L999N}$ )))EE JeFW$4555**E__)"G,!(!* %v$ $ #'%  )))CC*tyuVg=M7N7N1N'O'OOC__*"G,!(!( #" $ #'%  t|3888 4 999$-9>"CL$+M%1  CFIr))TNNrr) rrrrrrrrrrrrr)r*r+r,r-rr.)__name__ __module__ __qualname____doc__r rx __classcell__r's@r(rr"s""L# $ $DHLLLLLLLLLr)rc.eZdZdZ ddfd ZxZS) XavierNormala This class implements the Xavier weight initializer from the paper `Understanding the difficulty of training deep feedforward neural networks `_ by Xavier Glorot and Yoshua Bengio, using a normal distribution whose mean is :math:`0` and standard deviation is .. math:: gain \times \sqrt{\frac{2.0}{fan\_in + fan\_out}}. Args: fan_in (float|None, optional): fan_in for Xavier initialization, which is inferred from the Tensor. Default is None. fan_out (float|None, optional): fan_out for Xavier initialization, which is inferred from the Tensor. Default is None. gain (float, optional): Scaling Tensor. Default is 1.0. name (str|None, optional): For details, please refer to :ref:`api_guide_Name`. Generally, no setting is required. Default: None. Returns: A parameter initialized by Xavier weight, using a normal distribution. Examples: .. code-block:: python >>> import paddle >>> paddle.seed(1) >>> data = paddle.ones(shape=[3, 1, 2], dtype='float32') >>> weight_attr = paddle.framework.ParamAttr( ... name="linear_weight", ... initializer=paddle.nn.initializer.XavierNormal()) >>> bias_attr = paddle.framework.ParamAttr( ... name="linear_bias", ... initializer=paddle.nn.initializer.XavierNormal()) >>> linear = paddle.nn.Linear(2, 2, weight_attr=weight_attr, bias_attr=bias_attr) >>> print(linear.weight) Parameter containing: Tensor(shape=[2, 2], dtype=float32, place=Place(cpu), stop_gradient=False, [[-0.21607460, 0.08382989], [ 0.29147008, -0.07049121]]) >>> print(linear.bias) Parameter containing: Tensor(shape=[2], dtype=float32, place=Place(cpu), stop_gradient=False, [1.06076419, 0.87684733]) >>> res = linear(data) >>> print(res) Tensor(shape=[3, 1, 2], dtype=float32, place=Place(cpu), stop_gradient=False, [[[1.13615966, 0.89018601]], [[1.13615966, 0.89018601]], [[1.13615966, 0.89018601]]]) Nrrrrrrr8 str | NonerrcTtd||d|dS)NFrrrrrrrr r&rrrr8r's r(r zXavierNormal.__init__^s; &'      r)NNrN rrrrrrr8rrrryrzr{r|r r}r~s@r(rr's[44p $ $            r)rc.eZdZdZ ddfd ZxZS) XavierUniforma This class implements the Xavier weight initializer from the paper `Understanding the difficulty of training deep feedforward neural networks `_ by Xavier Glorot and Yoshua Bengio. This initializer is designed to keep the scale of the gradients approximately same in all the layers. In case of Uniform distribution, the range is :math:`[-x,x]`, where .. math:: x = gain \times \sqrt{\frac{6.0}{fan\_in + fan\_out}}. Args: fan_in (float|None, optional): fan_in for Xavier initialization, which is inferred from the Tensor. Default is None. fan_out (float|None, optional): fan_out for Xavier initialization, which is inferred from the Tensor. Default is None. gain (float, optional): Scaling Tensor. Default is 1.0. name (str|None, optional): For details, please refer to :ref:`api_guide_Name`. Generally, no setting is required. Default: None. Returns: A parameter initialized by Xavier weight, using a uniform distribution. Examples: .. code-block:: python >>> import paddle >>> paddle.seed(1) >>> data = paddle.ones(shape=[3, 1, 2], dtype='float32') >>> weight_attr = paddle.framework.ParamAttr( ... name="linear_weight", ... initializer=paddle.nn.initializer.XavierUniform()) >>> bias_attr = paddle.framework.ParamAttr( ... name="linear_bias", ... initializer=paddle.nn.initializer.XavierUniform()) >>> linear = paddle.nn.Linear(2, 2, weight_attr=weight_attr, bias_attr=bias_attr) >>> print(linear.weight) Parameter containing: Tensor(shape=[2, 2], dtype=float32, place=Place(cpu), stop_gradient=False, [[-1.18095720, 0.64892638], [ 0.43125069, -1.11156428]]) >>> print(linear.bias) Parameter containing: Tensor(shape=[2], dtype=float32, place=Place(cpu), stop_gradient=False, [-0.27524316, 1.13808715]) >>> res = linear(data) >>> print(res) Tensor(shape=[3, 1, 2], dtype=float32, place=Place(cpu), stop_gradient=False, [[[-1.02494967, 0.67544925]], [[-1.02494967, 0.67544925]], [[-1.02494967, 0.67544925]]]) Nrrrrrrr8rrrcTtd||d|dS)NTrrrrs r(r zXavierUniform.__init__s; qt      r)rrrr~s@r(rrjs[66t $ $            r)r) __future__rrdrNrbaserrrbase.data_feederr base.frameworkr r r initializerr__all__rrrr)r(rs^#""""" 0000000000888888 %$$$$$ BBBBB BBBJ@ @ @ @ @ $@ @ @ FB B B B B %B B B B B r)