j6 ddlZddlmZmZddlmZejfdedej deej gej ffdZ ddejfd ed edej deej gej ffd Z ddejfd ed edej deej gej ffdZ ejfdej deej gej ffdZdZejfdej deej egej ffdZejfdej deej egej ffdZejfdej deej edegej ffdZejfdej deej edegej ffdZddejfded ed edej deej gej ff dZdejfdedej deej gej ffdZdS)N)CallableLiteralvaluedtypereturncHdtjdtjffd }|S)aAn initializer that returns an array filled with ``value``. Args: value (float): The value to fill the array with. dtype (Dtype, optional): The data type of the array. Default: ``float32``. Returns: Callable[[array], array]: An initializer that returns an array with the same shape as the input, filled with ``value``. Example: >>> init_fn = nn.init.constant(0.5) >>> init_fn(mx.zeros((2, 2))) array([[0.5, 0.5], [0.5, 0.5]], dtype=float32) arc<tj|jSNr)mxfullshape)r rrs U/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/mlx/nn/init.py initializerzconstant..initializerswqwU3333r array)rrrs`` rconstantr sA,4rx4BH4444444 rg?meanstdcLdtjdtjffd }|S)aAn initializer that returns samples from a normal distribution. Args: mean (float, optional): Mean of the normal distribution. Default: ``0.0``. std (float, optional): Standard deviation of the normal distribution. Default: ``1.0``. dtype (Dtype, optional): The data type of the array. Default: ``float32``. Returns: Callable[[array], array]: An initializer that returns an array with the same shape as the input, filled with samples from a normal distribution. Example: >>> init_fn = nn.init.normal() >>> init_fn(mx.zeros((2, 2))) array([[-0.982273, -0.534422], [0.380709, 0.0645099]], dtype=float32) r rcTtj|jS)Nrscalelocr)r randomnormalr)r rrrs rrznormal..initializer>s$yagSd%PPPrr)rrrrs``` rrr%sQ2QrxQBHQQQQQQQQ rlowhighcLdtjdtjffd }|S)aAn initializer that returns samples from a uniform distribution. Args: low (float, optional): The lower bound of the uniform distribution. Default: ``0.0``. high (float, optional): The upper bound of the uniform distribution. Default: ``1.0`` dtype (Dtype, optional): The data type of the array. Default: ``float32``. Returns: Callable[[array], array]: An initializer that returns an array with the same shape as the input, filled with samples from a uniform distribution Example: >>> init_fn = nn.init.uniform(low=0, high=1) >>> init_fn(mx.zeros((2, 2))) array([[0.883935, 0.863726], [0.617261, 0.417497]], dtype=float32) r rcTtj|jSr )r runiformr)r rr!r s rrzuniform..initializer]s$y  dAG5 AAArr)r r!rrs``` rr$r$DsQ2BrxBBHBBBBBBBB rcDdtjdtjffd }|S)aAn initializer that returns an identity matrix. Args: dtype (Dtype, optional): The data type of the array. Default: ``float32``. Returns: Callable[[array], array]: An initializer that returns an identity matrix with the same shape as the input. Example: >>> init_fn = nn.init.identity() >>> init_fn(mx.zeros((2, 2))) array([[1, 0], [0, 1]], dtype=float32) arrrc|jdks|jd|jdkrtd|jdtj|jdS)Nrz6The input array must be a square matrix but got shape .)nr)ndimr ValueErrorr eye)r&rs rrzidentity..initializervsc 8q==CIaLCIaL88UUUU v ! E2222rrrrs` ridentityr0cs;&33bh333333 rc|jdkrtd|jd|jd}|jd}|jdkr#d}|jddD]}||z}||z}||z}||fS)Nr(zPGlorot / He initialization requires at least 2 dimensional input but input with z dimensions.rr))r,r-r)xfan_infan_outreceptive_fieldds r_calculate_fan_in_fan_outr8svzz 4 v 4 4 4   WR[FgajGvzz2 ! !A q OO/)O+ 7?rcTddtjdtdtjffd }|S)aWA Glorot normal initializer. This initializer samples from a normal distribution with a standard deviation computed from the number of input (``fan_in``) and output (``fan_out``) units according to: .. math:: \sigma = \gamma \sqrt{\frac{2.0}{\text{fan\_in} + \text{fan\_out}}} For more details see the original reference: `Understanding the difficulty of training deep feedforward neural networks `_ Args: dtype (Dtype, optional): The data type of the array. Default: ``float32``. Returns: Callable[[array, float], array]: An initializer that returns an array with the same shape as the input, filled with samples from the Glorot normal distribution. Example: >>> init_fn = nn.init.glorot_normal() >>> init_fn(mx.zeros((2, 2))) array([[0.191107, 1.61278], [-0.150594, -0.363207]], dtype=float32) >>> init_fn(mx.zeros((2, 2)), gain=4.0) array([[1.89613, -4.53947], [4.48095, 0.995016]], dtype=float32) rr gainrct|\}}|tjd||zz z}tj|j|S)Ng@rrr)r8mathsqrtr rrr)r r:r4r5rrs rrz"glorot_normal..initializersQ3A66TYsfw&67888yagSFFFrrr rfloatr/s` r glorot_normalrBsRFGGrxGuGrxGGGGGG rcTddtjdtdtjffd }|S)aQA Glorot uniform initializer. This initializer samples from a uniform distribution with a range computed from the number of input (``fan_in``) and output (``fan_out``) units according to: .. math:: \sigma = \gamma \sqrt{\frac{6.0}{\text{fan\_in} + \text{fan\_out}}} For more details see the original reference: `Understanding the difficulty of training deep feedforward neural networks `_ Args: dtype (Dtype, optional): The data type of the array. Default: ``float32``. Returns: Callable[[array, float], array]: An initializer that returns an array with the same shape as the input, filled with samples from the Glorot uniform distribution. Example: >>> init_fn = nn.init.glorot_uniform() >>> init_fn(mx.zeros((2, 2))) array([[0.223404, -0.890597], [-0.379159, -0.776856]], dtype=float32) >>> init_fn(mx.zeros((2, 2)), gain=4.0) array([[-1.90041, 3.02264], [-0.912766, 4.12451]], dtype=float32) rr r:rct|\}}|tjd||zz z}tj| ||jS)Ng@r )r8r=r>r rr$r)r r:r4r5limitrs rrz#glorot_uniform..initializersU3A66ty(8!9:::y  %u EEErr?r@r/s` rglorot_uniformrFsRFFFrxFuFrxFFFFFF rr4r5c r d dtjdtddtdtjffd }|S) aBuild a He normal initializer. This initializer samples from a normal distribution with a standard deviation computed from the number of input (``fan_in``) or output (``fan_out``) units according to: .. math:: \sigma = \gamma \frac{1}{\sqrt{\text{fan}}} where :math:`\text{fan}` is either the number of input units when the ``mode`` is ``"fan_in"`` or output units when the ``mode`` is ``"fan_out"``. For more details see the original reference: `Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification `_ Args: dtype (Dtype, optional): The data type of the array. Default: ``float32``. Returns: Callable[[array, str, float], array]: An initializer that returns an array with the same shape as the input, filled with samples from the He normal distribution. Example: >>> init_fn = nn.init.he_normal() >>> init_fn(mx.zeros((2, 2))) # uses fan_in array([[-1.25211, 0.458835], [-0.177208, -0.0137595]], dtype=float32) >>> init_fn(mx.zeros((2, 2)), mode="fan_out", gain=5) array([[5.6967, 4.02765], [-4.15268, -2.75787]], dtype=float32) r4rr moderGr:rct|\}}|dkr|}n|dkr|}ntd|d|tj|z }tj|j|S)Nr4r5Invalid mode: ". Valid modes are: fan_in, fan_outr<)r8r-r=r>r rrr)r rIr:r4r5fanrrs rrzhe_normal..initializers 4A66 8  CC Y  CCVdVVVWW WTYs^^#yagSFFFrr4rr rrrAr/s` r he_normalrPspR.6GG 8G)*GG  GGGGGG rc r d dtjdtddtdtjffd }|S) aA He uniform (Kaiming uniform) initializer. This initializer samples from a uniform distribution with a range computed from the number of input (``fan_in``) or output (``fan_out``) units according to: .. math:: \sigma = \gamma \sqrt{\frac{3.0}{\text{fan}}} where :math:`\text{fan}` is either the number of input units when the ``mode`` is ``"fan_in"`` or output units when the ``mode`` is ``"fan_out"``. For more details see the original reference: `Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification `_ Args: dtype (Dtype, optional): The data type of the array. Default: ``float32``. Returns: Callable[[array, str, float], array]: An initializer that returns an array with the same shape as the input, filled with samples from the He uniform distribution. Example: >>> init_fn = nn.init.he_uniform() >>> init_fn(mx.zeros((2, 2))) # uses fan_in array([[0.0300242, -0.0184009], [0.793615, 0.666329]], dtype=float32) >>> init_fn(mx.zeros((2, 2)), mode="fan_out", gain=5) array([[-1.64331, -2.16506], [1.08619, 5.79854]], dtype=float32) r4rr rIrGr:rct|\}}|dkr|}n|dkr|}ntd|d|tjd|z z}tj| ||jS)Nr4r5rKrLg@r )r8r-r=r>r rr$r)r rIr:r4r5rMrErs rrzhe_uniform..initializerNs 4A66 8  CC Y  CCVdVVVWW Wtys+++y  %u EEErrNrOr/s` r he_uniformrS%spV.6FF 8F)*FF  FFFFFF rsparsitycPdtjdtjffd }|S)aAn initializer that returns a sparse matrix. Args: sparsity (float): The fraction of elements in each column to be set to zero. mean (float, optional): Mean of the normal distribution. Default: ``0.0``. std (float, optional): Standard deviation of the normal distribution. Default: ``1.0``. dtype (Dtype, optional): The data type of the array. Default: ``float32``. Returns: Callable[[array], array]: An initializer that returns an array with the same shape as the input, filled with samples from a normal distribution. Example: >>> init_fn = nn.init.sparse(sparsity=0.5) >>> init_fn(mx.zeros((2, 2))) array([[-1.91187, -0.117483], [0, 0]], dtype=float32) r rc|jdkrtd|j\}}tt j|z}t jt j |jd}t j |j}d|t j | |d|ddd|ff<|S)Nr(z,Only tensors with 2 dimensions are supportedrr))axisrr) r,r-rintr=ceilr argsortrr$rarangereshape) r rowscols num_zerosorderrrrTrs rrzsparse..initializers 6Q;;KLL LW d (T/2233  29,,17,;;!DDD I  17#4u  M MDE")D// ! !$ * *E!!!ZiZ-,@ @Arr)rTrrrrs```` rsparserbasM< rx BH          rr:cHdtjdtjffd }|S)aAn initializer that returns an orthogonal matrix. Args: gain (float, optional): Scaling factor for the orthogonal matrix. Default: ``1.0``. dtype (Dtype, optional): Data type of the array. Default: ``float32``. Returns: Callable[[array], array]: An initializer that returns an orthogonal matrix with the same shape as the input. r rc|jdkrtd|j\}}t||}tj||f}tj|tj \}}t j |}|t j |z}|d|d|f}| z}| S)Nr(zHOrthogonal initialization requires a 2D array but got a {a.ndim}D array.rW)stream) r,r-rmaxr rrlinalgqrcpudiagsignastype) r r^r_r+rmatqrr7rr:s rrzorthogonal..initializers 6Q;;&  W d dOOyq!f--y||D|001 GAJJ  N eteUdUlO Hxxrr)r:rrs`` r orthogonalrpsArxBH4 r)r=typingrrmlx.corecorer float32rADtyperrrr$r0r8rBrFrPrSrbrprrrwse $$$$$$$$%'J  rxj"("#:C2: !02 rxj"("#@C2: !02 rxj"("#> "zBHhz287K.L:,j(( 8( rx)*((((Xj(( 8( rx)*((((Xj77 87 rx!45u=rxGH7777vj99 89 rx!45u=rxGH9999|j ,,, , , 8 , rxj"("# ,,,,`)) ) h) rxj"("#))))))r