j?@ ddlZddlmZddlmZddlmZddlm Z dZ eej ddZ eej dd Z eej dd Zeej dd Zeej ddad Zeej ddbdZeej ddcdZeej ddbdZeej ddZeej ddZeej ddddefdZeej ddcdZeej ddZeej ddZeej ddejfdZeej ddZeej ddZdbdejd edejfd!Z eej ddedejd#efd$Z!eej dd%Z"eej ddejd&ejdejfd'Z#eej ddejdejfd(Z$eej dd)Z%eej ddfd+Z&eej dddd,Z'eej ddbd-Z(d.Z)Gd/d0e Z*e e Gd1d2e Z+e e$Gd3d4e Z,e e Gd5d6e Z-e eGd7d8e Z.e eGd9d:e Z/Gd;de Z1e eGd?d@e Z2e eGdAdBe Z3e eGdCdDe Z4GdEdFe Z5GdGdHe Z6e eGdIdJe Z7e eGdKdLe Z8e eGdMdNe Z9GdOdPe Z:GdQdRe Z;e e)GdSdTe Z<e e%GdUdVe Z=GdWdXe Z>e e"GdYdZe Z?e e&Gd[d\e Z@e e'Gd]d^e ZAe e(Gd_d`e ZBdS)gN)partial)Any)Modulecfd}|S)Ncfd|_|S)Nc|SN)_xfs c/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/mlx/nn/layers/activations.pyz<_make_activation_module..decorator.. saadd)__call__)klassr s r decoratorz*_make_activation_module..decorator s**** rr )r rs` r_make_activation_moduler s$ rT) shapelessc*tj|S)zpApplies the sigmoid function. .. math:: \text{Sigmoid}(x) = \sigma(x) = \frac{1}{1 + \exp(-x)} mxsigmoidr s rrrs :a==rc,tj|dS)zIApplies the Rectified Linear Unit. Simply ``mx.maximum(x, 0)``. rrmaximumrs rrelurs :a  rcPtjtj|dS)u\Applies the ReLU² activation function. Applies :math:`\max(0, x)^2` element wise. r)rsquarerrs rrelu2r!&s 9RZ1%% & &&rcRtjtj|ddS)z`Applies the Rectified Linear Unit 6. Applies :math:`\min(\max(x, 0), 6)` element wise. rg@rminimumrrs rrelu6r%/s" :bjA&& , ,,r{Gz?c2tj||z|S)z`Applies the Leaky Rectified Linear Unit. Simply ``mx.maximum(negative_slope * x, x)``. r)r negative_slopes r leaky_relur)8s :nq(! , ,,rc6|tj||dz S)zaApplies the Log Softmax function. Applies :math:`x + \log \sum_i e^{x_i}` element wise. T)axiskeepdims)r logsumexpr r,s r log_softmaxr0As r|AD4888 88r?cftj|dk||tj|dz zS)zfApplies the Exponential Linear Unit. Simply ``mx.where(x > 0, x, alpha * (mx.exp(x) - 1))``. r)rwhereexpr alphas relur8Js- 8AE1ervayy1}5 6 66rc.tj||S)zdApplies the Softmax function. Applies :math:`\frac{e^{x_i}}{\sum_j e^{x_j}}` element wise. r,rsoftmaxr/s rr<r<Ss :ad # # ##rc,tj|dS)zXApplies the Softplus function. Applies :math:`\log(1 + \exp(x))` element wise. r)r logaddexprs rsoftplusr?\s <1  rcVtj|dtj|zS)zXApplies the Softsign function. Applies :math:`\frac{x}{1 + |x|}` element wise. r3)rdivideabsrs rsoftsignrCes" 9QBF1II & &&r?lambdctjtj||k|tj||zz dS)zApplies the Softshrink activation function. .. math:: \text{softshrink}(x) = \begin{cases} x - \lambda & \text{if } x > \lambda \\ x + \lambda & \text{if } x < -\lambda \\ 0 & \text{otherwise} \end{cases} r)rr4rBsignr rEs r softshrinkrIns7 8BF1II%q271::+='=q A AArctj|d|tjtj|d|z dz zzS)zApplies the Continuously Differentiable Exponential Linear Unit. Applies :math:`\max(0, x) + \min(0, \alpha * (\exp(x / \alpha) - 1))` element wise. r3)rrr5r$r6s rcelurL|s@ :a   1c0B0BU0J)K)Ka)O P PPrc0|tj|zS)zApplies the Sigmoid Linear Unit. Also known as Swish. Applies :math:`x \sigma(x)` element wise, where :math:`\sigma(\cdot)` is the logistic sigmoid. rrs rsilurNs rz!}} rc$t|  S)zmApplies the Log Sigmoid function. Applies :math:`\log(\sigma(x)) = -\log(1 + e^{-x})` element wise. )r?rs r log_sigmoidrPs aRLL=rreturncf|dtj|tjdz zzdz S)zApplies the Gaussian Error Linear Units function. .. math:: \textrm{GELU}(x) = x * \Phi(x) where :math:`\Phi(x)` is the Gaussian CDF. See also :func:`gelu_approx` and :func:`gelu_fast_approx` for faster approximations. r3)rerfmathsqrtrs rgelurWs0 BF1ty||+,,, - 11rc d|zdtjtjdtjz |d|dzzzzzzS)atAn approximation to Gaussian Error Linear Unit. See :func:`gelu` for the exact computation. This function approximates ``gelu`` with a maximum absolute error :math:`< 0.0005` in the range :math:`[-6, 6]` using the following .. math:: x = 0.5 * x * \left(1 + \text{Tanh}\left((\sqrt{2 / \pi} * \left(x + 0.044715 * x^3\right)\right)\right) rDr3rSgHm?)rtanhrUrVpirs r gelu_approxr\sD 7a"'$)AK"8"8A1a4 0 \\ \lambda \alpha (\exp(x) - 1) & \text{if } x \leq 0 \end{cases} where :math:`\lambda = 1.0507` and :math:`\alpha = 1.67326`. See also :func:`elu`. gGG?g䃞ͪ?)r8rs rselurks q'??V ##rr7c^tjd||tjd|zzS)zApplies the element-wise parametric ReLU. .. math:: \text{PReLU}(x) = \max(0,x) + a * \min(0,x) where :math:`a` is an array. rrrr$r6s rprelurns, :a  ebjA&6&66 66rcJ|tjt|zS)zApplies the Mish function, element-wise. Mish: A Self Regularized Non-Monotonic Neural Activation Function. Reference: https://arxiv.org/abs/1908.08681 .. math:: \text{Mish}(x) = x * \text{Tanh}(\text{Softplus}(x)) )rrZr?rs rmishrps rwx{{## ##rchtj|dzd}|tj|dzdz S)zApplies the hardswish function, element-wise. .. math:: \text{Hardswish}(x) = x * \min(\max(x + 3, 0), 6) / 6 rYrrm)r max_x_3s r hardswishrts5jQ""G rz'1%% % ))rcRtjtj|||S)zzApplies the HardTanh function. Applies :math:`\max(\min(x, \mathrm{max\_val}), \mathrm{min\_val})` element-wise. r#)r min_valmax_vals r hard_tanhry*s" :bjG,,g 6 66rcZtjtj||k|dS)zApplies the HardShrink activation function. .. math:: \text{hardshrink}(x) = \begin{cases} x & \text{if } x > \lambda \\ x & \text{if } x < -\lambda \\ 0 & \text{otherwise} \end{cases} r)rr4rBrHs r hard_shrinkr{3s% 8BF1II%q! , ,,rc0tj| |S)zfApplies the Softmin function. Applies :math:`\frac{e^{-x_i}}{\sum_j e^{-x_j}}` element-wise. r:r;r/s rsoftminr}As :qbt $ $ $$rc*tj|S)zIApplies the hyperbolic tangent function. Simply ``mx.tanh(x)``. )rrZrs rrZrZJs 71::rc6eZdZdZddeffd ZdefdZxZS)GLUr`r*r,cVt||_dSr )super__init__r,)selfr, __class__s rrz GLU.__init___s$  rrQc.t||jS)Nr/)rer,rr s rrz GLU.__call__csQTY''''rr*) __name__ __module__ __qualname____doc__intrrr __classcell__rs@rrrRsn  S(S((((((((rrceZdZdZdS)Sigmoidz~Applies the sigmoid function, element-wise. .. math:: \text{Sigmoid}(x) = \sigma(x) = \frac{1}{1 + \exp(-x)} Nrrrrr rrrrgrrceZdZdZdS)MishzApplies the Mish function, element-wise. Reference: https://arxiv.org/abs/1908.08681 .. math:: \text{Mish}(x) = x * \text{Tanh}(\text{Softplus}(x)) Nrr rrrrpsrrceZdZdZdS)ReLUzApplies the Rectified Linear Unit. Simply ``mx.maximum(x, 0)``. See :func:`relu` for the functional equivalent. Nrr rrrr|rrrceZdZdZdS)ReLU2ubApplies the ReLU² activation function. See :func:`relu2` for the functional equivalent. Nrr rrrrrrceZdZdZdS)ReLU6z_Applies the Rectified Linear Unit 6. See :func:`relu6` for the functional equivalent. Nrr rrrrrrrc*eZdZdZdfd ZdZxZS) LeakyReLUzApplies the Leaky Rectified Linear Unit. Simply ``mx.maximum(negative_slope * x, x)``. Args: negative_slope: Controls the angle of the negative slope. Default: ``1e-2`` r&cVt||_dSr )rr_negative_slope)rr(rs rrzLeakyReLU.__init__s' -rc,t||jSr )r)rrs rrzLeakyReLU.__call__s!T1222rr&rrrrrrrrs@rrrsV......3333333rrc*eZdZdZdfd ZdZxZS)ELUzApplies the Exponential Linear Unit. Simply ``mx.where(x > 0, x, alpha * (mx.exp(x) - 1))``. See :func:`elu` for the functional equivalent. Args: alpha: the :math:`\alpha` value for the ELU formulation. Default: ``1.0`` r1cVt||_dSr rr_alpharr7rs rrz ELU.__init__$  rc,t||jSr )r8rrs rrz ELU.__call__s1dk"""rr1rrs@rrrsV#######rrceZdZdZdS)SoftmaxzZApplies the Softmax function. See :func:`softmax` for the functional equivalent. Nrr rrrrrrrceZdZdZdS)Softplusz\Applies the Softplus function. See :func:`softplus` for the functional equivalent. Nrr rrrrrrrceZdZdZdS)Softsignz\Applies the Softsign function. See :func:`softsign` for the functional equivalent. Nrr rrrrrrrc*eZdZdZdfd ZdZxZS) SoftshrinkzApplies the Softshrink function. See :func:`softshrink` for the functional equivalent. Args: lambd: the :math:`\lambda` value for Softshrink. Default: ``0.5`` rDcVt||_dSr )rrrE)rrErs rrzSoftshrink.__init__s$  rc,t||jSr )rIrErs rrzSoftshrink.__call__s!TZ(((rrDrrs@rrrsV)))))))rrc*eZdZdZdfd ZdZxZS)CELUa<Applies the Continuously Differentiable Exponential Linear Unit. Applies :math:`\max(0, x) + \min(0, \alpha * (\exp(x / \alpha) - 1))` element wise. See :func:`celu` for the functional equivalent. Args: alpha: the :math:`\alpha` value for the CELU formulation. Default: ``1.0`` r1cVt||_dSr rrs rrz CELU.__init__rrc,t||jSr )rLrrs rrz CELU.__call__sAt{###rrrrs@rrrsV$$$$$$$rrceZdZdZdS)SiLUzoApplies the Sigmoid Linear Unit. Also known as Swish. See :func:`silu` for the functional equivalent. Nrr rrrrrrrceZdZdZdS) LogSoftmaxzbApplies the Log Softmax function. See :func:`log_softmax` for the functional equivalent. Nrr rrrrrrrceZdZdZdS) LogSigmoidzbApplies the Log Sigmoid function. See :func:`log_sigmoid` for the functional equivalent. Nrr rrrrrrrc:eZdZdZdfd ZdejfdZxZS)PReLUa[Applies the element-wise parametric ReLU. Applies :math:`\max(0, x) + a * \min(0, x)` element wise, where :math:`a` is an array. See :func:`prelu` for the functional equivalent. Args: num_parameters: number of :math:`a` to learn. Default: ``1`` init: the initial value of :math:`a`. Default: ``0.25`` r3?c~ttj|g||_dSr )rrrfullweight)rnum_parametersinitrs rrzPReLU.__init__s3 g~.55 rr c,t||jSr )rnrrs rrzPReLU.__call__sQ $$$r)r3r) rrrrrrarrayrrrs@rrr sd  666666%"(%%%%%%%%rrc*eZdZdZdfd ZdZxZS)GELUa0Applies the Gaussian Error Linear Units. .. math:: \textrm{GELU}(x) = x * \Phi(x) where :math:`\Phi(x)` is the Gaussian CDF. However, if ``approx`` is set to 'precise' or 'fast' it applies .. math:: \textrm{GELUApprox}(x) &= 0.5 * x * \left(1 + \text{Tanh}\left((\sqrt{2 / \pi} * \left(x + 0.044715 * x^3\right)\right)\right) \\ \textrm{GELUFast}(x) &= x * \sigma\left(1.702 * x\right) respectively. .. note:: For compatibility with the PyTorch API, 'tanh' can be used as an alias for 'precise'. See :func:`gelu`, :func:`gelu_approx` and :func:`gelu_fast_approx` for the functional equivalents and information regarding error bounds. Args: approx ('none' | 'precise' | 'fast'): Which approximation to gelu to use if any. nonect||_gd}||vrtd|d|ddS)N)rpreciserZfastzThe approximation should be in z but 'z ' was given)rr_approx ValueError)rapproxallowedrs rrz GELU.__init__<sf  555  T'TTTTT  ! rc|jdkrt|S|jdvrt|St|S)Nr)rrZ)rrWr\r^rs rrz GELU.__call__EsB <6 ! !77N \0 0 0q>> !"""r)rrrs@rrr sV6#######rrceZdZdZdS)TanhzbApplies the hyperbolic tangent function. See :func:`tanh` for the functional equivalent. Nrr rrrrMrrrceZdZdZdS) HardswishzlApplies the hardswish function, element-wise. See :func:`hardswish` for the functional equivalent. Nrr rrrrUrrrc@eZdZdZddeffd ZdejfdZxZ S)SteprhrKrfcVt||_dSr )rrrf)rrfrs rrz Step.__init__ms$ "rr c,t||jSr )rirfrs rrz Step.__call__qsAt~&&&rrK) rrrrfloatrrrrrrs@rrr]sp  ##%######'"(''''''''rrceZdZdZdS)SELUzeApplies the Scaled Exponential Linear Unit. See :func:`selu` for the functional equivalent. Nrr rrrrurrrceZdZdZdS)HardTanhz]Applies the HardTanh function. See :func:`hard_tanh` for the functional equivalent. Nrr rrrr}rrrceZdZdZdS) HardShrinkzApplies the HardShrink function. See :func:`hard_shrink` for the functional equivalent. Args: lambd: the :math:`\lambda` value for Hardshrink. Default: ``0.5`` Nrr rrrrsrrceZdZdZdS)SoftminzZApplies the Softmin function. See :func:`softmin` for the functional equivalent. Nrr rrrrrrrrrrrr)rur1)CrU functoolsrtypingrmlx.corecorermlx.nn.layers.baserrcompilerrr!r%r)r0r8r<r?rCrrIrLrNrPrrWr\r^rrerirkrnrprtryr{r}rZrrrrrrrrrrrrrrrrrrrrrrrrrr rrrso  %%%%%% t$$$%$ t$$$%$ t$$$''%$' t$$$--%$- t$$$---%$- t$$$999%$9 t$$$777%$7 t$$$$$$%$$ t$$$%$ t$$$''%$' t$$$ B B B B B%$ B t$$$QQQ%$Q t$$$%$ t$$$%$ t$$$ 2rx 2 2 2%$ 2 t$$$ S S%$ S  t$$$%%%$%*  28 3       t$$$))BH))))%$)& t$$$ $ $%$ $  t$$$7RX7bh728777%$7 t$$$ $BH $ $ $ $%$ $ t$$$**%$* t$$$777%$7 t$$$ - - -%$ - t$$$%%%%$%(((((&(((*!!f"!66F F 33333333"#####&###$!!f"!""v#"""v#"))))))))"$$$$$6$$$&6%%&%%%&%%%%%%F%%%(*#*#*#*#*#6*#*#*#Z6##$#'''''6'''06##v$#%%&%!!f"!r