o ~ri3@sddlZddlZddlmZddlZddlmZmZddlmZddl m Z ddl m Z e eZedGd d d ejZeZed Gd d d ejZedGdddejZedGdddejZedGdddejZedGdddejZGdddejZGdddejZGdddejZGd d!d!ejZGd"d#d#ejZGd$d%d%ejZGd&d'd'eZGd(d)d)ejZid*ed+ed,d-d.fd/ed0ed1ed2d3ifd4ed5ed6d3ifd7ed8ej d9ed:ej!d;edej"d?ed@ej#ej$eej%ej&ej'edAZ(ee(Z)dBdCZ*e*d1Z+e*d0Z,e*d*Z-e*d/Z.e*d=Z/e*dDZ0e*d<Z1e*d;Z2dS)EN) OrderedDict)Tensornn)use_kernel_forward_from_hub)logging)is_torchdynamo_compilingGeluTanhcLeZdZdZd deffdd Zdedefdd Zdedefd d ZZ S) GELUTanha& A fast C implementation of the tanh approximation of the GeLU activation function. See https://huggingface.co/papers/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. Fuse_gelu_tanh_pythoncs2t|r |j|_dStjtjjdd|_dS)Ntanh) approximate) super__init___gelu_tanh_pythonact functoolspartialr functionalgelu)selfr  __class__f/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/transformers/activations.pyr(s  zGELUTanh.__init__inputreturnc Cs6|ddttdtj|dt|dSN??@Hm?g@torchr mathsqrtpipowrrrrrr/6zGELUTanh._gelu_tanh_pythoncC ||SNrr)rrrforward2 zGELUTanh.forwardF) __name__ __module__ __qualname____doc__boolrrrr. __classcell__rrrrr s r NewGELUc@"eZdZdZdedefddZdS)NewGELUActivationz Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 rrc Cs6d|dttdtj|dt|dSrr#r)rrrr.Ar*zNewGELUActivation.forwardNr1r2r3r4rr.rrrrr9:sr9GeLUcr ) GELUActivationa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 Fuse_gelu_pythoncs(t|r |j|_dStjj|_dSr,)rr _gelu_pythonrrrr)rr=rrrrNs  zGELUActivation.__init__rrcCs |ddt|tdS)Nrr r!)r$erfr%r&r)rrrr>Us zGELUActivation._gelu_pythoncCr+r,r-r)rrrr.Xr/zGELUActivation.forwardr0) r1r2r3r4r5rrr>r.r6rrrrr<Es r<SiLUc@r8)SiLUActivationa See Gaussian Error Linear Units (Hendrycks et al., https://arxiv.org/abs/1606.08415) where the SiLU (Sigmoid Linear Unit) was originally introduced and coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (Elfwing et al., https://arxiv.org/abs/1702.03118) and Swish: a Self-Gated Activation Function (Ramachandran et al., https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with later. rrcCs tj|Sr,)rrsilur)rrrr.fs zSiLUActivation.forwardNr:rrrrrA\srAFastGELUc@r8)FastGELUActivationz} Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs rrcCs*d|dt|ddd||S)Nrr g3E?r")r$r r)rrrr.ps*zFastGELUActivation.forwardNr:rrrrrDjrD QuickGELUc@r8)QuickGELUActivationzr Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs rrcCs|td|S)NgZd;?)r$sigmoidr)rrrr.zszQuickGELUActivation.forwardNr:rrrrrGtrErGcs<eZdZdZdedeffdd Zdedefdd ZZS) ClippedGELUActivationa Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://huggingface.co/papers/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://huggingface.co/papers/1606.08415 minmaxcs8||krtd|d|dt||_||_dS)Nzmin should be < max (got min: z, max: )) ValueErrorrrrJrK)rrJrKrrrrs   zClippedGELUActivation.__init__xrcCstt||j|jSr,)r$cliprrJrK)rrNrrrr.zClippedGELUActivation.forward) r1r2r3r4floatrrr.r6rrrrrI~s rIcs2eZdZdZfddZdedefddZZS)AccurateGELUActivationz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cs ttdtj|_dS)N)rrr%r&r'precomputed_constantrrrrrs zAccurateGELUActivation.__init__rrc Cs,d|dt|j|dt|dS)Nrrr")r$r rTr(r)rrrr.s,zAccurateGELUActivation.forward)r1r2r3r4rrr.r6rrrrrRs rRcsDeZdZdZfddZdedefddZdedefdd ZZS) MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://huggingface.co/papers/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish csttjj|_dSr,)rrrrmishrrUrrrrs zMishActivation.__init__rrcCs|ttj|Sr,)r$r rrsoftplusr)rrr _mish_pythonrPzMishActivation._mish_pythoncCr+r,r-r)rrrr.r/zMishActivation.forward) r1r2r3r4rrrZr.r6rrrrrWs  rWc@r8)LinearActivationz[ Applies the linear activation function, i.e. forwarding input directly to output. rrcCs|Sr,rr)rrrr.szLinearActivation.forwardNr:rrrrr[sr[c@seZdZdZdddZdS)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://huggingface.co/papers/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability 绹۞? ^/ ?cCs*|||td}ddt|S)Nr!rr )divr%r&r$r?)rrmusigmarrrr.szLaplaceActivation.forwardN)r]r^r1r2r3r4r.rrrrr\sr\c@seZdZdZddZdS)ReLUSquaredActivationz^ Applies the relu^2 activation introduced in https://huggingface.co/papers/2109.08668 cCstj|}t|}|Sr,)rrrelur$square)rr relu_appliedsquaredrrrr.s  zReLUSquaredActivation.forwardNrbrrrrrcs rccseZdZfddZZS)ClassInstantiercs4t|}t|tr |n|if\}}|di|S)Nr)r __getitem__ isinstancetuple)rkeycontentclskwargsrrrris zClassInstantier.__getitem__)r1r2r3rir6rrrrrhsrhcsfeZdZdZddddejdffdd Zded efd d Zded efd d Z ded efddZ Z S)XIELUActivationz Applies the xIELU activation function introduced in https://arxiv.org/abs/2411.13010 If the user has installed the nickjbrowning/XIELU wheel, we import xIELU CUDA Otherwise, we emit a single warning and use xIELU Python g?rgưFc sptttttj||dd|_ ttttj|||dd|_ | dtj||d| dtj||d||_ t ||_t ||_d|_zFddl}tjj|_d}zddlm} | |j|_|d7}Wnty} z|d| d 7}|j|_WYd} ~ nd} ~ wwt|WdSty} ztd | d WYd} ~ dSd} ~ ww) N)dtyperbetaepszUsing experimental xIELU CUDA.)allow_in_graphz& Enabled torch._dynamo for xIELU CUDA.z+ Could not enable torch._dynamo for xIELU (z*) - this may result in slower performance.z CUDA-fused xIELU not available (u) – falling back to a Python version. For CUDA xIELU (experimental), `pip install git+https://github.com/nickjbrowning/XIELU`)rrr Parameterr$logexpm1tensor unsqueezealpha_palpha_nregister_bufferwith_vector_loadsrQ _beta_scalar _eps_scalar_xielu_cuda_obj xielu.opsclassesxieluXIELUtorch.compilerrt _xielu_cuda_xielu_cuda_fn Exceptionlogger warning_once) r alpha_p_init alpha_n_initrrrsrqr}rmsgrterrrrrrs> ("      zXIELUActivation.__init__rNrc Cshtj|j}|jtj|j}t|dk||||j|tt ||j |||j|S)Nr) rrrYrzrrr{r$whererwrJrs)rrNrzr{rrr _xielu_pythons$zXIELUActivation._xielu_pythoncCs|j}|dkr|d}|dks |dkr$|dd|d}||jkr1td||j|j||j |j |j |j |j |j|j}||S)zDFirewall function to prevent torch.compile from seeing .item() callsrVrrz_Warning: xIELU input tensor expects 3 dimensions but got (shape: %s). Reshaping to (shape: %s).)shapedimryviewsizerrrr.rztorqr{r~rr})rrNoriginal_shaperesultrrrrs*       zXIELUActivation._xielu_cudarcCs4|jdur|jrts||Std||S)Nz:torch._dynamo is compiling, using Python version of xIELU.)ris_cudarrrrrr)rrrr.4s    zXIELUActivation.forward) r1r2r3r4r$bfloat16rrrrr.r6rrrrrps * rprgelu_10i )rJrK gelu_fastgelu_new gelu_pythonr=Tgelu_pytorch_tanhgelu_python_tanhr gelu_accurate hardswishlaplace leaky_relulinearrX quick_gelurdrelu2relu6)rHrBswishr prelurcCs,|tvrt|Std|dtt)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_stringrrrget_activationYsrrB)3rr% collectionsrr$rrintegrations.hub_kernelsrutilsrutils.import_utilsr get_loggerr1rModuler PytorchGELUTanhr9r<rArDrGrIrRrWr[r\rcrhrp Hardswish LeakyReLUReLUReLU6Sigmoidr@TanhPReLUACT2CLSrrrrrrrrBrX linear_actrrrrs            ]