o ~ri]@sZddlmZddlmZddlmZddlmZddlm Z m Z m Z e r)ddl Z e eZ  d8d e jd e jd e jdBd ed e jf ddZde jjde jde jde jd e jf ddZd e jd e jde jd e jfddZd e jd e jde jd e jfddZddZddZe re jjded d!e jdee jjdeed"d e jd e jde jd efd#d$Zd e jd e jde jd e jfd%d&Z  d8d e jd e jde jd e jdBd ed e jf d'd(Z de jjde jde jde jd e jf d)d*Z!Gd+d,d,eZ"e"Z#d-e jd e jfd.d/Z$ d9e#ddd0d1d2e%e jjdBd3e"d ed4ed5ed e%e jjf d6d7Z&dS):)Callable)wraps)logging)GeneralInterface)is_torch_availableis_torch_less_or_equalis_torchdynamo_compilingNFinputweightbias is_transposedreturncCsJ|rt|d|d}n t||dd}|dur#||}|S)aBatched linear layer supporting optional bias and transposed weights. Args: input (`torch.Tensor`): Input tensor of shape (batch_size, input_dim). weight (`torch.Tensor`): Weight tensor of shape (batch_size, output_dim, input_dim) if transposed is `False`, else of shape (batch_size, input_dim, output_dim). bias (`torch.Tensor`, *optional*): Bias tensor of shape (batch_size, output_dim). Default is `None`. is_transposed (`bool`, *optional*, defaults to `False`): Whether the weight tensor is transposed. Returns: `torch.Tensor`: Output tensor of shape (batch_size, output_dim). N)torchbmm unsqueezesqueeze)r r r r outrk/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/transformers/integrations/moe.py_batched_linearEs rself hidden_states top_k_index top_k_weightscCsb|j}|d}|d}|d}tj||ddd|d}|d} |d} | |jk} | d|jd} ||} |j rR|j | } |j rO|j | nd}n|j | } |j r_|j| nd}t| | ||jd}|j rs||}n||}|j| } |j r|j| nd}t|| ||jd}|| d}|| dd||||jdd}||jS)Nrrdevicerr r gdim)rsizerarangerexpandreshape num_expertsclamphas_gate gate_up_projhas_biasgate_up_proj_biasup_proj up_proj_biasrr _apply_gateact_fn down_projdown_proj_bias masked_fill_viewsumtodtype)rrrrr num_top_k num_tokens hidden_dim token_idxsample_weights expert_ids invalid_maskselected_hidden_statesselected_weightsselected_biasesproj_out weighted_outfinal_hidden_statesrrrbatched_mm_experts_forwardgs<   "           rDoffscCsrtj|d|d|j|jd}d}t|D]\}}||kr"qtj||||||||d|}q|S)a( Fallback grouped matrix multiplication used when `torch.nn.functional.grouped_mm` and `torch._grouped_mm` are unavailable or incompatible with `torch.compile` (e.g. non-bfloat16 weights). Args: input (`torch.Tensor`): Input of shape (S, input_dim), sorted by expert id. weight (`torch.Tensor`): Expert weights of shape (num_experts, input_dim, output_dim). offs (`torch.Tensor`): Cumulative token counts per expert of shape (num_experts,). Returns: `torch.Tensor`: Output of shape (S, output_dim). rrrr6r)rzerosr"rr6 enumeratetolistmm)r r rEoutputstartiendrrr_grouped_mm_fallbacks" $rPcCs |dksJdt|j|dks Jdt|j|dks0Jdt|j|d|dksJJd|dd |d|d|dksdJd |dd |d|jtjtjfvsuJd |jtj|d|d|j |jd S)zRShape/dtype inference stub for `_grouped_mm_fallback` required by `torch.compile`.rz+input must be 2D (S, input_dim), got shape zBweight must be 3D (num_experts, input_dim, output_dim), got shape rz*offs must be 1D (num_experts,), got shape rz offs length z must match number of experts zinput_dim mismatch: input has z , weight has z$offs must be an integer tensor, got rF) r!tupleshaper"r6rint32int64emptyrr r rErrr_grouped_mm_fallback_fakes  4""rXcCs"||d|d|d|_dS)zjSaves input and weight for backward; offs is stored directly as it is a non-differentiable integer tensor.rrrN)save_for_backwardrE)ctxinputsrLrrr"_grouped_mm_fallback_setup_contextsr\c Cs|j\}}t|}t|}d}t|jD]1\}}||kr!qtj|||||j|||dtj|||j|||||d|}q||dfS)zuBackward pass for `_grouped_mm_fallback`. Computes grad_input and grad_weight per expert group; offs has no gradient.rrGN) saved_tensorsr zeros_likerIrErJrKT) rZ grad_outputr r grad_input grad_weightrMrNrOrrr_grouped_mm_fallback_backwards   && rcz!transformers::grouped_mm_fallbackr) mutates_args) setup_contextcCsftr |jtjks%|jjdkr'tdddr'|ddks%|ddkr'dSttj j dp2ttd S) a Check if torch.nn.functional.grouped_mm or torch._grouped_mm can be used based on availability and compatibility with torch.compile. Args: input (`torch.Tensor`): Input tensor of shape (S, input_dim). weight (`torch.Tensor`): Weight tensor of shape (num_experts, input_dim, output_dim). offs (`torch.Tensor`): Offsets tensor indicating the boundaries of each group in the input tensor. Returns: `bool`: True if grouped_mm can be used, False otherwise. cpuz2.10.0T) accept_devrF grouped_mm _grouped_mm) r r6rbfloat16rtyperdata_ptrhasattrnn functionalrWrrr_can_use_grouped_mms   rqcCslt|||r,ttjjdrtjjj||j||dSttdr,tj||j||dStj j j |||dS)aGrouped matrix multiplication dispatcher that uses torch.nn.functional.grouped_mm if available, else falls back to torch._grouped_mm. Args: input (`torch.Tensor`): Input tensor of shape (S, input_dim). weight (`torch.Tensor`): Weight tensor of shape (num_experts, input_dim, output_dim). offs (`torch.Tensor`): Offsets tensor indicating the boundaries of each group in the input tensor. Returns: `torch.Tensor`: Output tensor of shape (S, output_dim). rirErj) rqrnrrorprir5r6rjops transformersgrouped_mm_fallbackrWrrrrjs  rjcCs>|r t|||d}n t||dd|d}|dur||}|S)aGrouped linear layer supporting optional bias and transposed weights. Args: input (`torch.Tensor`): Input tensor of shape (S, input_dim). weight (`torch.Tensor`): Weight tensor of shape (num_experts, input_dim, output_dim) if `is_transposed`, else of shape (num_experts, output_dim, input_dim). offs (`torch.Tensor`): Offsets tensor indicating the boundaries of each group in the input tensor. bias (`torch.Tensor`, *optional*): Bias tensor of shape (num_experts, output_dim). Default is `None`. is_transposed (`bool`, *optional*, defaults to `False`): Whether the weight tensor is transposed. Returns: `torch.Tensor`: Output tensor of shape (S, output_dim). rrrN)rj transpose)r r rEr r rrrr_grouped_linear8s rxcCs|j}|d}|d}|d}tj||ddd|d}|d} |d} ||} t| } t| } tj| d|d| | <| | }| | }| | }|j dkr\| n| }tj ||j d|j dd}tj|dtjd}|jr|j}|jr|j|nd}n |j}|jr|j|nd}t|||||jd}|jr||}n||}|j}|jr|j|nd}t|||||jd}||d}|| }||||jdd }||jS) Nrrrrrf)binsminmax)r!r6rr ) rr"rr#rr$r%argsort empty_likerlfloatinthistcr&cumsumrTr(r)r*r+r,r-rxr r.r/r0r1r3r4r5r6)rrrrrr7r8r9r:r;r<r>perminv_perm expert_ids_gsample_weights_gselected_hidden_states_g histc_inputtokens_per_expertoffsetsr?r@rArBrCrrrgrouped_mm_experts_forward^sJ   "         rcs8eZdZdZeedZdededeffdd Z Z S)ExpertsInterfacez;Interface for registering custom experts forward functions.) batched_mmriexperts_implementationdefaultrcsB|dur tdn|dkr||vrtd|dt||S)zfReturn the requested `experts_implementation`. Also strictly check its validity, and raise if invalid.Na You tried to access the `ExpertsInterface` with a `config._experts_implementation` set to `None`. This is expected if you use an Expert Module as a standalone Module. If this is not the case, something went wrong with the dispatch of `config._experts_implementation`eager`zL` is not a valid experts implementation registered in the `ExpertsInterface`)logger warning_onceKeyErrorsuperget)rrr __class__rr get_interfaces zExpertsInterface.get_interface) __name__ __module__ __qualname____doc__rDr_global_mappingstrrr __classcell__rrrrrs "r gate_up_outcCs |jddd\}}|||S)a Default gating mechanism: splits the gate_up_out into gate and up parts, applies the activation function to the gate part, and multiplies it with the up part. Args: gate_up_out (`torch.Tensor`): The output tensor from the gate and up projection of shape (S, 2 * intermediate_dim). Returns: `torch.Tensor`: The gated output tensor of shape (S, intermediate_dim). rrr )chunkr/)rrgateuprrr_default_apply_gates rT)experts_interfacer r*r( experts_classrr*r(cs@dttjjdttjjffdd }|dur||S|S)aDecorator to modify experts class to support different experts implementations. Args: experts_class (`type[torch.nn.Module]`, *optional*): The experts class to modify. If not provided, returns a decorator that can be applied to the class. experts_interface (`ExpertsInterface`, *optional*, defaults to `ALL_EXPERTS_FUNCTIONS`): The experts interface to use for dispatching the forward method. is_transposed (`bool`, *optional*, defaults to `False`): Whether the expert weights are stored in transposed format. has_bias (`bool`, *optional*, defaults to `False`): Whether the expert layers include bias terms. Returns: `type[torch.nn.Module]`: The modified experts class. rrcs\|j|jtfdd}tfdd}t|ds&t|_||_||_|S)Ncs4||g|Ri|||_|_|_|_dSN)configr(r*r )rrargskwargs)r*r(r original_initrr__init__s  z=use_experts_implementation..wrapper..__init__cs&|jj}||g|Ri|Sr)rr_experts_implementation)rrrexperts_forward)roriginal_forwardrrforwardsz.wrapper..forwardr.)rrrrnrr.)rrrrr*r(r )rrrwrappers z+use_experts_implementation..wrapperN)rlrroModule)rrr r*r(rrrruse_experts_implementations,r)NFr)'collections.abcr functoolsrutilsr utils.genericrutils.import_utilsrrr r get_loggerrrTensorboolrrorrDrPrXr\rclibrary custom_op register_fakeregister_autogradrqrjrxrrALL_EXPERTS_FUNCTIONSrrlrrrrrs     - " "G"  # & U