o ~riGn@sfddlmZmZerddlZddlmZddlmZddlmZddl m Z m Z e e ZgdZed d ZGd d d eZGd ddeZGdddeZGdddeZddZddZejdddejdedejfddZejdddejdedejfddZGd d!d!ejZd"d#Zd$d%Z d&d'Z!d(d)Z"d*d+Z#d,d-Z$d1d.e%e&dBfd/d0Z'dS)2)is_torch_availableloggingN)nn)contextmanager) ConversionOps)get_module_from_nameshould_convert_module)gg?g?g?g@g@g@g@ggggggggccstreddl}t||jr|j}n t|tr||}t|dd}|dkrA|j| dV WddS1s.topkr)binsmaxiT)r)src_indxdst_indx)osr7routing GatherIndx RoutingData ScatterIndxcompute_expt_data_torchrrr distributedget_world_sizerenvirongetrsoftmaxsortgatherrmhistcrrvrwherer)logits n_expts_actrrrrr world_sizerank replace_valuen_tokens n_expts_totn_local_expertslocal_expert_startlocal_expert_end n_gates_padr expt_scal expt_indx sort_indiceshistvar topk_indx gate_indxrrr expt_data hit_expertsrrrrouting_torch_distsN       4rc Csddlm}|r|rt|drt}ntjj}|jd}| d|j j }t j ||j j|j j}t|j|||j j\}}}Wdn1sMwY|j||||d} | |d|j j } | |fS)Nr _is_hookedr')r)torch.distributedr is_availableis_initializedrrr7rrrmrouter hidden_dimr functionallinearweightrarrtop_kexperts) rrdistr batch_size router_logitsrrr routed_outrrr mlp_forwards   rc Ksddlm}|d}|d}|d} |d} |d} |d} d D]W} | |vr}| dur;|||||| | | | }| d }| d }t||d d d |t||r}t||r}tt||t||}t|| tj | |t ||t ||q&dS)Nrshard_and_distribute_moduler# empty_param casting_dtype to_contiguousr device_mesh)r)r*rQrR.r`) integrations.tensor_parallelrrr>rgrrrr rrrvdelattr)rD param_name param_value target_device dq_param_namerArr#rrrrrrF blocks_attr scales_attrrVrrr dequantizes<          r"cCst||}tj|Sr)rr rr)rrrVrrrrTs  rTc Ks|jj|jj|jj}}}ddlm} |d} |d} |d} |d} |d}|d}d |vrB|d d d d }d|vrR|d d dd }|durb| | || || | ||nt|| d ddt j j |dd|d }|d}t ||}t ||}|jjdkrB|jjdkrD|d }|dkr|||jdd }n ||d |jd}t |d|dkrtt drt jdurt jj}||}||}t|t|dd |dd |\}}Wdn1swY|dkrt ||j|jdg|_n t ||j|jg|_t|||t||d||||ddt||t||~dSdSdS)q This transforms the weights obtained using `convert_gpt_oss.py` to load them into `Mxfp4GptOssExperts`. rrr#rrrrrrrr'rQrrrRNr`Frmetar)r r acceleratorr(r+r,r.)r8r9r:r;rrrsplitr>rgr rrrrr sizermrrr%current_acceleratorrvrrr<r6Sizer_rr)rDrrrr7rAr9r:r;rr#rrrrrrFr r!rr local_expertsrEr/rrrload_and_swizzle_mxfp4#sl       $            r+c Csn|jj|jj|jj}}}|d} t|d|dkr-ttdr-tj dur-tj j }| | }| | }|dkrJ| | |jdd}n | | d|jd}t|t|d d|d d|\} } Wdn1sswY|dkrt| |j|jdg| _n t| |j|jg| _||jvr|j|=t||| t||d || ||d d dS) r#rr rr%Nr)rr'r(r+r,r.)r8r9r:r;r'rrr r%r(r rvrrmrrr<r6r)r_rr=r>) rrrDrFrr7r9r:r;r*rEr/rrrrYjs>      rYmodules_to_not_convertc Cs|jr|Sddlm}|dad}|D]H\}}t||sq|jjdkrH|jsHt d| |t |j d}Wdn1sCwY|jjd kr]|js]d d l m}|t||_q|setd |S) aD Public method that replaces the expert layers of the given model with mxfp4 quantized layers. Args: model (`torch.nn.Module`): The model to convert, can be any `torch.nn.Module` instance. quantization_config (`Mxfp4Config`, defaults to `None`): The quantization config object that contains the quantization parameters. modules_to_not_convert (`list`, *optional*, defaults to `None`): A list of modules to not convert. If a module name is in the list (e.g. `lm_head`), it will not be converted. r`) get_kernelz(kernels-community/gpt-oss-triton-kernelsF GptOssExpertsr$TN GptOssMLPr) MethodTypezYou are loading your model using mixed-precision FP4 quantization but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.)r" hub_kernelsr-r7 named_modulesr rrJr r set_submoduler4rftypesr0rrloggerwarning)r#quantization_configr,r-has_been_replaced module_namerDr0rrrreplace_with_mxfp4_linears,      r:)NN)(utilsrrr r contextlibrcore_model_loadingrquantizers.quantizers_utilsrr get_loggerrJr5rrrrPrXrZr5r<rwr}rrrrrNr4rrr"rTr+rYr3rr:rrrrsV     004 : CD!G1