#jV) UdZddlZddlZddlmZddlmZmZddlZddlm Z ddl m Z m Z ddl mZmZmZmZdd lmZdd lmZer dd lmZdd lmZejeZiZeeee eee e j!fffe"d <GddZ#GddZ$Gdde#Z%dddddede#deee e j!ff dZ&dddddede#ddf dZ'de%iZ(eee#fe"d<deee)eeefzfde*efd Z+ d"dddeee)eeefzfdzddfd!Z,dS)#zrFusion registration helpers. See `docs/source/en/fusion_mapping.md` for the design overview and extension guide. N)Mapping) TYPE_CHECKINGAny)nn)!get_checkpoint_conversion_mapping®ister_checkpoint_conversion_mapping)Conv3dToLinearWeightConverterWeightRenamingWeightTransform)register_patch_mapping)logging)PretrainedConfig)PreTrainedModel_FUSION_DISCOVERY_CACHEceZdZUdZdZeedfed<dedefdZde j de fd Z d e e j de e j fd Zd d deefdZdS)ModuleFusionSpeczBase recipe for a fusion family. A fusion spec decides which modules are eligible for a fusion, how to build the runtime replacement class, and which weight transforms are needed to map checkpoints between the original and fused layouts. .target_modules_patterns model_namereturnc6dt|jd|S)zDReturn the log message emitted when no compatible modules are found.zNo compatible z classes found to fuse for )type__name__)selfrs e/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/transformers/fusion_mapping.py get_empty_logzModuleFusionSpec.get_empty_log6s"\T 3\\PZ\\\modulect)z>Return whether `module` is compatible with this fusion family.NotImplementedError)rr s r is_fusablezModuleFusionSpec.is_fusable:!!r original_clsct)zBBuild the runtime replacement class for a compatible module class.r")rr&s rmake_fused_classz!ModuleFusionSpec.make_fused_class>r%rconfigrct)zMBuild the weight transforms needed to load and save the fused runtime layout.r")rr)s rmake_transformsz ModuleFusionSpec.make_transformsBr%rN)r __module__ __qualname____doc__rtuplestr__annotations__rrModuleboolr$rr(listr r+rrrrr,s02U38_111]]]]]]""t"""""T")_"bi"""""&8"T/=R""""""rrcBeZdZfdZdejdejfdZxZS)_FusedPatchEmbeddingMixincLtj|i||jjt j|jjz|_tj |j|jj |jj du|jj j |jj j|_|`dS)N)biasdevicedtype)super__init__proj in_channelsmathprod kernel_size patch_volumerLinear out_channelsr8weightr9r: linear_proj)rargskwargs __class__s rr<z"_FusedPatchEmbeddingMixin.__init__Hs$)&))) I1DIdi>S4T4TT9   I "t+9#*)"(     IIIr hidden_statesrc|jjj}|d|j}|||}|d|jS)N)r:)rFrEr:viewrBto embed_dim)rrJ target_dtypes rforwardz!_FusedPatchEmbeddingMixin.forwardWsd'.4 %**2t/@AA (()9)9 )9)M)MNN !!"dn555r)rr,r-r<torchTensorrQ __classcell__)rIs@rr6r6Gs^     6U\6el66666666rr6ceZdZdZdZdejdefdZde ejde ejfdZ dd de e fd Z d S) PatchEmbeddingsFusionSpeczJFuse compatible Conv3d patch embeddings into flattened Linear projections.)z(^|\.)patch_embed$r rctt|ddx}tjsdS|j|jko |jdko|jdko |jdkS)Nr=F)rrr)rrrr) isinstancegetattrrConv3dstriderApaddingdilationgroups)rr r=s rr$z$PatchEmbeddingsFusionSpec.is_fusablecso'&&$"?"??$KK 5 K4+ + ! ) ! * ! q  rr&cdtd|jt|fi}d|j|_|S)NFused)rrr6r-)rr& fused_clss rr(z*PatchEmbeddingsFusionSpec.make_fused_classos@8!688;TVb:ceghh !D)B!D!D rr)rct|d|}|j}t|tr||f}|jgt |R}|j}tddt||gtddgS) N vision_configzpatch_embed\.proj\.weight$z!patch_embed\.linear_proj\.weight$)r>rA)source_patternstarget_patterns operationszpatch_embed\.proj\.bias$zpatch_embed\.linear_proj\.bias$)rdre) rY patch_sizerXinttemporal_patch_sizer/r>r r r )rr)rcrgrAr>s rr+z)PatchEmbeddingsFusionSpec.make_transformsts@@ "- j# & & 2$j1J$8M5;L;LMM #/   = D"$/$/     ; B     rN)rr,r-r.rrr2r3r$rr(r4r r+rrrrVrV^sTT6    t     T")_bi  &8 T/=R      rrVclsztype[PreTrainedModel]r)r fusion_namespecrcrt|i}||vr||Stjd5||}dddn #1swxYwYt }i}|jr,t jd|jnd}| D]y\} } t| } | |vr|| | 1| | sG| | || || j<z|||<|S)arDiscover compatible module classes for one fusion family on a meta-initialized model. This function: - instantiates `cls(config)` on the meta device - scans `named_modules()` for candidate modules - optionally pre-filters them with `target_modules_patterns` - uses `is_fusable(...)` as the final structural check - builds the class-level patch mapping used by monkey patching Results are cached per `(fusion_name, cls)` to avoid repeated meta-initialization. This matches the current class-level fusion behavior, where one compatible module class maps to one fused replacement class. metaN|)r setdefaultrRr9setrrecompilejoin named_modulesrsearchr$addr(r) rjr)rkrlcachemodel seen_classes patch_mappingtarget_module_pattern module_namer module_clss r_discover_fusable_modulesrs( $ . .{B ? ?E e||Sz f  F 55LM>B>Zd 388D899:::`d %2244 O O V&\\  % %  ,1F1M1Mk1Z1Z1b v&&  $$$-1-B-B:-N-N j)**E#J s AAAc pt||||}|s4t||jdSt |dt |drt |jdstd|jd|jj }| |}t|}|Zd |D}|D]F} t| j } || } | td |d |d | d G||z}t||ddS)aRegister one fusion family for `cls`. This function updates the two global registries used by fused loading: - the monkey-patching registry, so compatible module classes are replaced before initialization - the checkpoint conversion mapping, so fused runtime modules still load from the original checkpoint layout Notes: - conflicting checkpoint transforms fail fast )rkrlNT) overwrite config_class model_typezModel z" has no config class or model typec8i|]}t|j|Sr)r/rd).0existings r z+_register_module_fusion..s&%t%t%tT\eH,D&E&Ex%t%t%trzFusion z for model type zC conflicts with an existing conversion mapping for source patterns .)rloggerinforrrhasattrr ValueErrorrr+rr/rdgetr ) rjr)rkrlfusable_classesr convertersexisting_convertersexisting_converter_sources converterrdexisting_converters r_register_module_fusionrs0V[_```O  D&&s|44555?d;;;; 3 ' 'Tws7G/V/VTR#,RRRSSS!,J%%f--J;JGG&&u%t`s%t%t%t"#  I#I$=>>O!;!?!?!P!P !- >k>>:>>+:>>>.):5 *:zTRRRRRRrpatch_embeddings_FUSION_REGISTRY fusion_configcg}|D]f\}}|tvrtd||dur%|dur(t|tstd|d||g|S)zQValidate `fusion_config` and return enabled fusion names in user-specified order.zUnknown fusion type: FTzInvalid fusion config for z4: expected `True`, `False`, or a mapping of options.)itemsrrrXrappend)renabled_fusionsrkfusion_optionss r_iter_enabled_fusionsrsO'4':':'<'< , ,# ^ . . .B[BBCC C U " "   % %j.Q.Q %n[nnn  {++++ rcl|sdSt|D]}t|||t| dS)aRegister requested runtime fusions for `cls`. This function: - validates `fusion_config` against `_FUSION_REGISTRY` - resolves the enabled fusion families in user order - registers monkey patches and checkpoint transforms before model instantiation N)rrr)rjr)rrks rregister_fusion_patchesrsS ,];;YY V[:J;:WXXXXYYr)N)-r.r?rrcollections.abcrtypingrrrRrconversion_mappingrr core_model_loadingr r r r monkey_patchingrutilsrconfiguration_utilsrmodeling_utilsr get_loggerrrrdictr0rr2r1rr6rVrrrr3r4rrrrrrs5  ######%%%%%%%% iiiiiiii````````````3333330555555//////  H % %MOc4d3RY3G.H(H#IIJOOO""""""""666666666.- - - - - 0- - - `- - --  -  #tBI  ----`*S *S*<*SKN*SVf*S *S*S*S*SZ2DE^E^E`E`0a$s,,-aaadWS#X=N6N1N)OTXY\T]$jnYY Y9@dWUXZ]U]M^F^A^9_bf9fY YYYYYYr