#jrdZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z e r ddlZddlmZe jeZdZe rerdd lmZndd lmZGd d eZGd deZdadZdZdZdZdZdZ ddZ!dZ"ddZ#ddZ$ddZ%dZ&dS) z Integration with Deepspeed N) partialmethod)dep_version_check)is_accelerate_availableis_torch_availablelogging)nnctjddu}|r= tjd}dS#tjj$rYdSwxYwdS)N deepspeedTF) importlibutil find_specmetadataPackageNotFoundError)package_exists_s m/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/transformers/integrations/deepspeed.pyis_deepspeed_availabler$sx^--k::$FN "++K88A4!6   55  sAAA)HfDeepSpeedConfig)objectc"eZdZdZfdZxZS)raJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. ct|tdtdt|dS)N accelerater )set_hf_deepspeed_configrsuper__init__selfconfig_file_or_dict __class__s rrzHfDeepSpeedConfig.__init__JsL%%%,'''+&&& ,-----)__name__ __module__ __qualname____doc__r __classcell__r s@rrr9sB .........r!rc^eZdZdZfdZdZdZd dZeed Z d d Z d Z xZ S)HfTrainerDeepSpeedConfigz The `HfTrainerDeepSpeedConfig` object is meant to be created during `TrainingArguments` object creation and has the same lifespan as the latter. cft|d|_g|_dSN)rr_dtype mismatchesrs rrz!HfTrainerDeepSpeedConfig.__init__Xs. ,--- r!c<|jtd|jS)Nz8trainer_config_process() wasn't called yet to tell dtype)r, ValueError)rs rdtypezHfTrainerDeepSpeedConfig.dtype]s! ; WXX X{r!c@||}|dS|dkS)NFauto) get_value)r ds_key_longvals ris_autoz HfTrainerDeepSpeedConfig.is_autobs'nn[)) ;5&= r!NTc ||\}}|dS||dkr|||<dS|sdS||}|.||kr*|jd|d|d|d|dSdSdS)a A utility method that massages the config file and can optionally verify that the values match. 1. Replace "auto" values with `TrainingArguments` value. 2. If it wasn't "auto" and `must_match` is true, then check that DS config matches Trainer config values and if mismatched add the entry to `self.mismatched` - will assert during `trainer_config_finalize` for one or more mismatches. Nr2z- ds =z vs hf )find_config_nodegetr-append)rr4hf_valhf_key must_matchconfigds_keyds_vals r fill_matchz#HfTrainerDeepSpeedConfig.fill_matchis..{;; > F ::f   ' '#F6N F  FF##  &F"2"2 O " "#Y;#Y#Y#Y#Y#Y#YQW#Y#Y Z Z Z Z Z  "2"2r!F)r>c0|j|jz|jz}|d|jd| |d|jd|d|d| |d|jd|d|jd |d |j|jgd |d |jd |d|j d| dd|d|jd |j r8|j di|j d<|j |j dd<|d|jp|jd|d|jp|jd|drt&j|_dS|drt&j|_dSt&j|_dS)z Adjust the config with `TrainingArguments` values. This stage is run during `TrainingArguments` object creation. train_micro_batch_size_per_gpuper_device_train_batch_sizegradient_accumulation_stepstrain_batch_sizeztrain_batch_size (calculated)gradient_clipping max_grad_normzoptimizer.params.lr learning_ratezoptimizer.params.betaszadam_beta1+adam_beta2zoptimizer.params.eps adam_epsilonzoptimizer.params.weight_decay weight_decayzscheduler.params.warmup_min_lrrzscheduler.params.warmup_max_lr checkpointuse_node_local_storagez fp16.enabledzfp16|fp16_full_evalz bf16.enabledzbf16|bf16_full_evalN) world_sizerErFrBrIrJ adam_beta1 adam_beta2rKrL fill_onlysave_on_each_noder?r:fp16fp16_full_evalbf16bf16_full_evalis_truetorchbfloat16r,float16float32)rargsauto_find_batch_sizerGs rtrainer_config_processz/HfTrainerDeepSpeedConfig.trainer_config_processs' ?T-MMPTPpp  ,  , )$ $     )  , )      +$ $    +T-?QQQ -t/A?SSS  $ _do . #   .0A>RRR 79JN[[[ 7;;; 8$:Lo^^^  ! Y(,  b(I(IDK %BFBXDK %&> ? )Id6ILabbb )Id6ILabbb << ' ' (.DKKK \\. ) ) (-DKKK-DKKKr!cJgd}fd|D}t|dkrud}t|drt|jdr |jj}nt|jdrt |jj}nt|jdr,t|jjdr|jjj}nMt|jdr8t|jjdrt |jjj}|td |d d ||z rBd td |z|zdd|z d|d d| |dtj dkr-dj }td|ddS)z This stage is run after we have the model and know num_training_steps. Now we can complete the configuration process. )$zero_optimization.reduce_bucket_size-zero_optimization.stage3_prefetch_bucket_size4zero_optimization.stage3_param_persistence_thresholdc>g|]}||S)r6).0xrs r zDHfTrainerDeepSpeedConfig.trainer_config_finalize..s) V V VqdllSToo V V V Vr!rNr? hidden_size hidden_sizes text_configzThe model's config file has neither `hidden_size` nor `hidden_sizes` entry, therefore it's not possible to automatically fill out the following `auto` entries in the DeepSpeed config file: zb. You can fix that by replacing `auto` values for these keys with an integer value of your choice.rarbg?rc z scheduler.params.total_num_stepsznum_training_steps (calculated)z!scheduler.params.warmup_num_steps warmup_steps z]Please correct the following DeepSpeed config values that mismatch TrainingArguments values: zF The easiest method is to set these DeepSpeed config values to 'auto'.)lenhasattrr?rimaxrjrkr/rRis_zero3intrBget_warmup_stepsr-join)rr]modelnum_training_stepshidden_size_based_keyshidden_size_auto_keysrir-s` rtrainer_config_finalizez0HfTrainerDeepSpeedConfig.trainer_config_finalizes" " "  !W V V V,B V V V $ % % ) )Kuh'' M5<77 M"',":KKU\>::M"%el&?"@"@KKU\=99MgelF^`m>n>nM"',":"FKKU\=99MgelF^`n>o>oM"%el&>&K"L"LK" Y5JYYY NNA;Q\C\ ] ] ]}} Ck)K788J$  .  -    /  ! !"4 5 5    t  ! # #4?33Jq'qqq  $ #r!)NTF) r"r#r$r%rr0r6rBrrRr_rzr&r's@rr)r)Rs   !!![[[[4 jU;;;I8(8(8(8(tCCCCCCCr!r)c.tj|adSr+)weakrefref_hf_deepspeed_config_weak_ref)hf_deepspeed_config_objs rrr s %,K0G$H$H!!!r!c dadSr+)rrer!runset_hf_deepspeed_configrs%)!!!r!cpt.t tSdS)NF)rrrrer!ris_deepspeed_zero3_enabledrs1$05R5T5T5`,..77999ur!cVt!ttjSdSr+)rr?rer!rdeepspeed_configrs($05R5T5T5`,..55tr!c$ddlddl}ddlm}ddlm|fd|j5|5||jdddn #1swxYwYddddS#1swxYwYdS)aA DeepSpeed ZeRO-3 variant of `PreTrainedModel.initialize_weights`. Mirrors the `smart_apply` dispatch logic but gathers each module's partitioned parameters before calling `_initialize_weights`, so initialization operates on full tensors instead of empty shards. Only rank 0 performs the actual init. rNr)guard_torch_init_functions)PreTrainedModelc|D]0}t|r||j$||1t|d}|r`j|d5jdkr ||ddddS#1swxYwYdS||dS)NF)recurser modifier_rank) children isinstance_initialize_weightslist parameterszeroGatheredParameterscommget_rank)model_or_modulefnchildparamsr _apply_zero3r is_remote_codes rrz.initialize_weights_zero3.._apply_zero34sG$--// ( (E%11 ( UE$=>>>> UB''''o000??@@  02262KK 8 8>**,,11B777 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 B / / / / /s*B??CC) r rYinitializationrmodeling_utilsrrno_gradr)rvrYrrrr rs @@@@rinitialize_weights_zero3r%sgLLL;;;;;;000000))++N 0 0 0 0 0 0 0 0 ;; ' ' ) ) ; ; L 9 : : : ; ; ; ; ; ; ; ; ; ; ; ; ; ; ;;;;;;;;;;;;;;;;;;;s6 BA-! B-A1 1B4A1 5BB  B ct}||didd}|di}t|tr8t ||didd}|dkrt ddd lmmm m }t|d d}|j }i} | D])\} } tj| j| jd | | <*fd |D} fd|D} t'| dkrBi}|D] \}}||| g|| \}}|| vr|||<!|||_|Sd| D}i}i}t+|fd}|D]}||}||| | || \}}|| vr[|T||}|j|j|j}|||}||||||||<|D]\}} ||||j}|D])\}} t| t>r| dn| } | ||<*b#t@$r}tCd|d||d}~wwxYw|||_|S)z Apply weight conversions (renaming and merging/splitting operations) to a state dict. This is a simplified version that handles the conversion without loading into the model. Ntensor_parallel autotp_size inferencetp_sizezWeight conversions (e.g., MoE expert fusion) with DeepSpeed Tensor Parallelism are not yet implemented but support is coming soon. Please disable tensor_parallel in your DeepSpeed config or convert your checkpoint to the expected format first.r)WeightConverterWeightRenamingdot_natural_keyrename_source_key _metadatameta)r0devicec4g|]}t||Srer)rfentryrs rrhz;_apply_weight_conversions_to_state_dict..ks(XXX5j6W6WXXXXr!c4g|]}t||Srer)rfrrs rrhz;_apply_weight_conversions_to_state_dict..ls(ZZZEz%7Y7YZ%ZZZr!rc(i|]}|jD]}||Sre)source_patterns)rf converterks r z;_apply_weight_conversions_to_state_dict..{s*hhhYiNghhAyhhhhr!c|Sr+re)rrs rz9_apply_weight_conversions_to_state_dict..s//!:L:Lr!)key)rtarget_patterns operations)rvr?z'Failed to apply weight conversion for 'zb'. This likely means the checkpoint format is incompatible with the current model version. Error: )"rr:rdictrqNotImplementedErrorcore_model_loadingrrrrgetattrbase_model_prefix state_dictitemsrYemptyshaper0rorsortedkeyspoprrr setdefault add_tensorconvertr?r Exception RuntimeError) rvrweight_mapping ds_configrinference_configrrprefixmodel_state_dictrparam renamings convertersnew_state_dict original_keytensor renamed_keyrpattern_to_converterconversion_mapping sorted_keyssource_patternr new_convertermappingrealized_value target_nameerrrs @@@r'_apply_weight_conversions_to_state_dictrHs!""I-- 1266::=!LL$==b99 & - - b'#3#7#78I2#N#N#R#RS\^_#`#`aaG Q;;%d  ihhhhhhhhhhhz;55H  $F&&((..00[[ U % EKu{SY Z Z ZXXXXNXXXIZZZZ^ZZZJ :!$.$4$4$6$6 5 5 L&..|YFTdeeNK....4{+  '/N $ih*hhh N**0L0L0L0LMMMK#55  --&7&7 iQ[]ceu&v&v# ^ * * *)1@ /$-$=$-$=(3!!! -77 ]SS""; nfUUUU/5{+!3 8 8 : : W $__|-N '5&:&:&<&< 4 4" U$.ud$;$;Fa.3{++ 4   +   #+ s8AK K9K44K9c t|dd|}|_d}|t|dd}|+t|dkrt |||}||_g|tt|dd  fd| D}d d tj ffd ||d fS) a Loads state dict into a model specifically for Zero3, since DeepSpeed does not support the `transformers` tensor parallelism API. Nearly identical code to PyTorch's `_load_from_state_dict` Args: model_to_load: The model to load weights into state_dict: The state dict containing the weights load_config: Optional LoadStateDictConfig containing weight_mapping and other loading options rNrrrc^i|])\}}d|d|n||*S).)r:)rfrvmeta_model_state_dict prefix_models rrz5_load_state_dict_into_zero3_model..se Aq#8";";|.loads'/X\\&"+r5R5R5M12FND"b*M & ' ' 2     $F$;$;6#2#;X]$;$^$^__ ! % , , ??,Q/E/3E,$++E222 ((+++#$$q((^667GWX6YY<<(1133q8844d;;<<<<<<<<<<<<<<< !!5!5VCRC[RW!5!X!XYYM'--// 2 23 ??s ((+++11 *Q-000111111111111111-1C*!?0022 W WKD% UJ (;=UVVV W Ws$$-DD!$D!GG G )r)rF) rcopyrror_weight_conversionsrsetrrr Module) model_to_loadr load_configrrrrrrrs @@@@@@r!_load_state_dict_into_zero3_modelrsrz;55H""J' N .>EE!c.&9&9A&=&=<]JXfgg ,: )J)4466,113344L=*=tDDL$$&&J)W)WRY)W)W)W)W)W)W)W)W)WV D UCCCC | ##r!cJddlm}m}|j}d}d|vr ||}nG|rt d}d|d<d} d |vr ||} n#t||rfd } ||| } || fS) zY A convenience wrapper that deals with optimizer and lr scheduler configuration. r) DummyOptimDummySchedulerN optimizer)rzDetected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)Tzero_allow_untested_optimizer schedulercltj}d|_||}|S)N)rwr)r lr_schedulercreate_scheduler)r trainer_copyr rwtrainers r_lr_scheduler_callablez5deepspeed_optim_sched.._lr_scheduler_callable3sC#y11 -1 )+<<'9Y =   $#r!)lr_scheduler_callable) accelerate.utilsrrr? is_offloadloggerinfocreate_optimizerr) rhf_deepspeed_configr]rwmodel_parametersrrr?rr rs ` ` rdeepspeed_optim_schedr s<;;;;;;; 'FIfJ&6777  ) ) + +  KKV   ,,.. 26./Lf%~i00 i , , c $ $ $ $ $ $*>)KabbbL l ""r!Fcddlm}|j}|j}|jjjj}||||| | |rU| std| d| dd\}}d} nd|_|jdid d } | d kr4ddl} | || ||j }t)t+d |} t/||||| \}}||fS) a Init DeepSpeed, after updating the DeepSpeed configuration with any relevant Trainer's args. If `resume_from_checkpoint` was passed then an attempt to resume from a previously saved checkpoint will be made. Args: trainer: Trainer object num_training_steps: per single gpu resume_from_checkpoint: path to a checkpoint if to resume from after normal DeepSpeedEngine load inference: launch in inference mode (no optimizer and no lr scheduler) auto_find_batch_size: whether to ignore the `train_micro_batch_size_per_gpu` argument as it's being set automatically by the auto batch size finder Returns: optimizer, lr_scheduler We may use `deepspeed_init` more than once during the life of Trainer, when we do - it's a temp hack based on: https://github.com/deepspeedai/DeepSpeed/issues/1394#issuecomment-937405374 until Deepspeed fixes a bug where it can't resume from a checkpoint after it did some stepping https://github.com/deepspeedai/DeepSpeed/issues/1612 r)rzMZeRO inference only makes sense with ZeRO Stage 3 - please adjust your configrr )NNNrrr)rvrr0r?c|jSr+) requires_grad)ps rrz deepspeed_init..{sr!)deepspeed.utilsrrvr] acceleratorstatedeepspeed_plugin hf_ds_configrzsetLevelget_process_log_levelrrr/del_config_sub_treerr?r:r tp_model_initr0rfilterrr) rrwr ds_loggerrvr]rrr rdeepspeed_tp_sizer s rdeepspeed_initr)Cs*433333 ME I 11$8LMMMMMr!c d|vrd|vr |d|d<|di|}|j}|jdkr&|jdkrddlm}|}n6|j |jd}ntd |j} tj j j || |dd kd } tj j j | |  t fd t!| D} t } | t#| dz }|r||fn|S)aq Computes the loss under sequence parallelism with `sp_backend="deepspeed"` and `sp_size > 1`. Performs weighted loss aggregation across SP ranks, accounting for varying numbers of valid tokens per rank (e.g., when some ranks receive only padding or prompt tokens that are masked with -100). Args: accelerator (`Accelerator`): The accelerator instance with `torch_device_mesh` support. model (`torch.nn.Module`): The model to compute the loss for. inputs (`dict[str, torch.Tensor | Any]`): The input data for the model. Must include `"shift_labels"` key. return_outputs (`bool`): Whether to return the model outputs along with the loss. pc (`accelerate.parallelism_config.ParallelismConfig`): The parallelism configuration. Returns: The loss, or a tuple of `(loss, outputs)` if `return_outputs` is `True`. labels shift_labelsr rr)groupsNspzSequence parallelism is enabled but no SP process group is available. Ensure torch_device_mesh is initialized or sp_backend='deepspeed' with sp_size > 1.)groupirc3RK|]!}|dk||zV"dS)rNre)rfrankgood_tokens_per_ranklosses_per_ranks r z,deepspeed_sp_compute_loss..sO   % ) )  4T :: ) ) ) )r!re)loss sp_backendsp_sizerr;_get_sequence_parallel_grouptorch_device_mesh get_groupr/rYrr functional all_gatherviewsumrangerq)rrvinputsreturn_outputspcoutputsrCr;sp_group sp_world_size good_tokens total_losstotal_good_tokensr@rAs @@rdeepspeed_sp_compute_lossrWs,v.F":":!.1xeoofooG rasn ######999999HHHHHHHHHHLLL  H % %   3!7!7!9!93OOOOOOO322222........2ppppp0ppph!%III)))  ; ; ;FhhhVW$W$W$W$t3#3#3#l@#@#@#@#FPPPP0NNNN"7777777777r!