o dix@s2ddlZddlZddlZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZddlmZddlmZmZmZmZmZdd lmZmZmZdd lmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$dd l%m&Z&d d l'm(Z(e rwddl)Z)ere(ddrddl*m+Z+GdddeZ,eGdddeZ-dS)N) dataclassfield)Path)Optional)version)TrainingArguments DebugOption)EvaluationStrategy FSDPOption HubStrategyIntervalStrategy SchedulerType)OptimizerNamesdefault_logdirlogger) ExplicitEnumget_full_repo_nameis_accelerate_availableis_safetensors_availableis_torch_availableis_torch_bf16_cpu_availableis_torch_bf16_gpu_availableis_torch_tf32_availablelogging) strtobool)is_transformers_version>=4.38.0)AcceleratorConfigc@seZdZdZdZdS)ORTOptimizerNameszb Stores the acceptable string identifiers for optimizers in `onnxruntime.training.optim`. adamw_ort_fusedN)__name__ __module__ __qualname____doc__ADAMW_ORT_FUSEDr(r(o/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/optimum/onnxruntime/training_args.pyr!9sr!c@seZdZUdZedddidZeeed<edddidZ ee ed <eddd idZ ee ed <ed dd idZ eeed<edddidZ eeed<ddZd S)ORTTrainingArgumentsaf Parameters: optim (`str` or [`training_args.ORTOptimizerNames`] or [`transformers.training_args.OptimizerNames`], *optional*, defaults to `"adamw_hf"`): The optimizer to use, including optimizers in Transformers: adamw_hf, adamw_torch, adamw_apex_fused, or adafactor. And optimizers implemented by ONNX Runtime: adamw_ort_fused. adamw_hfhelpzThe optimizer to use.)defaultmetadataoptimFzzUse ModuleWithLoss Wrapper to compute loss inside the training loop, having this will help save memory for ORTModule Runs.use_module_with_lossa-Configure ORTModule to save onnx models. Defaults to False. The output directory of the onnx models by default is set to args.output_dir. To change the output directory, the environment variable ORTMODULE_SAVE_ONNX_PATH can be set to the destination directory path. save_onnxNzQPrefix for the saved ORTModule file names. Must be provided if save_onnx is True. onnx_prefixWARNINGzConfigure ORTModule log level. Defaults to WARNING. onnx_log_level can also be set to one of VERBOSE, INFO, WARNING, ERROR, FATAL.onnx_log_levelcCs|jdur tj|j|_|jdur!|jdur!tj|jt|_|jdur.tj|j|_|jdur;t t j k|_t |j trLtdt|j j|_ t|j |_ t|j|_t|j|_t|j|_t|j|_|jdurx|j tjkrxd|_|j tjkr|jdus|jdkr|jdkrtd|j|j|_n t d|j d|jtjkr|jdkrt d|jd |jtjkr|jd kr|jt!|jkrt d |jt!|j|_|j tjkr|jd kr|jt!|jkrt d |jt!|j|_|jtjkr|j"d kr|j"t!|j"krt d |j"t!|j"|_"|j#r|j |jkr6t d|j"d|jd|j tjkr|j"|jdkr|jd ksR|j"d kr|jd kr^|j"d kskt d|j"d|jdd}|j"||j|dkrt d|j"d|jdt d|j"d|jdt$}|j%r|st d|j%d|j%s|rtd|j%d|j#s|jtj&kr|j'durd|_'|j(dur|j'dur|j'dv|_(|j)dur|j|_)|j*dkr,t+r,|j,r|j,dkrtdt|j,|_-|j.s|j/r,|j0rt1st d |j0s,t2j34r,t5s,t d!|j6r8|j.r8t d"|j7rD|j/rDt d#|j.rR|j-d$krRt d%|jtj&krl|j tjkrdt d&t+slt d'zt8|j9|_9Wnt yt:|j9|_9Ynw|j;rtd(tt:j<|_9|j9t:j=krt+rt>?t>?t2j@jAt>?d)krt d*t>?t>?t2j@jAt>?d)kr|j6rt d+|jBr|jCst d,tDd-ds|jtjEd-<|jFtjEd.<t+r |jGjHd/kr |jGjHd0krd1tjEvs |j6s |j7r t d2t+r8|jGjHd/kr8|jGjHd0kr%d1tjEvs8|jGjHd3kr8|j.s4|j/r8t d4|jIdurHtd5t|jI|_J|jKdusT|jJdur[|jLs[d|_L|jLrh|jJdurhd6|_J|jLrd7}|jJtjE|d8<|jKdur|jKtjE|d9<|j*dkrt+r|jLrtMr|jNdur|j6r|j.rtd:dt2jOj3jP_Qdt2jOjR_QntSd;t+r|jNdur|jNrtMrdt2jOj3jP_Qdt2jOjR_Qnt d<tMrdt2jOj3jP_Qdt2jOjR_Q|j-d$kr tjETd=d>}|j6rd?}n|j.rd@}|tjEd=<|jUdurtdAdB|_U|jUdBks%|jUdBgkr0ddClVmW}||_Un|jUdDks=|jUdDgkrAg|_Un t |jUtXsM|jUg|_U|jYdksY|jYd kr]t dE|jYdkrn|jZdkrntdFt |j[t\r~|j[r{dGndH|_[t |j[t]rdIdJ|j[^D|_[|j[t_j`gkrt dKt_ja|j[vrt_jb|j[vrt dL|jcduri|_ct |jct]r td|j[dkrtdMtejf|jcdNdOdP.}tgh||_ctX|jciD]}|jdQr|jck|}||jc|dRd<qWdn 1swY|jldkrtdSttm|jcTdTd|jl|jcdT<t |jcTdUdt]r9|jcdUg|jcdU<|jndurStdVt|jcTdUg|jng|jcdU<td|j[dkrh|jcdTdkrhtdWtd|j[dkr|jcTdUddurtdXtd|j[dkr|jcdTdkr|jcTdUddurt dY|jcTd0d|jcd0<|jcTdZd|jcdZ<|jcTd[d|jcd[<|jcd0rtd|j[dkr|jcTd\i|_od]|jovrtpt2|jod]|jod]<d^|jovrtpt2|jod^|jod^<ntd_n |jcd[rtd`td|j[dkr|jcd0sdatjEdb<ddclqmr} ms} dd}|j[D]v} | t| vrAt]| u| td tjE|de<q%| t_j`krQdatjE|df<q%| t_jvkr| dtjE|dg<|jcdTdkrt]|jcdTtjE|dh<| d tjE|dg<q%|jcTdUddurdi|jcdUtjE|dj<q%|jcTdkdl} | ttjE|dm<|jcTdndotjE|dp<|jcTdqdatjE|dr<|jcTdsdotjE|dt<twr*txdudvr*t |jytzs|jydurtz|_ynt |jyt{rtzdi|jy|_yntz||jy|_y|j}durtdwt|j}|jy_}|j~dur*tdxt|j~|jy_~|jrHtdyt|jdur>dz|_n|jdz7_d|_t |jt]r[d{dJ|j^D|_n |jdurdg|_d|_|jrtwsst d|dd}lm} | |j|_|j|dd~lm}datjEd<||jd|_n&ttjETddordd~lm}||_tjETd=d>}|j||j|jdurtdt|j|_|jdurt|j|j|jd|_|jdurtd|jdtn*td|jdtn|jdur|jdt|jj|_td|jdt|j-d$kr:tjETd=d>}|j6r/d?}n|j.r5d@}|tjEd=<|jdurGtddStddS)Nuusing `EvaluationStrategy` for `eval_strategy` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `IntervalStrategy` insteadFTrz4using `logging_steps` to initialize `eval_steps` to zevaluation strategy z9 requires either non-zero --eval_steps or --logging_stepszlogging strategy z" requires non-zero --logging_stepsz5--logging_steps must be an integer if bigger than 1: z2--eval_steps must be an integer if bigger than 1: z2--save_steps must be an integer if bigger than 1: z--load_best_model_at_end requires the saving steps to be a multiple of the evaluation steps, which cannot get guaranteed when mixing ratio and absolute steps for save_steps z and eval_steps .z--load_best_model_at_end requires the saving steps to be a multiple of the evaluation steps, which cannot get guaranteed when mixing ratio and absolute steps for save_stepsi@Bzg--load_best_model_at_end requires the saving steps to be a multiple of the evaluation steps, but found z, which is not a multiple of zm--load_best_model_at_end requires the saving steps to be a round multiple of the evaluation steps, but found z#, which is not a round multiple of z--save_safetensors=z& requires safetensors to be installed!z7Found safetensors installation, but --save_safetensors=z. Safetensors should be a preferred weights saving format due to security and performance reasons. If your model cannot be saved by safetensors please feel free to open an issue at https://github.com/huggingface/safetensors!loss)r7 eval_lossptautoux`fp16_backend` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `half_precision_backend` insteadzLYour setup doesn't support bf16/(cpu, tpu, neuroncore). You need torch>=1.10z[Your setup doesn't support bf16/gpu. You need torch>=1.10, using Ampere GPU with cuda>=11.0z6At most one of fp16 and bf16 can be True, but not bothzDAt most one of fp16 and bf16 can be True for full eval, but not bothapexzt `--half_precision_backend apex`: GPU bf16 is not supported by apex. Use `--half_precision_backend cuda_amp` insteadz@lr_scheduler_type reduce_lr_on_plateau requires an eval strategyz=0.2.0ur`--adafactor` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--optim adafactor` insteadz2.0.0z8--optim adamw_torch_fused requires PyTorch 2.0 or higherz:--optim adamw_torch_fused with --fp16 requires PyTorch>2.0z1onnx_prefix must be provided if save_onnx is TrueORTMODULE_SAVE_ONNX_PATHORTMODULE_LOG_LEVELcudaxlaGPU_NUM_DEVICESzFP16 Mixed precision training with AMP or APEX (`--fp16`) and FP16 half precision evaluation (`--fp16_full_eval`) can only be used on CUDA devices.cpuzBF16 Mixed precision training with AMP (`--bf16`) and BF16 half precision evaluation (`--bf16_full_eval`) can only be used on CUDA or CPU/TPU/NeuronCore devices.uv`torchdynamo` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `torch_compile_backend` insteadinductorACCELERATE_DYNAMO_BACKENDMODEz`Setting TF32 in CUDA backends to speedup torch compile, you won't see any improvement otherwise.zaThe speedups for torchdynamo mostly come wih GPU Ampere or higher and which is not detected here.zC--tf32 requires Ampere or a newer GPU arch, cuda>=11 and torch>=1.7ACCELERATE_MIXED_PRECISIONnofp16bf16a The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).all)$get_available_reporting_integrationsnonez$warmup_ratio must lie in range [0,1]zoBoth warmup_ratio and warmup_steps given, warmup_steps will override any effect of warmup_ratio during training full_shardcSg|]}t|qSr()r .0sr(r(r) z6ORTTrainingArguments.__post_init__..z`--fsdp offload` can't work on its own. It needs to be added to `--fsdp full_shard` or `--fsdp shard_grad_op`. For example, `--fsdp "full_shard offload"`.zB`--fsdp full_shard` is not compatible with `--fsdp shard_grad_op`.z:`--fsdp_config` is useful only when `--fsdp` is specified.rzutf-8)encodingfsdp_zEusing `--fsdp_min_num_params` is deprecated. Use fsdp_config instead min_num_paramstransformer_layer_cls_to_wrapzTusing `--fsdp_transformer_layer_cls_to_wrap` is deprecated. Use fsdp_config instead z;`min_num_params` is useful only when `--fsdp` is specified.zJ`transformer_layer_cls_to_wrap` is useful only when `--fsdp` is specified.zL`min_num_params` and `transformer_layer_cls_to_wrap` are mutually exclusive. xla_fsdp_v2xla_fsdp_grad_ckptxla_fsdp_settings compute_dtype buffer_dtypez5XLA FSDP can be used only when `--fsdp` is specified.zB`--xla_fsdp_grad_ckpt` is useful only when `--xla` is set to true.trueACCELERATE_USE_FSDP)FSDP_AUTO_WRAP_POLICYFSDP_SHARDING_STRATEGYFSDP_SHARDING_STRATEGYOFFLOAD_PARAMSAUTO_WRAP_POLICYMIN_NUM_PARAMS,TRANSFORMER_CLS_TO_WRAPfsdp_backward_prefetch NO_PREFETCHBACKWARD_PREFETCHforward_prefectfalseFORWARD_PREFETCHsync_module_statesSYNC_MODULE_STATESuse_orig_paramsUSE_ORIG_PARAMSrruUsing `--dispatch_batches` is deprecated and will be removed in version 4.41 of 🤗 Transformers. Use `--accelerator_config {'dispatch_batches':VALUE} insteaduUsing `--split_batches` is deprecated and will be removed in version 4.41 of 🤗 Transformers. Use `--accelerator_config {'split_batches':VALUE} insteaduusing `--tpu_metrics_debug` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--debug tpu_metrics_debug` insteadz tpu_metrics_debugcSrOr(rrPr(r(r)rS rTzJ--deepspeed requires Accelerate to be installed: `pip install accelerate`.)HfTrainerDeepSpeedConfig)DeepSpeedPluginACCELERATE_USE_DEEPSPEED) hf_ds_configuu`--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead.) organizationtokenu`--push_to_hub_model_id` and `--push_to_hub_organization` are deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_model_id` instead and pass the full repo name to this argument (in this case z).u`--push_to_hub_model_id` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_model_id` instead and pass the full repo name to this argument (in this case /u`--push_to_hub_organization` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_model_id` instead and pass the full repo name to this argument (in this case zeUsing ModuleWithLoss Wrapper.loss will be computed during training loop and it will save memory peak z!Not Using ModuleWithLoss Wrapper.r() output_dirospath expanduser logging_dirjoinr disable_tqdmrgetEffectiveLevelrWARN isinstance eval_strategyr warningswarn FutureWarningvaluer logging_strategy save_strategyr hub_strategyrlr_scheduler_typedo_evalNOSTEPS eval_steps logging_stepsinfo ValueErrorint save_stepsload_best_model_at_endrsave_safetensorsREDUCE_ON_PLATEAUmetric_for_best_modelgreater_is_betterrun_name frameworkr fp16_backendhalf_precision_backendrIbf16_full_evaluse_cpurtorchr> is_availablerrHfp16_full_evalr!r/r adafactor ADAFACTORADAMW_TORCH_FUSEDrparse __version__ base_versionr1r2getenvenvironr4devicetype torchdynamotorch_compile_backendtorch_compile_mode torch_compilertf32backendsmatmul allow_tf32cudnnwarningget report_totransformers.integrationsrKlist warmup_ratio warmup_stepsfsdpboolstrsplitr OFFLOAD FULL_SHARD SHARD_GRAD_OP fsdp_configlenioopenjsonloadkeys startswithpopfsdp_min_num_paramsmax"fsdp_transformer_layer_cls_to_wrapxla_fsdp_configgetattraccelerate.utils.constantsrbrcupperindex AUTO_WRAPrraccelerator_configr dictfrom_json_filedispatch_batches split_batchestpu_metrics_debugdebugdeepspeed_plugin deepspeed#transformers.integrations.deepspeedruhf_deepspeed_configtrainer_config_processaccelerate.utilsrvrset_mixed_precisionset_deepspeed_weakrefpush_to_hub_token hub_tokenpush_to_hub_model_idrpush_to_hub_organization hub_model_idrnamer0)selfLARGE_MULTIPLIERsafetensors_availableprefixmixed_precision_dtyperKfkvrbrc fsdp_optionprefetch_policyrurvmixed_precisionr(r(r) __post_init__js*                        (                           $                                   z"ORTTrainingArguments.__post_init__)r#r$r%r&rr/rr__annotations__r0rr1r2r4rr(r(r(r)r*As.   r*).rrr}r dataclassesrrpathlibrtypingr packagingr transformersrtransformers.debug_utilsr transformers.trainer_utilsr r r r rtransformers.training_argsrrrtransformers.utilsrrrrrrrrrtransformers.utils.genericrutils.import_utilsrrtransformers.trainer_pt_utilsr r!r*r(r(r(r)s,     ,