o diD @sdZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZddlZddlZddlmZddlmZddlmZdd lmZdd lmZdd lmZddl Z!dd l"m#Z#d dl$m%Z%m&Z&e r|ddl'm(Z(ddl)m*Z*e+e,Z-dZ.dZ/dZ0dZ1dZ2ddZ3ddZ4GdddZ5ddZ6de%d e%fd!d"Z7d#e8d$e d ej9fd%d&Z:d'ej9d e8fd(d)Z;d'eej9e8e .  dJd#e8d/eee8d$eeee e8e fe e8e fffd0d1Z?dKd/e e8d2ee@d e@fd3d4ZAd5e e8d6e d e e8fd7d8ZBGd9d:d:eZC  dLd;dd?ee e8d@ee ege ffdAdBZDdCdDZEGdEdFdFZFdGe!jGd ejHfdHdIZIdS)Mz:Utility functions, classes and constants for ONNX Runtime.N)Enum) signature) TYPE_CHECKINGAnyCallableDictListOptionalSequenceTupleUnion)version)tqdm)EvalPrediction) nested_concat)EvalLoopOutput)logging) TypeHelper) OnnxConfigOnnxConfigWithLoss)Dataset)ORTModelz model.onnxzencoder_model.onnxzdecoder_model.onnxzdecoder_with_past_model.onnxzdecoder_model_merged.onnxcCs(tjtjdd}tj|rdSdS)z6 Checks if onnxruntime-training is available. rtrainingTF)ospathjoinort__path__exists)path_training_dependecyr"g/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/optimum/onnxruntime/utils.py!is_onnxruntime_training_available6s r$cCstjdduS)z& Checks if CuPy is available. cupyN) importlibutil find_specr"r"r"r#is_cupy_availableAsr)c@s,eZdZdZidddddddddddddd d dd d d dd dddddddd d dd dd idd dd dd dddd dddddd ddddddddd dd!dd"dd#dd$dddddd%Zed&ed'efd(d)Zed&efd*d+Zed&efd,d-Z d.S)/ORTConfigManagera A class that contains all the information needed by ONNX Runtime optimization for a given model type. Attributes: _conf (`Dict[str]`): A dictionary mapping each supported model type to the corresponding ONNX Runtime model type. albertbertbartbig_birdbigbird_pegasus blenderbotbloomgpt2 camembertcodegendebertaz deberta-v2dinov2vit distilbertelectra gpt_bigcodegpt_neogpt_neoxgptjgranitelongt5llamamarianmbartmistral modernbertmpnetmt5m2m_100 nystromformerpegasusroberta segformert5)r7whisperz xlm-roberta pix2struct model_typereturncCs|||j|SN)check_supported_model_conf)clsrOr"r"r#get_model_ort_type~s  z#ORTConfigManager.get_model_ort_typecCs8||jvrd|j}t|d|d|ddS)Nz, z' model type is not supported yet. Only ' are supported. If you want to support z) please propose a PR or open up an issue.)rSrkeysKeyError)rTrO model_typesr"r"r#rRs  z&ORTConfigManager.check_supported_modelcCsJgd}||jvs|j||vr#td|dt|jd|ddS)N) r-r,r2tnlrrLunetvaeclipr7swinswinv2z7ONNX Runtime doesn't support the graph optimization of z yet. Only rVzc please propose a PR or open up an issue in ONNX Runtime: https://github.com/microsoft/onnxruntime.)rSNotImplementedErrorlistrW)rTrOoptimization_config&supported_model_types_for_optimizationr"r"r#"check_optimization_supported_models z3ORTConfigManager.check_optimization_supported_modelN) __name__ __module__ __qualname____doc__rS classmethodstrrUrRrdr"r"r"r#r*Js       !"#)r*cCs|j|j||jSrQ)parentjoinpathstem with_suffixsuffix)filename identifierr"r"r#generate_identified_filenamesrr onnx_configrPcCst|SrQ)r)rsr"r"r#wrap_onnx_config_for_losssrtproviderprovider_optionscCs*|dvrtd|ddStdS)zV Gets the PyTorch device (CPU/CUDA) associated with an ONNX Runtime provider. )CUDAExecutionProviderTensorrtExecutionProviderROCMExecutionProviderzcuda: device_idrcpu)torchdeviceget)rurvr"r"r#get_device_for_providers rr}cCs&|jdkrdtvrdSdSdS)zW Gets the ONNX Runtime provider associated with the PyTorch device (CPU/CUDA). cudaryrwCPUExecutionProvider)typelowerrget_available_providersr}r"r"r#get_provider_for_devices  rcCsZ|dkr td}n tjj|d}i}|jdkr)|jdur$td}|j|d<||fS)zuGets the relevant torch.device from the passed device, and if relevant the provider options (e.g. to set the GPU id).r{rrNzcuda:0rz)r|r}_C_nn _parse_torindex)r}rvr"r"r# parse_devices     rcCsttjtdkrtjdkr|dvrtjtjddd}tjtjddd}tjtjddd}t |d k}| }d |vsGd |vr|tj |r[tj |r[t d |d tj |rtt rt|dkrst d|d|dnt d|d|dkrtjdddkrt d|dkrtjdddkrt dWdn1swYt}||vrtd|d|d dS)z Ensure the ONNX Runtime execution provider `provider` is available, and raise an error if it is not. Args: provider (str): Name of an ONNX Runtime execution provider. z1.16.0nt)rwrxrcapiz libonnxruntime_providers_cuda.soz$libonnxruntime_providers_tensorrt.soz_ld_preload.pyrORT_CUDA ORT_TENSORRTz`onnxruntime-gpu` is installed, but GPU dependencies are not loaded. It is likely there is a conflicting install between `onnxruntime` and `onnxruntime-gpu`. Please install only `onnxruntime-gpu` in order to use .rxz Asked to use z5, but `onnxruntime-training` package doesn't support z-. Please use `CUDAExecutionProvider` instead.zw, but `onnxruntime-gpu` package was not found. Make sure to install `onnxruntime-gpu` package instead of `onnxruntime`.rwORT_CUDA_UNAVAILABLE01z`onnxruntime-gpu` package is installed, but CUDA requirements could not be loaded. Make sure to meet the required dependencies: https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.htmlORT_TENSORRT_UNAVAILABLEa4`onnxruntime-gpu` package is installed, but TensorRT requirements could not be loaded. Make sure to meet the required dependencies following https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html and https://hf.co/docs/optimum/onnxruntime/usage_guides/gpu#tensorrtexecutionprovider .NzR as an ONNX Runtime execution provider, but the available execution providers are )r parser __version__rnamerrropenreadisfile ImportErrorr$environr~r ValueError)ru path_cuda_lib path_trt_libpath_dependecy_loadingf file_stringavailable_providersr"r"r#validate_provider_availabilitysP    rr providerscCs|dur|g}|D]}t|q |durigt|}||fSt|tr4|gigt|d}||fSt|t|krKtdt|dt|d||fS)a Prepare the providers and provider options for ONNX Runtime. Args: provider (`str`): The provider to use. If `None`, the default provider will be used. providers (`Sequence[str]`, `optional`): The list of providers to use. If `None`, the default provider will be used. provider_options (`Union[Sequence[Dict[str, Any]], Dict[str, Any]]`, `optional`): The options to use for the providers. If `None`, the default options will be used. NrzdWhen passing a list of provider options, it should be the same length as the list of providers. Got z provider options for z providers.)rlen isinstancedictr)rurrvr"r"r#&prepare_providers_and_provider_options s&  ruse_io_bindingcCsN|dur|ddkrd}|S|ddkr%|ddkr%|dur#tdd}|S)z* Whether to use IOBinding or not. NrrwTrzNo need to enable IO Binding if the provider used is neither CPUExecutionProvider nor CUDAExecutionProvider. IO Binding will be turned off.F)loggerwarning)rrr"r"r#check_io_binding+sr input_namesfunccCsLt|}g}|jD]}t|d}|D] }t||r"||qq |S)a Returns the input names from input_names keys ordered according to the signature of func. This is especially useful with the forward function when using IO Binding, as the input order of the ONNX and forward may be different. Method inspired from OnnxConfig.ordered_inputs. Args: input_names (`List[str]`): Names of the inputs of the ONNX model. func (`Callable`): Callable to remap the input_names order to. z(\.\d*)?)r parametersrecompilesearchappend)rrsignature_func_ordered_input_namesparam param_regexrr"r"r#get_ordered_input_names;s   rc@s`eZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdS)ORTQuantizableOperatorGather TransposeEmbedLayerNormalizationConvMatMulAddMulReluClip LeakyReluSigmoidMaxPoolGlobalAveragePoolSplitPadReshapeSqueeze UnsqueezeResize AveragePoolConcatN)rerfrgrrEmbedLayerNormalizationQuantrrrrrrrrrrrrrrrrrrr"r"r"r#rSs,rmodelrdatasetr label_namescompute_metricsc sd}d}t|ddD]\tfdd|D}|r/tfdd|D}t|dkr.|d}nd}fd d |jD|di}t|dkrL|d}|durR|nt||d d }|dur_|nt||d d }q |dur||dur||dur||t||d } ni} t||| t|dS)az Run evaluation and returns metrics and predictions. Args: model (`ORTModel`): The ONNXRuntime model to use for the evaluation step. dataset (`datasets.Dataset`): Dataset to use for the evaluation step. label_names (`List[str]`, `optional`): The list of keys in your dictionary of inputs that correspond to the labels. compute_metrics (`Callable[[EvalPrediction], Dict]`, `optional`): The function that will be used to compute metrics at evaluation. Must take an `EvalPrediction` and return a dictionary string to metric values. N Evaluation)descc3s|] }|duVqdSrQ)r~).0kinputsr"r# sz"evaluation_loop..c3s"|] }t|gVqdSrQ)nparrayr~)rrrr"r#rs rrcs&i|]}|vr|t|gqSr")rr)rkeyrr"r# s&z#evaluation_loop..i) padding_index) predictions label_ids)rrmetrics num_samplesr")ralltuplerrrrr) rrrr all_preds all_labels has_labelslabelspredsrr"rr#evaluation_loopns(  rcst|tjjrtjdt|ddSt|tjjr0tjdt|j j ddSt|t rKt|dtjjtjjfrKfdd|DSt|t rltt t|tjjtjjfrlfdd|DS|S)Nrrrcsg|]}t|qSr"np_to_pt_generators)rarr"r# sz'np_to_pt_generators..csi|] \}}|t|qSr"r)rrvrr"r#rsz'np_to_pt_generators..)rrrandom RandomStater| Generator manual_seedint get_state bit_generatorstaterarnextitervaluesitems) np_objectr}r"rr#rs""$ rc@"eZdZddZGdddZdS)DummyWhisperModelcCs||_dSrQ)Encoderencoderselfr"r"r#__init__szDummyWhisperModel.__init__c@r)zDummyWhisperModel.EncodercCs |jdd|_|jdd|_dS)N)rstride)r)rconv1conv2rr"r"r#rsz"DummyWhisperModel.Encoder.__init__c@seZdZddZdS)zDummyWhisperModel.Encoder.ConvcCs ||_dSrQr)rr r"r"r#rs z'DummyWhisperModel.Encoder.Conv.__init__N)rerfrgrr"r"r"r#rs rN)rerfrgrrr"r"r"r#rsrN)rerfrgrrr"r"r"r#rsrsessioncCsV|D]}t|j}|jr|Sq|D]}t|j}|jr'|SqtjS)z Returns the `torch.dtype` associated with the ONNX Runtime session. This dtype is inferred from the input/output dtypes of the session. If no floating point type is found, it defaults to `torch.float32`. ) get_inputsrort_type_to_torch_typeris_floating_point get_outputsr|float32)r input torch_dtypeoutputr"r"r#get_dtype_from_sessions    r)rNNrQ)NN)Jrhr&rrenumrinspectrtypingrrrrrr r r r numpyrr| packagingr r transformersrtransformers.trainer_pt_utilsrtransformers.trainer_utilsrtransformers.utilsr onnxruntimer*onnxruntime.transformers.io_binding_helperrexporters.onnxrrdatasetsr modeling_ortr get_loggerrerONNX_WEIGHTS_NAMEONNX_ENCODER_NAMEONNX_DECODER_NAMEONNX_DECODER_WITH_PAST_NAMEONNX_DECODER_MERGED_NAMEr$r)r*rrrtrjr}rrrrrrboolrrrrrrInferenceSessiondtyperr"r"r"r#s~   ,            Y ( 7 " "  2