o did@sdZddlZddlZddlmZddlmZmZmZm Z m Z m Z m Z ddl ZddlZddlmZmZddlmZddlmZdd lmZd d lmZmZmZmZmZeeZ e!dZ"Gd d d Z#GdddZ$dS)zWDefines the base classes that are used to perform inference with ONNX Runtime sessions.N)Path)AnyDictListOptionalSetTupleUnion)InferenceSession IOBinding) TypeHelper)_get_model_external_data_paths) get_logger)get_device_for_providerget_dtype_from_sessionget_provider_for_device parse_devicevalidate_provider_availabilityc@szeZdZdZd>dedeefddZede fdd Z ede fd d Z ede e fd d Z ede fddZedee effddZedee effddZedejfddZejddZedejfddZedeefddZejdefddZddZdefd d!Zded"ee eejejffdee ejffd#d$Zded%e ejdee eejejfffd&d'Zd(e d)e e!dejfd*d+Z"d(e d,ee e!fde e!fd-d.Z#d/ee d,ee e!fde!fd0d1Z$   d?d"ee ejfd2ee%e d3eee e fd4eee e e!fde ee e e!fee ejfff d5d6Z&d7d8Z'd9d:Z(d;ee e)ffdPz=ORTSessionMixin.initialize_ort_attributes..cSrrr)rroutputrrr!r"Qr#cSi|]}|j|jqSrrshaperr rrr!r"RcSr%rr&rr$rrr!r"Sr)cSr%rrtyper(rrr!r"Tr)cSr%rr+r*rrr!r"Ur))rr _model_pathpathproviderloggerinfo_use_io_bindingr _io_bindingr_dtyperprovider_option_device enumerate get_inputs input_names get_outputs output_names input_shapes output_shapes input_dtypes output_dtypes)selfrrrrr!initialize_ort_attributes2s&    z)ORTSessionMixin.initialize_ort_attributesreturncCstd|jS)zW Returns the path of the onnx file from which the session was created. zThe `ORTSessionMixin.model_path` property is deprecated and will be removed in a future version. Please use `ORTSessionMixin.path` instead (`ORTSessionMixin.path` is a proper Path object).)r0warningr.r@rrr! model_pathWszORTSessionMixin.model_pathcCstd|jjS)zW Returns the name of the onnx file from which the session was created. zThe `ORTSessionMixin.model_name` property is deprecated and will be removed in a future version. Please use `ORTSessionMixin.path.name` instead (`ORTSessionMixin.path` is a proper Path object).)r0rCr.rrDrrr! model_namebszORTSessionMixin.model_namecC |jS)T Returns a list of Execution Providers registered with the session. )r get_providersrDrrr! providersm zORTSessionMixin.providerscCs |jdS)R Returns the main Execution Provider registered with the session. r)rJrDrrr!r/trKzORTSessionMixin.providercCrG)U Returns a dictionary of Execution Providers configurations/options. )rget_provider_optionsrDrrr!provider_options{rKz ORTSessionMixin.provider_optionscCs |j|jS)S Returns the configuration/options of the main Execution Provider. )rOr/rDrrr!r5s zORTSessionMixin.provider_optioncC|jS) Returns the `torch.device` associated with the ONNX Runtime session. This device is inferred from the provider and provider options. )r6rDrrr!deviceszORTSessionMixin.devicecOtd)NzThe device attribute is read-only, please use the `.to(device)` method to change both the device and the execution provider accordingly.)AttributeErrorr@argskwargsrrr!rSscCrQ) Returns the `torch.dtype` associated with the ONNX Runtime session. This dtype is inferred from the input/output dtypes of the session. If no floating point type is found, it defaults to `torch.float32`. )r4rDrrr!dtypeszORTSessionMixin.dtypecCrQ)< Returns whether IO Binding is used or not. )r2rDrrr!rszORTSessionMixin.use_io_bindingvaluecCst|ts td||_dS)z, Sets the IO Binding usage. z+`use_io_binding` should be a boolean value.N) isinstancebool ValueErrorr2)r@r\rrr!rs  c Osd}d}|D])}t|ttjfr|}qt|trt|}qt|tjr'|}qt|tjr/|}q|D]\}}|dkr?|}q4|dkrE|}q4|durL|S|durR|St|\}}t|} t | ||jkrg|S|j j | g|gd|j dur|j dkrtdd|_ ||_|S)a  Moves the session to the specified device by updating the execution provider and its options. Args: device (`str`, `int`, `torch.device`): The device to move the session to. It can be a string (e.g., "cuda", "cpu"), an integer (e.g., 0 for GPU 0), or a `torch.device` object. Returns: `ORTSessionMixin`: The updated session. Raises: ValueError: If the device is not supported or if the provider is not available. NrSrZ)rOrz`use_io_binding` was set to `None` before the provider was changed to CUDAExecutionProvider. Setting `use_io_binding=True` to leverage IO Binding and improve performance. You can disable it by setting `model.use_io_binding=False`.T)r]strtorchrSintrZitemsrrrr set_providersrr/r0r1r6) r@rWrXrZrSargkeyr\r5r/rrr!tosH         zORTSessionMixin.to use_torchcCs"|dur |jdurtddSdS)z Raises an error if IO Binding is requested although the tensor used are numpy arrays. Args: use_torch (`bool`): Whether the tensor used during inference are of type torch.Tensor or not. FTzIO Binding can not be used when passing numpy inputs. Please disable IO Binding with `model.use_io_binding=False`, or pass `torch.Tensor` inputs instead.N)rr_)r@rhrrr!raise_on_numpy_input_io_bindings z/ORTSessionMixin.raise_on_numpy_input_io_binding model_inputscCsi}|jD]=}||ddurtd|d|r&||jdd||<n||||<t|j|}||j|krD|| |||<q|S)a Prepares the inputs for ONNX Runtime by converting them to numpy arrays with the expected dtype. Args: use_torch (`bool`): Whether the inputs are torch.Tensor or not. inputs (`Dict[str, Union[torch.Tensor, np.ndarray]]`): The inputs to prepare for ONNX Runtime. Returns: `Dict[str, np.ndarray]`: The inputs prepared for ONNX Runtime. NzInput z' is required by model but not provided.T)force) r9keysgetr_numpyr ort_type_to_numpy_typer>rZastype)r@rhrj onnx_inputs input_nameexpected_dtyperrr!_prepare_onnx_inputss z$ORTSessionMixin._prepare_onnx_inputs onnx_outputscCsFi}|jD]\}}||||<|r t|||j||<q|S)a Prepares the outputs from ONNX Runtime by converting them to torch.Tensor if requested. Args: use_torch (`bool`): Whether the outputs should be torch.Tensor or not. onnx_outputs (`List[np.ndarray]`): The outputs from ONNX Runtime. Returns: `Dict[str, Union[torch.Tensor, np.ndarray]]`: The outputs prepared for the user. )r;rcra from_numpyrgrS)r@rhru model_outputs output_namerrrr!_prepare_onnx_outputs s z%ORTSessionMixin._prepare_onnx_outputsrx output_shapecCst|dkr tdtdd|Dstd|dtdd|Ds,td|dt|j|}t|dkrHtjt |||j d }|Stj d||j d }|S) a[ Prepares an output buffer for ONNX Runtime IO Binding. Args: output_name (`str`): The name of the output for which to prepare the buffer. output_shape (`Tuple[int]`): The shape of the output buffer. Returns: `torch.Tensor`: The output buffer. rz"`output_shape` should not be emptycss|]}t|tVqdSN)r]rbrdimrrr! Jsz9ORTSessionMixin._prepare_output_buffer..z4`output_shape` should only contain integers but got .css|]}|dkVqdSrNrr|rrr!r~Lsz=`output_shape` should only contain positive integers but got )rZrS) lenr_allr ort_type_to_torch_typer?raemptynpprodrStensor)r@rxrz output_dtype output_bufferrrr!_prepare_output_buffer:s  z&ORTSessionMixin._prepare_output_bufferknown_axes_valuescCs>t|j|}t|D]\}}t|tr|||||<q |S)a Infers the shape of a given output by using the `known_axes_values` mapping. Args: output_name (`str`): The name of the output for which to infer the shape. known_axes_values (`Dict[str, int]`): A mapping of the axis names to their values. Returns: `List[int]`: The inferred shape of the output. )listr=r7r]r`_dynamic_axis_inference)r@rxrrzr axis_namerrr!_output_shape_inferenceXs  z'ORTSessionMixin._output_shape_inferencercCsV||vr||S|d}t|D]\}}||vr!t||||<qttd|S)au Infers the value of a given dynamic axis by using the `known_axes_values` mapping. For instance, for the following inputs: axis_name = "sequence_length + past_sequence_length" known_axes_values = {"batch_size": 2, "sequence_length": 3, "past_sequence_length": 7} The inferred value will be: 3 + 7 = 10  )splitr7r`rbevaljoin)r@rrtokensrtokenrrr!rns  z'ORTSessionMixin._dynamic_axis_inferenceoutputs_to_not_bindknown_output_buffersknown_output_shapesc Csi}|jD]k}||j}||s||||<||j}t|j|} || kr6|| | ||<|| } | dkrDt } |j ||jj|jjpPdt|j||| t|j|D]\} } t| trq|| || <qbqi} i}|pzi}|p~i}|pt}|jD]H}||vrq||vr||}n|||}||vr||}n|||}| } |j ||jj|jjpdt|j||| |||<|| |<q| |fS)a= Prepares IO binding for ONNX Runtime. Args: model_inputs (`Dict[str, torch.Tensor]`): The inputs to bind to the model. outputs_to_not_bind (`Optional[Set[str]]`, defaults to `None`): The names of the outputs that should not be bound. known_output_buffers (`Optional[Dict[str, str]]`, defaults to `None`): Sometimes we can reuse the same input buffer for the output. This is the case for the output sample in a diffusion pipeline. It is possible to explicitely pass the buffer via this argument. known_output_shapes (`Optional[Dict[str, Tuple[int]]]`, defaults to `None`): It can be hard to infer all the output shapes from the inputs only. For instance for the past key / values. It is possible to explicitely pass the shape via this argument. Returns: `TupleDict[str, Tuple[int]], Dict[str, torch.Tensor]`: A dictionary of the output shapes and a dictionary of the output buffers. r)r9rlr' is_contiguous contiguousrZr rr>rgdata_ptrNON_EMPTY_TENSORr3 bind_inputrSr,indexror7r<r]r`setr;rr bind_outputr?)r@rjrrrrrr input_shape tensor_dtypersrrrr=output_buffersrxrzrrrr!_prepare_io_bindingsf              z#ORTSessionMixin._prepare_io_bindingcOrT)NztThe `forward` method should be implemented in the derived class. Please refer to the documentation for more details.)NotImplementedErrorrVrrr!forwardszORTSessionMixin.forwardcOs|j|i|Sr{)rrVrrr!__call__szORTSessionMixin.__call__save_directorycsrtjddt|jj}t|j}t|}fdd|D}t||t ||D] \}}t||q,dS)z Saves the ONNX Runtime session to the specified directory. Args: save_directory (`Union[str, Path]`): The directory where to save the ONNX Runtime session. T)exist_okcsg|] }t|jqSr)rr)rexternal_data_pathrrr! sz0ORTSessionMixin.save_session..N) osmakedirsrrr-rrshutilcopyzip)r@rrEmodel_save_pathexternal_data_pathsexternal_data_save_pathssrc_pathdst_pathrrr! save_sessions   zORTSessionMixin.save_sessionr{)NNN)+__name__ __module__ __qualname____doc__r rr^rApropertyr`rErFrrJr/rrrOr5rarSsetterrZrrgrir Tensorrndarrayrtryrrbrrrrrrrrrrrrr!r+sz%    >  " ""    `rc@seZdZdZdeefddZeddZeddZ ed d Z ed d Z ed dZ eddZ eddZejdefddZddZdS)ORTParentMixinaC Wrapper class for multiple ORTSessionMixin instances. This class allows to combine multiple parts into a single wrapper. It is useful for pipelines/models that require multiple parts to work together, such as diffusion pipelines or encoder-decoder models, as it provides a unified interface for inference. partscCs8t|dkr tdtdd|Drtd||_dS)z Initializes the ORTParentMixin class. Args: parts (`List[ORTSessionMixin]`): List of ORTSessionMixin instances to wrap. rz.zGAll parts passed to ORTParentMixin should be ORTSessionMixin instances.N)rr_anyr)r@rrrr!rA s  z(ORTParentMixin.initialize_ort_attributesc.tfddjDstdjdjS)rHc3"|] }|jjdjkVqdSr)rJrrrDrr!r~! z+ORTParentMixin.providers..zCalling `ORTParentMixin.providers` when the underlying parts have different values for `providers` is not recommended. The value of the first session will be returned. r)rrr0rCrJrDrrDr!rJ  zORTParentMixin.providerscr)rLc3rr)r/rrrDrr!r~-rz*ORTParentMixin.provider..zCalling `ORTParentMixin.provider` when the underlying parts have different values for `provider` is not recommended. The value of the first session will be returned. r)rrr0rCr/rDrrDr!r/(rzORTParentMixin.providercr)rMc3rr)rOrrrDrr!r~9rz2ORTParentMixin.provider_options..zCalling `ORTParentMixin.provider_options` when the underlying parts have different values for `provider_options` is not recommended. The value of the first session will be returned. r)rrr0rCrOrDrrDr!rO4rzORTParentMixin.provider_optionscr)rPc3rr)r5rrrDrr!r~Erz1ORTParentMixin.provider_option..zCalling `ORTParentMixin.provider_option` when the underlying parts have different values for `provider_option` is not recommended. The value of the first session will be returned. r)rrr0rCr5rDrrDr!r5@rzORTParentMixin.provider_optioncr)rRc3rr)rSrrrDrr!r~Rrz(ORTParentMixin.device..zCalling `ORTParentMixin.device` when the underlying parts have different values for `device` is not recommended. The value of the first session will be returned. r)rrr0rCrSrDrrDr!rSLs  zORTParentMixin.devicecr)rYc3rr)rZrrrDrr!r~`rz'ORTParentMixin.dtype..zCalling `ORTParentMixin.dtype` when the underlying parts have different values for `dtype` is not recommended. The value of the first session will be returned. r)rrr0rCrZrDrrDr!rZYs  zORTParentMixin.dtypecr)r[c3rr)rrrrDrr!r~lrz0ORTParentMixin.use_io_binding..zCalling `ORTParentMixin.use_io_binding` when the underlying parts have different values for `use_io_binding` is not recommended. The value of the first session will be returned. r)rrr0rCrrDrrDr!rgrzORTParentMixin.use_io_bindingr\cCs|jD]}||_qdS)z9 Setter for the use_io_binding property. N)rr)r@r\rrrr!rss cOs |jD] }|j|i|q|S)a Moves all parts to the specified device by updating the execution provider and its options. Args: device (`str`, `int`, `torch.device`): The device to move the session to. It can be a string (e.g., "cuda", "cpu"), an integer (e.g., 0 for GPU 0), or a `torch.device` object. Returns: `ORTParentMixin`: The updated session. Raises: ValueError: If the device is not supported or if the provider is not available. )rrg)r@rWrXrrrr!rg{s zORTParentMixin.toN)rrrrrrrArrJr/rOr5rSrZrrr^rgrrrr!rs(         r)%rrrpathlibrtypingrrrrrrr rnrra onnxruntimer r *onnxruntime.transformers.io_binding_helperr onnx.utilsr utils.loggingrutilsrrrrrrr0rrrrrrrr!s&  $    ]