o [5iF@sdZddlmZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddl mZddlmZmZddlZddlZdd lmZdd lmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)m*Z*m+Z+dd l,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8dd l9m:Z:ddl;mm?Z?ddl@mAZAmBZBddlCmDZDddlEmFZFddlGmHZHmIZIddlJmKZKddlLmMZMmNZNddlOmPZPddlQmRZRddlSmTZTerddlmUZUeVeWZXdZYdZZdZ[dZ\dZ]dZ^d Z_e Gd!d"d"e2Z`Gd#d$d$eKeIZad%Zbe)e\Gd&d'd'eaZcd(Zde)e\Gd)d*d*eaZed+Zfe)e\Gd,d-d-eaZgd.Zhe)e\Gd/d0d0eaZid1Zje)e\Gd2d3d3eaZkd4Zle)e\Gd5d6d6eaZmd7Zne)e\Gd8d9d9eaZoe)e\Gd:d;d;eaZpdd>eaZrd?Zse)e\Gd@dAdAeaZtdBZue)e\GdCdDdDeaZvdEZwe)e\GdFdGdGeaZxdHZye)e\GdIdJdJeaZzdKZ{e)e\GdLdMdMeaZ|dNZ}e)e\GdOdPdPeaZ~dS)QziORTModelForXXX classes, allowing to run ONNX Models with ONNX Runtime using the same API as Transformers.) annotationsN)Sequence) dataclass)Path)TemporaryDirectory) TYPE_CHECKINGAny)HUGGINGFACE_HUB_CACHE) AutoConfig AutoModelAutoModelForAudioClassification$AutoModelForAudioFrameClassificationAutoModelForAudioXVectorAutoModelForCTCAutoModelForImageClassificationAutoModelForImageToImageAutoModelForMaskedLMAutoModelForMultipleChoiceAutoModelForQuestionAnswering AutoModelForSemanticSegmentation"AutoModelForSequenceClassificationAutoModelForTokenClassification'AutoModelForZeroShotImageClassificationGenerationMixin)add_end_docstringsadd_start_docstrings%add_start_docstrings_to_model_forward) BaseModelOutputCausalLMOutputImageClassifierOutputImageSuperResolutionOutputMaskedLMOutput ModelOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSemanticSegmenterOutputSequenceClassifierOutputTokenClassifierOutput XVectorOutput)-MODEL_FOR_SEMANTIC_SEGMENTATION_MAPPING_NAMES) cached_fileis_offline_mode)Self)InferenceSessionSessionOptions) main_export) TasksManager)FROM_PRETRAINED_START_DOCSTRINGOptimizedModel)ORTSessionMixin)ONNX_FILE_PATTERNONNX_WEIGHTS_NAME)&prepare_providers_and_provider_options)find_files_matching_pattern)maybe_save_preprocessors)PretrainedConfig AutoTokenizerAutoFeatureExtractor AutoProcessora2 This model inherits from [`~onnxruntime.modeling_ort.ORTModel`], check its documentation for the generic methods the library implements for all its model (such as downloading or saving). This class should be initialized using the [`onnxruntime.modeling_ort.ORTModel.from_pretrained`] method. a. Args: input_ids (`Union[torch.Tensor, np.ndarray, None]` of shape `({0})`, defaults to `None`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`](https://huggingface.co/docs/transformers/autoclass_tutorial#autotokenizer). See [`PreTrainedTokenizer.encode`](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizerBase.encode) and [`PreTrainedTokenizer.__call__`](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizerBase.__call__) for details. [What are input IDs?](https://huggingface.co/docs/transformers/glossary#input-ids) attention_mask (`Union[torch.Tensor, np.ndarray, None]` of shape `({0})`, defaults to `None`): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](https://huggingface.co/docs/transformers/glossary#attention-mask) token_type_ids (`Union[torch.Tensor, np.ndarray, None]` of shape `({0})`, defaults to `None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 1 for tokens that are **sentence A**, - 0 for tokens that are **sentence B**. [What are token type IDs?](https://huggingface.co/docs/transformers/glossary#token-type-ids) aj Args: pixel_values (`Union[torch.Tensor, np.ndarray, None]` of shape `({0})`, defaults to `None`): Pixel values corresponding to the images in the current batch. Pixel values can be obtained from encoded images using [`AutoFeatureExtractor`](https://huggingface.co/docs/transformers/autoclass_tutorial#autofeatureextractor). a; Args: input_values (`torch.Tensor` of shape `({0})`): Float values of input raw speech waveform.. Input values can be obtained from audio file loaded into an array using [`AutoFeatureExtractor`](https://huggingface.co/docs/transformers/autoclass_tutorial#autofeatureextractor). c@sLeZdZUdZdZded<dZded<dZded<dZded<d d d Z dS) !ZeroShotImageClassificationOutputaMlogits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`MetaClip2TextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`MetaClip2VisionModel`]. Nztorch.FloatTensorlogits_per_imagelogits_per_text text_embeds image_embedsreturn tuple[Any]cstfddDS)Nc3s|]}|VqdSN).0kselfrEj/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/optimum/onnxruntime/modeling.py sz=ZeroShotImageClassificationOutput.to_tuple..)tuplekeysrHrErHrJto_tuplesz*ZeroShotImageClassificationOutput.to_tuple)rBrC) __name__ __module__ __qualname____doc__r>__annotations__r?r@rArNrErErErJr=s    r=cseZdZUdZdZeZdZded<ddddddKfddZ dLddZ e ddddde fdMd%d&Z e dNdOd,d-Zeddddde ddd.ddddejdfdPd;d<Zeddddde dfdQd=d>Zeeeddddddde dd.ddddfdRfdBdC ZdSdDdEZdTdIdJZZS)UORTModelaBase class for implementing models using ONNX Runtime. The ORTModel implements generic methods for interacting with the Hugging Face Hub as well as exporting vanilla transformers models to ONNX using `optimum.exporters.onnx` toolchain. Class attributes: - model_type (`str`, *optional*, defaults to `"onnx_model"`) -- The name of the model type to use when registering the ORTModel classes. - auto_model_class (`Type`, *optional*, defaults to `AutoModel`) -- The "AutoModel" class to represented by the current ORTModel class. Args: - config ([`~transformers.PretrainedConfig`] -- The configuration of the model. - session (`~onnxruntime.InferenceSession`) -- The ONNX Runtime InferenceSession that is running the model. - use_io_binding (`bool`, *optional*, defaults to `True`) -- Whether to use I/O bindings with **ONNX Runtime with the CUDAExecutionProvider**, this can significantly speedup inference depending on the task. - model_save_dir (`Path`) -- The directory where the model exported to ONNX is saved. By defaults, if the loaded model is local, the directory where the original model will be used. Otherwise, the cache directory is used. onnx_modelN str | None _library_nameconfigsessionuse_io_bindingmodel_save_dirrYr9rZr-r[ bool | Noner\&str | Path | TemporaryDirectory | Nonecstj||d|j||dd|_|durt|jj|_nt|t r-||_t|j |_nt|t r8t||_n||_t |jt t|jdrR|j t |jdSdS)N)modelrY)rZr[register)super__init__initialize_ort_attributes&_model_save_dir_tempdirectory_instancer _model_pathparentr\ isinstancernamestrr r` model_typehasattrauto_model_class __class__)rIrYrZr[r\rmrErJrbs    zORTModel.__init__save_directory str | PathcCs||dS)aSaves the underlying ONNX model and its external data files (if any) to the specified directory. This method is called by the `save_pretrained` method of the `OptimizedModel` class. The model is copied from `self.session._model_path` to `save_directory`. Args: save_directory (`Union[str, Path]`): Directory where to save the model file. N) save_session)rIrorErErJ_save_pretraineds zORTModel._save_pretrainedmainFpath_or_repo_idfilenameri subfolderrevisionforce_downloadboollocal_files_onlytokenbool | str | None cache_dirrBrc Cslt||||||||d}ttt||d||||||dWdt|S1s-wYt|S)zMReturns the path to the cached file and downloads external data if necessary.)rvrwrxr~ryr{r|_dataN)r* contextlibsuppressOSErrorr) rurvrwrxryr{r|r~ cached_pathrErErJ _cached_files2     zORTModel._cached_filepattern onnx_files list[Path]standard_file_nametarget_file_namecsVdur;fdd|D}t|dkrtdd|dt|dkr(|dStd d |djd |dSfd d|D}t|dkrWtd d|dnt|dkra|dStdd |djd|dSfdd|D}t|dkrtd|ddt|dkr|dStdd |djd|dS)aInfer the ONNX file path to load among a list of candidates. The inference is done in three steps: 1. We look for the `target_file_name` if provided. 2. Otherwise, we look for the `standard_file_name`. 3. Finally, we look for files matching the `pattern`. In case of multiple matches, the first one is returned and a warning is logged. Args: pattern (`str`): The regex pattern that the target ONNX file should match. onnx_files (`List[Path]`): List of candidate ONNX files. standard_file_name (`str`): The standard file name to look for. target_file_name (`str`, *optional*): The target file name to look for first. Returns: `Path`: The inferred ONNX file path. Raises: `FileNotFoundError`: If no ONNX file is found with the target file name when provided. Ncg|] }|jkr|qSrErhrFfile)rrErJ "z-ORTModel._infer_file_path..rz4Could not find any ONNX files with target file name z, only found: z|. Please make sure that the `file_name` and/or `subfolder` arguments correspond to an ONNX file in the repository/directory.z0Found multiple ONNX files with target file name z, using z. Please make sure to pass a `subfolder` argument to `from_pretrained` when loading from a repository/directory containing multiple ONNX files with the same target file name.crrErr)rrErJr3rz6Could not find any ONNX files with standard file name z, files found: z. Please make sure to pass a `file_name` and/or `subfolder` argument to `from_pretrained` when loading an ONNX file with non-standard file names.z2Found multiple ONNX files with standard file name z. Please make sure to pass a `subfolder` argument to `from_pretrained` when loading from a repository/directory containing multiple standard-named ONNX files in different subfolders.cs g|] }tt|r|qSrE)researchri)rFp)rrErJrE z=Could not find any ONNX model file in the list of candidates z matching the pattern z. Please make sure to pass a `file_name` and/or `subfolder` argument to `from_pretrained` when loading from a repository/directory containing non-pattern-matching ONNX files.z/Found multiple ONNX files matching the pattern z. Please make sure to pass a `file_name` and/or `subfolder` argument to `from_pretrained` when loading from a repository/directory containing multiple pattern-matching ONNX files.)lenFileNotFoundErrorloggerwarningrh)rrrr target_filesstandard_files pattern_filesrE)rrrrJ_infer_file_pathsD      zORTModel._infer_file_pathCPUExecutionProvidermodel_idtrust_remote_code file_nameprovider providersSequence[str] | Noneprovider_options0Sequence[dict[str, Any]] | dict[str, Any] | Nonesession_optionsSessionOptions | Nonedtype torch.dtypec sttd||| d}t|dkrtdtr&fdd|D}|jt|| td}|j|j |j ||||| d}|durG|j }t | | | d \} } t || | |d }|||||d S) N **/*.onnx) glob_patternrwrxr|rz&Could not find any ONNX model file in csg|]}|qSrE) relative_to)rFfrrErJr{sz-ORTModel._from_pretrained..)rrr)rvrwr{ryr~rxr|)rrr)rr sess_optionsrX)r7r4rrris_dirrr5rrhrfas_posixr6r-)clsrrYrwrxryr{rr~r|rrrrrr[rr\r file_path model_pathrZrErrJ_from_pretrainedVsZ    zORTModel._from_pretrainedc Kstj|jd} | ddurtd|jd| dt} t| j} t || | dd|||| ||||j d t || |d |j | |fd | i| S) N) model_classtaskz?The `task` argument is not needed when exporting a model with `z;`. The `task` is automatically inferred from the class as `z`.FT) model_name_or_pathoutputr do_validationno_post_processrwrxr~r|r{ryr library_name) src_subfolderr\) r0%_infer_task_from_model_or_model_classrlget ValueErrorrOrrrhr/rWr8r)rrrYrwrxryr{rr~r|kwargsrr\model_save_pathrErErJ_exports4  zORTModel._exportPretrainedConfig | Noneexportr,c st|tr |}tr|stdd}|}zs|r^tj|s^| dd}tj | d|}tj tj |d|p Parameters for decoder models (ORTModelForCausalLM, ORTModelForSeq2SeqLM, ORTModelForSeq2SeqLM, ORTModelForSpeechSeq2Seq, ORTModelForVision2Seq) use_cache (`Optional[bool]`, defaults to `True`): Whether or not past key/values cache should be used. Defaults to `True`. use_merged (`Optional[bool]`, defaults to `None`): whether or not to use a single ONNX that handles both the decoding without and with past key values reuse. This option defaults to `True` if loading from a local repository and a merged decoder is found. When exporting with `export=True`, defaults to `False`. This option should be set to `True` to minimize memory usage. Returns: `ORTModel`: The loaded ORTModel model. z-Offline mode: setting `local_files_only=True`T/z--zmodels--refsrtN snapshotsr)rrrwr|rxrz The model z was already converted to ONNX but got `export=True`, the model will be converted to ONNX once again. Don't forget to save the resulting model with `.save_pretrained()`zNo ONNX files were found for zx, setting `export=True` to convert the model to ONNX. Don't forget to save the resulting model with `.save_pretrained()`zXCould not infer whether the model was already converted or not to ONNX, keeping `export=z`. rz`file_name` was set to `z<` but will be ignored as the model will be converted to ONNX)rYrryr|r~rwr{rrxrrrrr[)rgrrr+rinfoospathisdirreplacejoinopenreadr7r4rr Exceptionpoprafrom_pretrained)rrrYrrwrxryr{rr~r|rrrrr[rr object_idcached_model_dir refs_filer _revisionr exceptionrrnrErJrs 8          zORTModel.from_pretrainedcCs t|tS)zEReturns whether this model can generate sequences with `.generate()`.)rgrrHrErErJ can_generateRs zORTModel.can_generaterdict[str, Any]NonecCs0|rt|jjdd|ddSdS)zzWarn about unhandled input arguments. Args: kwargs: Dictionary of unhandled input arguments. z received z, a, but do not handle those arguments. Please use `ORTModelForCustomTasks` if your model takes/returns arbitrary or custom tensor inputs/outputs. Or open an issue/PR in optimum repository (https://github.com/huggingface/optimum) if this argument needs to be supported in this class.N)rrrmrOrrM)rIrrErErJ_warn_on_unhandled_inputsVs z"ORTModel._warn_on_unhandled_inputs)rYr9rZr-r[r]r\r^)rorp)rurprvrirwrirxriryrzr{rzr|r}r~rirBrrD) rrirrrrirrVrBr)$rrprYr9rwrirxriryrzr{rzrrzr~rir|r}rrVrrirrrrrrr[r]rrr\r^rBrT)rrprYr9rwrirxriryrzr{rzrrzr~rir|r}rBrT) rrprYrrrzrwrirxriryrzr{rzrrzr~rir|r}rrirrrrrrr[r]rBr,)rBrz)rrrBr)rOrPrQrRrjr rlrWrSrbrr staticmethodr rr classmethodtorchfloat32rrrr1rrr __classcell__rErErnrJrTs    "SM/ |rTa! Example of feature extraction: ```python >>> from transformers import {processor_class} >>> from optimum.onnxruntime import {model_class} >>> import torch >>> tokenizer = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> inputs = tokenizer("My name is Philipp and I live in Germany.", return_tensors="np") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> list(last_hidden_state.shape) [1, 12, 384] ``` Example using `transformers.pipeline`: ```python >>> from transformers import {processor_class}, pipeline >>> from optimum.onnxruntime import {model_class} >>> tokenizer = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> onnx_extractor = pipeline("feature-extraction", model=model, tokenizer=tokenizer) >>> text = "My name is Philipp and I live in Germany." >>> pred = onnx_extractor(text) ``` c@sfeZdZUdZeZdZded<ee de j e ddd          dd d dddZ dS)ORTModelForFeatureExtractionz'ONNX Model for feature-extraction task. transformersrVrWbatch_size, sequence_lengthzoptimum/all-MiniLM-L6-v2processor_classr checkpointNT return_dict input_ids torch.Tensor | np.ndarray | Noneattention_masktoken_type_ids position_ids pixel_values visual_embedsvisual_attention_maskvisual_token_type_idsinput_features input_valuesrrzc  Ks6|| ||||||||| | g } ttdd| }t|tj}|||dur:d|jvr:|r5t|nt |}||||||||| | d }|j rz| |\}}|j j dkr_|j|jn|j|j|j|j|d|d}n|||}|jd|}|||}|d}| s|fSt|dS)NcSs|duSrDrE)xrErErJsz6ORTModelForFeatureExtraction.forward..r) rrrrrrrrrrcpulast_hidden_state)r)rnextfilterrgrTensorraise_on_numpy_input_io_binding input_names zeros_likenpr[_prepare_io_bindingdevicetyperZrun_with_iobinding _io_bindingsynchronize_inputssynchronize_outputsview_prepare_onnx_inputsrun_prepare_onnx_outputsr)rIrrrrrrrrrrrr tensor_inputs first_tensor use_torch model_inputs output_shapesoutput_buffersr onnx_inputs onnx_outputs model_outputsrErErJforwardsV          z$ORTModelForFeatureExtraction.forward) NNNNNNNNNN)rrrrrrrrrrrrrrrrrrrrrrz)rOrPrQrRr rlrWrSrONNX_TEXT_INPUTS_DOCSTRINGformatFEATURE_EXTRACTION_EXAMPLE_TOKENIZER_FOR_DOCrrErErErJrs4    ra Example of feature extraction: ```python >>> from transformers import {processor_class} >>> from optimum.onnxruntime import {model_class} >>> import torch >>> tokenizer = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> inputs = tokenizer("The capital of France is [MASK].", return_tensors="np") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 8, 28996] ``` Example using `transformers.pipeline`: ```python >>> from transformers import {processor_class}, pipeline >>> from optimum.onnxruntime import {model_class} >>> tokenizer = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> fill_masker = pipeline("fill-mask", model=model, tokenizer=tokenizer) >>> text = "The capital of France is [MASK]." >>> pred = fill_masker(text) ``` c@XeZdZUdZeZdZded<ee de j e ddd   dd d dddZ dS)ORTModelForMaskedLMaONNX Model with a MaskedLMOutput for masked language modeling tasks. This class officially supports albert, bert, camembert, convbert, data2vec-text, deberta, deberta_v2, distilbert, electra, flaubert, ibert, mobilebert, roberta, roformer, squeezebert, xlm, xlm_roberta.rrVrWrz'optimum/bert-base-uncased-for-fill-maskrNTrrrrrrrzcKs||t|tj}||d|jvr%|dur%|r t|nt|}|||d}|jr^| |\}} |j j dkrC|j |jn|j|j |j|j| d|d} n|||} |j d| } ||| } | d} |sz| fSt| dSNrrrrrlogitsr)rrgrrrrrrr[rrrrZrrrrrrr r r!rIrrrrrr rrrrrrrrErErJr s0         zORTModelForMaskedLM.forwardNNNrrrrrrrrz)rOrPrQrRrrlrWrSrrrMASKED_LM_EXAMPLErrrErErErJr&   ra Example of question answering: ```python >>> from transformers import {processor_class} >>> from optimum.onnxruntime import {model_class} >>> import torch >>> tokenizer = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" >>> inputs = tokenizer(question, text, return_tensors="np") >>> start_positions = torch.tensor([1]) >>> end_positions = torch.tensor([3]) >>> outputs = model(**inputs, start_positions=start_positions, end_positions=end_positions) >>> start_scores = outputs.start_logits >>> end_scores = outputs.end_logits ``` Example using `transformers.pipeline`: ```python >>> from transformers import {processor_class}, pipeline >>> from optimum.onnxruntime import {model_class} >>> tokenizer = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> onnx_qa = pipeline("question-answering", model=model, tokenizer=tokenizer) >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" >>> pred = onnx_qa(question, text) ``` c@r)ORTModelForQuestionAnsweringaNONNX Model with a QuestionAnsweringModelOutput for extractive question-answering tasks like SQuAD. This class officially supports albert, bart, bert, camembert, convbert, data2vec-text, deberta, deberta_v2, distilbert, electra, flaubert, gptj, ibert, mbart, mobilebert, nystromformer, roberta, roformer, squeezebert, xlm, xlm_roberta.rrVrWrzoptimum/roberta-base-squad2rNTrrrrrrrzcKs||t|tj}|||dur%d|jvr%|r t|nt|}|||d}|jrg| |\}} |j j dkrC|j |jn|j|j |j|j| d|d} | d|d} n|||} |j d| } ||| }|d} |d} |s| | fSt| | dS)Nrrr start_logits end_logits)r%r&)rrgrrrrrrr[rrrrZrrrrrrr r r$)rIrrrrrr rrrr%r&rrrrErErJros4         z$ORTModelForQuestionAnswering.forwardr r!)rOrPrQrRrrlrWrSrrrQUESTION_ANSWERING_EXAMPLErrrErErErJr$hr#r$a Example of single-label classification: ```python >>> from transformers import {processor_class} >>> from optimum.onnxruntime import {model_class} >>> import torch >>> tokenizer = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="np") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 2] ``` Example using `transformers.pipelines`: ```python >>> from transformers import {processor_class}, pipeline >>> from optimum.onnxruntime import {model_class} >>> tokenizer = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> onnx_classifier = pipeline("text-classification", model=model, tokenizer=tokenizer) >>> text = "Hello, my dog is cute" >>> pred = onnx_classifier(text) ``` Example using zero-shot-classification `transformers.pipelines`: ```python >>> from transformers import {processor_class}, pipeline >>> from optimum.onnxruntime import {model_class} >>> tokenizer = {processor_class}.from_pretrained("optimum/distilbert-base-uncased-mnli") >>> model = {model_class}.from_pretrained("optimum/distilbert-base-uncased-mnli") >>> onnx_z0 = pipeline("zero-shot-classification", model=model, tokenizer=tokenizer) >>> sequence_to_classify = "Who are you voting for in 2020?" >>> candidate_labels = ["Europe", "public health", "politics", "elections"] >>> pred = onnx_z0(sequence_to_classify, candidate_labels, multi_label=True) ``` c@r)!ORTModelForSequenceClassificationaqONNX Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. This class officially supports albert, bart, bert, camembert, convbert, data2vec-text, deberta, deberta_v2, distilbert, electra, flaubert, ibert, mbart, mobilebert, nystromformer, roberta, roformer, squeezebert, xlm, xlm_roberta. rrVrWrz7optimum/distilbert-base-uncased-finetuned-sst-2-englishrNTrrrrrrrzcK||t|tj}|||dur%d|jvr%|r t|nt|}|||d}|jr^| |\}} |j j dkrC|j |jn|j|j |j|j| d|d} n|||} |j d| } ||| } | d} |sz| fSt| dSr)rrgrrrrrrr[rrrrZrrrrrrr r r&rrErErJr0         z)ORTModelForSequenceClassification.forwardr r!)rOrPrQrRrrlrWrSrrrSEQUENCE_CLASSIFICATION_EXAMPLErrrErErErJr(&   r(a Example of token classification: ```python >>> from transformers import {processor_class} >>> from optimum.onnxruntime import {model_class} >>> import torch >>> tokenizer = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> inputs = tokenizer("My name is Philipp and I live in Germany.", return_tensors="np") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 12, 9] ``` Example using `transformers.pipelines`: ```python >>> from transformers import {processor_class}, pipeline >>> from optimum.onnxruntime import {model_class} >>> tokenizer = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> onnx_ner = pipeline("token-classification", model=model, tokenizer=tokenizer) >>> text = "My name is Philipp and I live in Germany." >>> pred = onnx_ner(text) ``` c@r)ORTModelForTokenClassificationavONNX Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. This class officially supports albert, bert, bloom, camembert, convbert, data2vec-text, deberta, deberta_v2, distilbert, electra, flaubert, gpt2, ibert, mobilebert, roberta, roformer, squeezebert, xlm, xlm_roberta. rrVrWrzoptimum/bert-base-NERrNTrrrrrrrzcKr)r)rrgrrrrrrr[rrrrZrrrrrrr r r'rrErErJrJs0         z&ORTModelForTokenClassification.forwardr r!)rOrPrQrRrrlrWrSrrrTOKEN_CLASSIFICATION_EXAMPLErrrErErErJr-@s&   r-a Example of multiple choice: ```python >>> from transformers import {processor_class} >>> from optimum.onnxruntime import {model_class} >>> tokenizer = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}", export=True) >>> num_choices = 4 >>> first_sentence = ["Members of the procession walk down the street holding small horn brass instruments."] * num_choices >>> second_sentence = [ ... "A drum line passes by walking down the street playing their instruments.", ... "A drum line has heard approaching them.", ... "A drum line arrives and they're outside dancing and asleep.", ... "A drum line turns the lead singer watches the performance." ... ] >>> inputs = tokenizer(first_sentence, second_sentence, truncation=True, padding=True) # Unflatten the inputs values expanding it to the shape [batch_size, num_choices, seq_length] >>> for k, v in inputs.items(): ... inputs[k] = [v[i: i + num_choices] for i in range(0, len(v), num_choices)] >>> inputs = dict(inputs.convert_to_tensors(tensor_type="pt")) >>> outputs = model(**inputs) >>> logits = outputs.logits ``` c@r)ORTModelForMultipleChoiceapONNX Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. This class officially supports albert, bert, camembert, convbert, data2vec-text, deberta_v2, distilbert, electra, flaubert, ibert, mobilebert, nystromformer, roberta, roformer, squeezebert, xlm, xlm_roberta. rrVrWrz!ehdwns1516/bert-base-uncased_SWAGrNTrrrrrrrzcKr)r)rrgrrrrrrr[rrrrZrrrrrrr r r#rrErErJrr*z!ORTModelForMultipleChoice.forwardr r!)rOrPrQrRrrlrWrSrrrMULTIPLE_CHOICE_EXAMPLErrrErErErJr/r,r/a Example of image classification: ```python >>> import requests >>> from PIL import Image >>> from optimum.onnxruntime import {model_class} >>> from transformers import {processor_class} >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> preprocessor = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> inputs = preprocessor(images=image, return_tensors="np") >>> outputs = model(**inputs) >>> logits = outputs.logits ``` Example using `transformers.pipeline`: ```python >>> import requests >>> from PIL import Image >>> from transformers import {processor_class}, pipeline >>> from optimum.onnxruntime import {model_class} >>> preprocessor = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> onnx_image_classifier = pipeline("image-classification", model=model, feature_extractor=preprocessor) >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> pred = onnx_image_classifier(url) ``` c@sBeZdZdZeZeede je ddddddd d Z d S)ORTModelForImageClassificationzONNX Model for image-classification tasks. This class officially supports beit, convnext, convnextv2, data2vec-vision, deit, dinov2, levit, mobilenet_v1, mobilenet_v2, mobilevit, poolformer, resnet, segformer, swin, swinv2, vit.'batch_size, num_channels, height, widthzoptimum/vit-base-patch16-224rTrrtorch.Tensor | np.ndarrayrrzc K||t|tj}||d|i}|jrG||\}}|jjdkr,|j |j n|j |j |j |j |d|d}n|||} |j d| } ||| } | d}|sc|fSt|dS)Nrrrr)rrgrrrr[rrrrZrrrrrrr r r) rIrrrr rrrrrrrrErErJrs(         z&ORTModelForImageClassification.forwardNrr3rrz) rOrPrQrRrrlrONNX_IMAGE_INPUTS_DOCSTRINGrIMAGE_CLASSIFICATION_EXAMPLE_FEATURE_EXTRACTOR_FOR_DOCrrErErErJr1 s r1c@s:eZdZdZeZeede d d d d d Z dS)&ORTModelForZeroShotImageClassificationzeONNX Model for zero-shot-image-classification tasks. This class officially supports clip, metaclip-2.rr2Nrr3rrrcKsH||t|tj}|||dur%d|jvr%|r t|nt|}|||d}|jry| |\}}|j j dkrC|j |jn|j|j |j|j|d|d} |d|d} |d|d} |d|d} n#|||} |j d| }|||}|d} |d} |d} |d} t| | | | dS) Nr)rrrrr?r>r@rA)r?r>r@rA)rrgrrrr ones_likerr[rrrrZrrrrrrr r r=)rIrrrrr rrrr?r>r@rArrrrErErJrGsB       z.ORTModelForZeroShotImageClassification.forwardrD)rr3rr3rr) rOrPrQrRrrlrrrr6rrErErErJr9Asr9a Example of semantic segmentation: ```python >>> import requests >>> from PIL import Image >>> from optimum.onnxruntime import {model_class} >>> from transformers import {processor_class} >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> preprocessor = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> inputs = preprocessor(images=image, return_tensors="np") >>> outputs = model(**inputs) >>> logits = outputs.logits ``` Example using `transformers.pipeline`: ```python >>> import requests >>> from PIL import Image >>> from transformers import {processor_class}, pipeline >>> from optimum.onnxruntime import {model_class} >>> preprocessor = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> onnx_image_segmenter = pipeline("image-segmentation", model=model, feature_extractor=preprocessor) >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> pred = onnx_image_segmenter(url) ``` csReZdZdZeZfddZee de j e ddddddd dZ Z S)ORTModelForSemanticSegmentationzONNX Model for semantic-segmentation, with an all-MLP decode head on top e.g. for ADE20k, CityScapes. This class officially supports maskformer, segformer.cs"tj|i||jjtd<dS)Nort_semantic_segmentation)rarbrmrOr))rIargsrrnrErJrbsz(ORTModelForSemanticSegmentation.__init__r2z*optimum/segformer-b0-finetuned-ade-512-512rTrrr3rrzc Ks&||t|tj}||d|i}|jrV||\}}|jjdkr,|j |j n|j |j |j |j |d|d}d} d|vrU|d|d} n!|||} |j d| } ||| } | d}d} d| vrw| d} |s| dur|| fS|fS| durt|| dSt|dS)Nrrr pred_masks)rr>r)rrgrrrr[rrrrZrrrrrrr r r"r%) rIrrrr rrrrr>rrrrErErJrs>          z'ORTModelForSemanticSegmentation.forwardr5)rOrPrQrRrrlrbrr6rSEMANTIC_SEGMENTATION_EXAMPLEr8rrrErErnrJr;s   r;a Example of audio classification: ```python >>> from transformers import {processor_class} >>> from optimum.onnxruntime import {model_class} >>> from datasets import load_dataset >>> import torch >>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation") >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> feature_extractor = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> # audio file is decoded on the fly >>> inputs = feature_extractor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt") >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_ids = torch.argmax(logits, dim=-1).item() >>> predicted_label = model.config.id2label[predicted_class_ids] ``` Example using `transformers.pipeline`: ```python >>> from transformers import {processor_class}, pipeline >>> from optimum.onnxruntime import {model_class} >>> feature_extractor = {processor_class}.from_pretrained("{checkpoint}") >>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation") >>> dataset = dataset.sort("id") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> onnx_ac = pipeline("audio-classification", model=model, feature_extractor=feature_extractor) >>> pred = onnx_ac(dataset[0]["audio"]["array"]) ``` c@r)ORTModelForAudioClassificationaIONNX Model for audio-classification, with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. This class officially supports audio_spectrogram_transformer, data2vec-audio, hubert, sew, sew-d, unispeech, unispeech_sat, wavlm, wav2vec2, wav2vec2-conformer. rrVrWrzoptimum/hubert-base-superb-ksrNTrrrrrrrzcKs|||jjdkr|durtdd}|}n|dur&td|jjdd}|}t|tj}||||d|i} |jrn| | \} } |j j dkrS|j |jn|j|j |j|j| d | d } n||| } |j d| }|||}|d } |s| fSt| d S) Nwhisperz8input_features must be provided for this model (whisper)rz.input_values must be provided for this model ()rrrrr)rrYrjrrgrrrr[rrrrZrrrrrrr r r&)rIrrrrr input_name model_inputr rrrrrrrrErErJr(s<          z&ORTModelForAudioClassification.forwardr )rrrrrrrrz)rOrPrQrRr rlrWrSrONNX_AUDIO_INPUTS_DOCSTRINGrAUDIO_CLASSIFICATION_EXAMPLEr8rrErErErJr@r,r@a Example of CTC: ```python >>> from transformers import {processor_class}, HubertForCTC >>> from optimum.onnxruntime import {model_class} >>> from datasets import load_dataset >>> import torch >>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation") >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> processor = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> # audio file is decoded on the fly >>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt") >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_ids = torch.argmax(logits, dim=-1) >>> transcription = processor.batch_decode(predicted_ids) ``` c@sVeZdZUdZeZdZded<ee de j e ddd  dd d dddZ dS)ORTModelForCTCzONNX Model with a language modeling head on top for Connectionist Temporal Classification (CTC). This class officially supports data2vec-audio, hubert, sew, sew-d, unispeech, unispeech_sat, wavlm, wav2vec2, wav2vec2-conformer.rrVrWrzoptimum/hubert-large-ls960-ftrNTrrrrrrzcKs|jjdkr|dusJdd}|}n |dusJdd}|}||t|tj}||||i}|jrf||\} } |j j dkrK|j |j n|j |j |j |j | d| d} n|||} |j d| } ||| }|d} |s| fSt| dS) Nmctctz.input_features must be provided for this modelrz,input_values must be provided for this modelrrrr)rYrjrrgrrrr[rrrrZrrrrrrr r r)rIrrrrrCrDr rrrrrrrrErErJrs6          zORTModelForCTC.forward)NN)rrrrrrz)rOrPrQrRrrlrWrSrrEr CTC_EXAMPLE_PROCESSOR_FOR_DOCrrErErErJrGs$   rGa Example of Audio XVector: ```python >>> from transformers import {processor_class} >>> from optimum.onnxruntime import {model_class} >>> from datasets import load_dataset >>> import torch >>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation") >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> feature_extractor = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> # audio file is decoded on the fly >>> inputs = feature_extractor( ... [d["array"] for d in dataset[:2]["audio"]], sampling_rate=sampling_rate, return_tensors="pt", padding=True ... ) >>> with torch.no_grad(): ... embeddings = model(**inputs).embeddings >>> embeddings = torch.nn.functional.normalize(embeddings, dim=-1).cpu() >>> cosine_sim = torch.nn.CosineSimilarity(dim=-1) >>> similarity = cosine_sim(embeddings[0], embeddings[1]) >>> threshold = 0.7 >>> if similarity < threshold: ... print("Speakers are not the same!") >>> round(similarity.item(), 2) ``` c@TeZdZUdZeZdZded<ee de j e ddd dd d dddZ dS)ORTModelForAudioXVectorzONNX Model with an XVector feature extraction head on top for tasks like Speaker Verification. This class officially supports data2vec-audio, unispeech_sat, wavlm, wav2vec2, wav2vec2-conformer.rrVrWrzoptimum/wav2vec2-base-superb-svrNTrrrrrzc Ks||t|tj}||d|i}|jrP||\}}|jjdkr,|j |j n|j |j |j |j |d|d}|d|d} n|||} |j d| } ||| } | d}| d} |sq|| fSt|| dS)Nrrr embeddings)rrM)rrgrrrr[rrrrZrrrrrrr r r() rIrrrr rrrrrMrrrrErErJrs,         zORTModelForAudioXVector.forwardrDrrrrz)rOrPrQrRrrlrWrSrrErAUDIO_XVECTOR_EXAMPLEr8rrErErErJrL"   rLa Example of audio frame classification: ```python >>> from transformers import {processor_class} >>> from optimum.onnxruntime import {model_class} >>> from datasets import load_dataset >>> import torch >>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation") >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> feature_extractor = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> inputs = feature_extractor(dataset[0]["audio"]["array"], return_tensors="pt", sampling_rate=sampling_rate) >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> probabilities = torch.sigmoid(logits[0]) >>> labels = (probabilities > 0.5).long() >>> labels[0].tolist() ``` c@rK)#ORTModelForAudioFrameClassificationzONNX Model with a frame classification head on top for tasks like Speaker Diarization. This class officially supports data2vec-audio, unispeech_sat, wavlm, wav2vec2, wav2vec2-conformer.rrVrWrzoptimum/wav2vec2-base-superb-sdrNTrrrrrzc Kr4)Nrrrr)rrgrrrr[rrrrZrrrrrrr r r') rIrrrr rrrrrrrrErErJrGs(         z+ORTModelForAudioFrameClassification.forwardrDrN)rOrPrQrRr rlrWrSrrEr"AUDIO_FRAME_CLASSIFICATION_EXAMPLEr8rrErErErJrQ@rPrQa Example of image-to-image (Super Resolution): ```python >>> from transformers import {processor_class} >>> from optimum.onnxruntime import {model_class} >>> from PIL import Image >>> image = Image.open("path/to/image.jpg") >>> image_processor = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> inputs = image_processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... logits = model(**inputs).logits ``` c@sPeZdZUdZeZdZded<ee de j e ddddd dddZ dS)ORTModelForImageToImageztONNX Model for image-to-image tasks. This class officially supports pix2pix, cyclegan, wav2vec2, wav2vec2-conformer.rrVrWr2z-caidas/swin2SR-realworld-sr-x4-64-bsrgan-psnrrTrrr3rrzc Kr4)Nrrreconstruction)rT)rrgrrrr[rrrrZrrrrrrr r r ) rIrrrr rrrrTrrrrErErJrs(         zORTModelForImageToImage.forwardNr5)rOrPrQrRrrlrWrSrr6rIMAGE_TO_IMAGE_EXAMPLErJrrErErErJrSs   rSa* Example of custom tasks(e.g. a sentence transformers taking `pooler_output` as output): ```python >>> from transformers import {processor_class} >>> from optimum.onnxruntime import {model_class} >>> tokenizer = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> inputs = tokenizer("I love burritos!", return_tensors="np") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooler_output = outputs.pooler_output ``` Example using `transformers.pipelines`(only if the task is supported): ```python >>> from transformers import {processor_class}, pipeline >>> from optimum.onnxruntime import {model_class} >>> tokenizer = {processor_class}.from_pretrained("{checkpoint}") >>> model = {model_class}.from_pretrained("{checkpoint}") >>> onnx_extractor = pipeline("feature-extraction", model=model, tokenizer=tokenizer) >>> text = "I love burritos!" >>> pred = onnx_extractor(text) ``` c@s.eZdZdZeejedddd ddZdS) ORTModelForCustomTaskszONNX Model for any custom tasks. It can be used to leverage the inference acceleration for any single-file ONNX model, that may use custom inputs and outputs.z'optimum/sbert-all-MiniLM-L6-with-poolerrrr3c sttt|tj}|||jrF||\}|j j dkr)|j |j n|j |j |j |j fdd|D}n|||}|j d|}|||}tdi|S)Nrcs i|] \}}|||qSrE)r)rFrhshaperrErJ rz2ORTModelForCustomTasks.forward..rE)rgritervaluesrrrr[rrrrZrrrritemsrr r r")rIrr rrrrrErXrJrs      zORTModelForCustomTasks.forwardN)rr3) rOrPrQrRrCUSTOM_TASKS_EXAMPLErrrrErErErJrVsrV)rR __future__rrloggingrrcollections.abcr dataclassesrpathlibrtempfilertypingrrnumpyrrhuggingface_hub.constantsr rr r r r rrrrrrrrrrrrtransformers.file_utilsrrrtransformers.modeling_outputsrrrr r!r"r#r$r%r&r'r(&transformers.models.auto.modeling_autor)transformers.utilsr*r+typing_extensionsr, onnxruntimer-r.optimum.exporters.onnxr/optimum.exporters.tasksr0optimum.modeling_baser1r2optimum.onnxruntime.baser3optimum.onnxruntime.constantsr4r5optimum.onnxruntime.utilsr6optimum.utils.file_utilsr7optimum.utils.save_utilsr8r9 getLoggerrOrrr8rJONNX_MODEL_END_DOCSTRINGrr6rEr=rTrrr"rr'r$r+r(r.r-r0r/r7r1r9r?r;rFr@rIrGrOrLrRrQrUrSr]rVrErErErJs      H8          K#Z#>$@2A#AA'7?'K+JA#:76!