o dii[@sjdZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZddlZddlmZmZddlmZddlmZdd lmZmZmZmZdd lmZdd l m!Z!dd l"m#Z#d dl$m%Z%d dl&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3e rddl4m5Z5ddlm6Z6e7e8Z9GdddeZ:Gddde%Z;dS)z0Classes handling quantization with ONNX Runtime.N) defaultdict)Path) TYPE_CHECKINGCallableDictListOptionalTupleUnion)Versionparse) AutoConfig) __version__)CalibrationDataReader QuantFormatQuantizationMode QuantType) ONNXQuantizer) QDQQuantizer)requires_backends)OptimumQuantizer)maybe_save_preprocessors)ORTQuantizableOperator)CalibrationConfig ORTConfigQuantizationConfig)ORTModel) ORTModelForConditionalGeneration)QuantizationPreprocessor)Dataset)PretrainedConfigc@s0eZdZgdZd dddefddZdd Zd S) ORTCalibrationDataReader) batch_sizedataset _dataset_iterrr%r!r$cCsD|durtd|dkrtd|d||_||_t|j|_dS)NzProvided dataset is None.rz)Provided batch_size should be >= 1 (got: ).) ValueErrorr%r$iterr&)selfr%r$r+n/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/optimum/onnxruntime/quantization.py__init__4sz!ORTCalibrationDataReader.__init__cCsd}z7|jdkrddt|jD}n#tt}t|jD]}t|j}|D] \}}|||g7<q)qWn tyBYnw|durOt|dkrO|SdS)NrcSsi|]\}}||gqSr+r+).0keyvaluer+r+r, Dsz5ORTCalibrationDataReader.get_next..r) r$nextr&itemsrlistrange StopIterationlen)r*featurized_samples_samplenamer0r+r+r,get_next@s"   z!ORTCalibrationDataReader.get_nextN)r)__name__ __module__ __qualname__ __slots__intr-r<r+r+r+r,r#1s r#cseZdZdZd8dededffdd Ze d8ded e efd ee d dfd d Z      d9ddde dee efdee e de dededed ee eeefffddZ      d9ddde dee efdee e de dededefddZd ee eeefffddZ   d:d!ed"ee efd#ee d$eee eeeffded%eed efd&d'Z (    ) *  d;d+e d,e d-ee d.ee d/eed0ed1e d2eeee fd3eeee fd dfd4d5Zd= 2Gb. use_gpu (`bool`, defaults to `False`): Whether to use the GPU when collecting the quantization ranges values. force_symmetric_range (`bool`, defaults to `False`): Whether to make the quantization ranges symmetric. Returns: The dictionary mapping the nodes name to their quantization ranges. z+Using static quantization schema (dataset: z , method: ))rIinfo dataset_namemethod partial_fitcompute_ranges) r*r%rarbrcr$rdrerfr+r+r,fits$$ zORTQuantizer.fitc Csx|jdur td|jd|d|j|j||||d|_|r*|jjdgdtdt||} |j | dS) a Performs the calibration step and collects the quantization ranges without computing them. Args: dataset (`Dataset`): The dataset to use when performing the calibration step. calibration_config (`CalibrationConfig`): The configuration containing the parameters related to the calibration step. onnx_augmented_model_name (`Union[str, Path]`, defaults to `"augmented_model.onnx"`): The path used to save the augmented model used to collect the quantization ranges. operators_to_quantize (`Optional[List[str]]`, defaults to `None`): List of the operators types to quantize. batch_size (`int`, defaults to 1): The batch size to use when collecting the quantization ranges values. use_external_data_format (`bool`, defaults to `False`): Whether uto se external data format to store model which size is >= 2Gb. use_gpu (`bool`, defaults to `False`): Whether to use the GPU when collecting the quantization ranges values. force_symmetric_range (`bool`, defaults to `False`): Whether to make the quantization ranges symmetric. NzCreating calibrator: (rg)rCrdaugmented_model_namercrfCUDAExecutionProvider)execution_providersz Collecting tensors statistics...) rjrIrhcreate_calibratorrCas_posixrKset_execution_providersr# collect_data) r*r%rarbrcr$rdrerfreaderr+r+r,rks !  zORTQuantizer.partial_fitcCs@|jdur tdtdtttdkr|jS|jS)z Computes the quantization ranges. Returns: The dictionary mapping the nodes name to their quantization ranges. Nz^Calibrator is None, please call `partial_fit` or `fit` method at least ones to compute ranges.zComputing calibration ranges1.16.0) rKr(rIrhr ort_versionr compute_data compute_range)r*r+r+r,rls    zORTQuantizer.compute_ranges quantizedquantization_configsave_dir file_suffixcalibration_tensors_range preprocessorcCs|jo|jtjk}t|}|jddd|jr|durtd|jsB|jtj kr2t d|jd|j t jkrBt d|j dt d|jrJd nd d ||duryt d ||j\}} ||j| |jt||_t| |_d } tt|j} | jjD] } | jdvrd} nq| r|jrtdtttdkrtd|rt nt!} | |j|j"|j|j#|j ||j$|j|jdd|j%D|j&|j'| |j'o|j&|j(|j)|j*dd }|r|+dtttdkr|+d tttdkr|+d|d<| d#i|}t d|,|rd|nd}|-|jj.|/d}t d|d |d!|j01||t2||d"}|3||j4durV|j43|t5|jj6|t|S)$a Quantizes a model given the optimization specifications defined in `quantization_config`. Args: quantization_config (`QuantizationConfig`): The configuration containing the parameters related to quantization. save_dir (`Union[str, Path]`): The directory where the quantized model should be saved. file_suffix (`Optional[str]`, defaults to `"quantized"`): The file_suffix used to save the quantized model. calibration_tensors_range (`Optional[Dict[str, Tuple[float, float]]]`, defaults to `None`): The dictionary mapping the nodes name to their quantization ranges, used and required only when applying static quantization. use_external_data_format (`bool`, defaults to `False`): Whether to use external data format to store model which size is >= 2Gb. preprocessor (`Optional[QuantizationPreprocessor]`, defaults to `None`): The preprocessor to use to collect the nodes to include or exclude from quantization. Returns: The path of the resulting quantized model. T)parentsexist_okNzRequested static quantization in the QuantizationConfig, but no calibration ranges were provided. Please run calibration first using the quantizer fit method, or use dynamic quantization.zSONNX Runtime dynamic quantization mode should be QuantizationMode.IntegerOps (got: r'zYONNX Runtime dynamic quantization activations data type should be QuantType.QUInt8 (got: z Creating staticdynamicz quantizer: z:Preprocessor detected, collecting nodes to include/excludeF)IfLoopScan SequenceMapzIStatic quantization is currently not supported for models with subgraphs.rwzONNX Runtime version v1.16.0 is not compatible with quantization for models with subgraphs, please downgrade to 1.15.1 or upgrade to a higher version. Reference: https://github.com/microsoft/onnxruntime/pull/17651cSs g|] }t|tr |jn|qSr+)rRrr0)r.operatorr+r+r, wsz)ORTQuantizer.quantize..)WeightSymmetricActivationSymmetricEnableSubgraphForceSymmetricAddQDQPairToWeightDedicatedQDQPair QDQOpTypePerChannelSupportToAxis) rXr per_channelmode weight_qType input_qType tensors_range reduce_rangenodes_to_quantizenodes_to_excludeop_types_to_quantize extra_optionsrz1.18.0z1.13.0ractivation_qTypezQuantizing model...r9z.onnxzSaving quantized model at: z (external data format: rg) quantizationrdr+)7 is_staticformatrQDQrmkdirr(rr IntegerOpsrIrJactivations_dtyperQUInt8rhcollectrCupdaterrr4onnxloadrsgraphnodeop_typerTr rxr rrr weights_dtyperrcweights_symmetricactivations_symmetricqdq_add_pair_to_weightqdq_dedicated_pair'qdq_op_type_per_channel_support_to_axispopquantize_modeljoinpathstem with_suffixrXsave_model_to_filersave_pretrainedrDrrG)r*r|r}r~rrdruse_qdqrr has_subgraphs onnx_modelrquantizer_factoryquantizer_kwargs quantizersuffixquantized_model_path ort_configr+r+r,quantizes                  zORTQuantizer.quantizedTri num_samplesdataset_config_name dataset_splitpreprocess_functionpreprocess_batchseeduse_auth_tokentokenc Cs|durtdt| durtd|} |durtdt|dgddlm} | |||| d} |durFt|t| }| j |d  t |} |durR| j ||d } n| } | | S) a Creates the calibration `datasets.Dataset` to use for the post-training static quantization calibration step. Args: dataset_name (`str`): The dataset repository name on the Hugging Face Hub or path to a local directory containing data files to load to use for the calibration step. num_samples (`int`, defaults to 100): The maximum number of samples composing the calibration dataset. dataset_config_name (`Optional[str]`, defaults to `None`): The name of the dataset configuration. dataset_split (`Optional[str]`, defaults to `None`): Which split of the dataset to use to perform the calibration step. preprocess_function (`Optional[Callable]`, defaults to `None`): Processing function to apply to each example after loading dataset. preprocess_batch (`bool`, defaults to `True`): Whether the `preprocess_function` should be batched. seed (`int`, defaults to 2016): The random seed to use when shuffling the calibration dataset. use_auth_token (`Optional[Union[bool,str]]`, defaults to `None`): Deprecated. Please use the `token` argument instead. token (`Optional[Union[bool,str]]`, defaults to `None`): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `huggingface-cli login` (stored in `huggingface_hub.constants.HF_TOKEN_PATH`). Returns: The calibration `datasets.Dataset` to use for the post-training static quantization calibration step. NznThe `use_auth_token` argument is deprecated and will be removed soon. Please use the `token` argument instead.zLYou cannot use both `use_auth_token` and `token` arguments at the same time.zkORTQuantizer: Static quantization calibration step requires a dataset_name if no calib_dataset is provided.datasetsr) load_dataset)r;splitr)r)batched)warningswarn FutureWarningr(rrrminr7shuffleselectr5mapclean_calibration_dataset) r*rirrrrrrrrr calib_datasetprocessed_calib_datasetr+r+r,get_calibration_datasets6*   z$ORTQuantizer.get_calibration_datasetcCs:t|j}dd|jjD}tt|j|}||S)NcSsh|]}|jqSr+)r;)r.inputr+r+r, sz9ORTQuantizer.clean_calibration_dataset..) rrrCrrr4set column_namesremove_columns)r*r%rX model_inputsignored_columnsr+r+r,rs  z&ORTQuantizer.clean_calibration_dataset)N)r`NrFFF)r{NFN)rNNNTrNN)r%r!rPr!)r=r>r?__doc__rrr- classmethodr rSrFrrrAboolrr floatrmrkrlrr rrrr __classcell__r+r+rLr,rBUs  3    :   2     OrB)s8   $              $