o diD(@sddlZddlZddlmZddlmZddlmZmZddl Z ddl Z ddl Z ddl Z ddlmZddlmZddlmZmZmZmZdd lmZmZerTdd lmZd ejd <d dZGdddZGdddZ dZ!dZ"ddZ#GdddZ$eeeedZ%dS)N)contextmanager)perf_counter_ns) TYPE_CHECKINGSet)trange)version)ImageClassificationProcessingQuestionAnsweringProcessingTextClassificationProcessingTokenClassificationProcessing) RunConfigcpu_info_command)DatasetfalseTOKENIZERS_PARALLELISMcCs|dvrd}|S|}|S)N)text-classificationzaudio-classificationzsequence-classification)taskautoclass_namerr_/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/optimum/runs_base.pyget_autoclass_names rc@s"eZdZ  dddZddZdS) Calibratorcalibration_datasetrcCs(||_||_||_||_||_||_dSN)r quantizer model_pathqconfigcalibration_paramsnode_exclusion)selfrrrrrrrrr__init__&s  zCalibrator.__init__cCtrNotImplementedErrorr rrrfit0szCalibrator.fitN)rr)__name__ __module__ __qualname__r!r&rrrrr%s  rc@sReZdZdefddZddZddZdd Zd d Zd d Z ddZ ddZ dS)Run run_configcCsbtd$i||d|_|ddkrd|_nd|_|d|dd}tjd d gtj|d |_tj t gdd  d }d}dt j vrRtj ddd }| d d}id|dd|jd|dd|dd|dd|dd|dd|dd|dd|dd|dd|dd|dtj t j |ddgiid d!d"|d"d#|d#|_dS)%adInitialize the Run class holding methods to perform inference and evaluation given a config. A run compares a transformers model and an optimized model on latency/throughput, model size, and provided metrics. Args: run_config (dict): Parameters to use for the run. See [`~utils.runs.RunConfig`] for the expected keys. rquantization_approachstaticTF batch_sizes input_lengths) batch_size input_lengthmaximizeminimize) directionssampler)shellzutf-8NdevzQgit ls-remote https://github.com/huggingface/optimum.git HEAD | awk '{ print $1}' model_name_or_path task_argsdatasetoperators_to_quantizeraware_training per_channel calibration frameworkframework_argshardwareversions) transformersoptimum optimum_hash evaluation)baseline optimized)timeothersmax_eval_samplestime_benchmark_argsr)r rstatic_quantizationoptuna create_studysamplers GridSamplerstudy subprocess check_outputrdecodeoptimum_version __version__striprD return_body)r r+ search_spacecpu_inforFrrrr!5sv          z Run.__init__cCsBz|j|j|W|td|jS|tdw)aLaunch inference to compare metrics between the original and optimized model. These metrics are latency, throughput, model size, and user provided metrics. Returns: `dict`: Finalized run data with metrics stored in the "evaluation" key. z Finished run.)rSoptimize _launch_time launch_evalfinalizeprintrZr%rrrlaunchqs  z Run.launchcCr")zOptuna objective function to measure latency/throughput. Populate the `["evaluation"]["time"]` list of the run for various batch size and input length. Returns: Dummy data. r#)r trialrrrr^szRun._launch_timecCr")z Run evaluation on the original and optimized model. Populate the `["evaluation"]["others"]` subdictionary of the run. r#r%rrrr_szRun.launch_evalcCs,|j}|d|_|jr|d|_dSdS)zTLoad evaluation dataset, and if needed, calibration dataset for static quantization.evalr?N)task_processor load_datasets _eval_datasetrN_calibration_dataset)r datasets_dictrrrrfs  zRun.load_datasetscCt|ds td|jS)zGet calibration dataset. The dataset needs to be loaded first with [`~optimum.runs_base.Run.load_datasets`]. Returns: `datasets.Dataset`: Calibration dataset. rhz,No calibration dataset defined for this run.)hasattrKeyErrorrhr%rrrget_calibration_datasets zRun.get_calibration_datasetcCrj)z Get evaluation dataset. The dataset needs to be loaded first with [`~optimum.runs_base.Run.load_datasets`]. Returns: `datasets.Dataset`: Evaluation dataset. rgz+No evaluation dataset defined for this run.)rkrlrgr%rrrget_eval_datasets zRun.get_eval_datasetcCr")zCleanup intermediary files.r#r%rrrr`sz Run.finalizeN) r'r(r)dictr!rbr^r_rfrmrnr`rrrrr*4s<  r*iʚ;.AcCs|tSr)NS_TO_MS_SCALE)ns_timerrrns_to_mssrsc @sjeZdZdededeededef ddZedefd d Z e d d Z d efddZ ddZ ddZdS) TimeBenchmarkr0r1model_input_names warmup_runsdurationcCs8||_||_||_||_||_g|_td|_||_dS)Nz-inf) r0r1modelrvbenchmark_duration latenciesfloat throughputru)r rxr0r1rurvrwrrrr!s  zTimeBenchmark.__init__returncCs t|jSr)lenrzr%rrrnum_runss zTimeBenchmark.num_runsccsLt}dVt}|j||td||d||ddddS)NzTracked function took: zns (rpz.3fzms))rrzappendra)r startendrrrtracks (zTimeBenchmark.track duration_nscCstt|j|td|_dS)N)roundr~rzSEC_TO_NS_SCALEr|)r rrrrr`szTimeBenchmark.finalizec Cst|j|jtt|jtt|jtt|jdtt|jdtt|jdtt|jdtt|jdd }|S)Ng?g?gffffff?gGz?g+?) nb_forwardsr| latency_mean latency_std latency_50 latency_90 latency_95 latency_99 latency_999)r~rzr|rsnpmeanstdquantile)r benchmarks_statsrrrto_dicts zTimeBenchmark.to_dictcsi}hdd|jvrtjd|j|jfd|d<d|jvr*tj|j|jtjd|d<d|jvr input_ids pixel_valuesattention_masktoken_type_idsri)highsizer)dtyperrcsg|]}|vqSrr).0kchecked_inputsrr sz)TimeBenchmark.execute..zAt least an input in z, has no dummy generation for time benchmark.z Warming up)descrzRunning time tracking in z.1fzs.)rr|rr)rutorchrandintr0r1onesint64randrxconfig image_sizefloat32ranyr$rrvforwardryrrasumrzrr`r)r inputs_benchmark_duration_nsrrrrexecutesB          zTimeBenchmark.executeN)r'r(r)intrstrr{r!propertyrrrr`rrrrrrrts&    rt)rztoken-classificationzquestion-answeringzimage-classification)&osrT contextlibrrJrtypingrrnumpyrrOrrDtqdmrrrWutils.preprocessingr r r r utils.runsr rdatasetsrenvironrrr*rrqrsrttask_processing_maprrrrs:      a