o di*@sdZddlZddlZddlZddlmZmZddlmZm Z m Z m Z m Z m Z mZmZmZmZddlmZddlmZddlmZdd lmZerVdd lmZmZdd lmZeeZ eeefZ!Gd d d eZ"dS)z@Base class to peform task-specific preprocessing and evaluation.N)ABCabstractmethod) TYPE_CHECKINGAnyCallableDictListOptionalSetTupleTypeUnion)PreTrainedTokenizerBase)BaseImageProcessor)requires_backends)logging)Dataset DatasetDict)PretrainedConfigc@sFeZdZUeedfed<eeeee ffed<eeefed<e eed<e eed< d,dd d e d e eee ffd d Zd e eee fdeeee feee fffddZe d,deee fdeeefde e edeee ffddZ d,deeefde e edeeee fgeee fffddZ  d-deddeeefde e ede ededf ddZede ede eeeffddZede ede e efd d!Z   " "  " d.d#ede eeefde e ed$ed%ed&e ed'ededfd(d)Z " "  "d/d$ed%ed&e ed'efd*d+ZdS)0 TaskProcessor.ACCEPTED_PREPROCESSOR_CLASSESDEFAULT_DATASET_ARGSDEFAUL_DATASET_DATA_KEYSALLOWED_DATA_KEY_NAMESDEFAULT_REF_KEYSNconfigr preprocessorpreprocessor_kwargscCsZt||jstdt|dddd|jDd||_||_||\|_|_ dS)a Initializes the class in charge of loading processed datasets and of running evaluation. This class should be task-dependent, backend independent. Args: config (`PretrainedConfig`): The config of the model. preprocessor: (`Preprocessor`): The preprocessor associated to the model. This will be used to prepare the datasets. preprocessor_kwargs (`Optional[Dict[str, Any]]`, defaults to `None`): Keyword arguments that will be passed to the preprocessor during dataset processing. This allows customizing the behavior of the preprocessor. z3Preprocessor is incorrect, provided an instance of z) but expected one of the following type: , css|]}|jVqdSN)__name__).0cls_r$n/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/optimum/utils/preprocessing/base.py Gsz)TaskProcessor.__init__...N) isinstancer ValueErrortypejoinrr3create_defaults_and_kwargs_from_preprocessor_kwargsdefaultsr)selfrrrr$r$r%__init__0s  zTaskProcessor.__init__returncCs|duri}it|fS)ah Takes the dictionary of the preprocessor keyword arguments and return two dictionaries: - The first dictionary will either contain defaults values if not specified in preprocessor_kwargs or the values specified in preprocessor_kwargs. - The second dictionary will contain the rest of the keyword arguments. N)copydeepcopy)r.rr$r$r%r,Ps zATaskProcessor.create_defaults_and_kwargs_from_preprocessor_kwargsexample data_keysref_keyscCtdNz.This method must be implemented in subclasses.NotImplementedError)r.r3r4r5r$r$r%dataset_processing_func]sz%TaskProcessor.dataset_processing_funccCstj|j||dS)Nr4r5) functoolspartialr:)r.r4r5r$r$r%create_dataset_processing_funccsz,TaskProcessor.create_dataset_processing_funcdataset)rrsplitcCsTt|dgddlm}t||r|durtd|dur!||}||||S)Ndatasetsr)rzIA Dataset and a split name were provided, but splits are for DatasetDict.)rrArr(r)mapr>)r.r?r4r5r@rr$r$r%prepare_dataseths  zTaskProcessor.prepare_dataset column_namescCr6r7r8r.rDr$r$r%try_to_guess_data_keysxz$TaskProcessor.try_to_guess_data_keyscCr6r7r8rEr$r$r%try_to_guess_ref_keys|rGz#TaskProcessor.try_to_guess_ref_keysFpathonly_keep_necessary_columnsload_smallest_split num_samplesshufflec  szt|dgddlm} m} m} ddlm} ddlm}|dur&| |d}| |fi| d|i}t|| rd|rd| dd}|durKt d |d t | d d d d}t d|d||}|rj|}t|| r|dur||jkrt d|jd|d|t|}|j}t|trtttj|}|durt d||}|durt dd|t d|nt||jkst dt|d|jd|dur| |}t d|d|j!|||d}|r;|dur|ng}|j"j#|t|| r,| D]\}}fdd|jD}|$|||<q|Sfdd|jD}|$|}|S)NrAr)rrDownloadConfig) load_dataset)http_user_agent) user_agentdownload_configr@zA split name was provided (z?) but load_smallest_split is True, use either one or the other.cSs |djS)N)num_rows)itemr$r$r%s z,TaskProcessor.load_dataset..)keyzvSince no split was explicitely provided and load_smallest_split=True, using the smallest split of the dataset called "z".zThere are only z9 examples in the dataset, but it was requested to select z examples.z4As no data keys were provided, trying to guess them.zMData keys need to be specified manually since they could not be guessed from rz!Guessed the following data keys: z"data_keys contains unallowed keys z, allowed_keys: r'z3As no ref keys were provided, tried to guess them: r;cg|]}|vr|qSr$r$r"namenecessary_columnsr$r% z.TaskProcessor.load_dataset..crXr$r$rYr[r$r%r]r^)%rrArrrNrOtransformers.utilsrPr(getr)minitemsloggerinforMrTselectrangerDdictlistset itertoolschain from_iterablevalueswarningrFr+keysrrHrCrmodel_input_namesremove_columns)r.rIr4r5rJrKrLrMrRload_dataset_kwargsrrrNdatasets_load_datasetrPr?r@smallest_splitrD split_namecolumns_to_remover$r[r%rOs|               zTaskProcessor.load_datasetc Kst|jtrA|jdd}|durtdt|jt|@}|r.d|t dt |}| dd|j Dn|j}|}|j|f|j|j||||d|S)NrIzWhen DEFAULT_DATASET_ARGS is a dictionary, it must contain a key called "path" corresponding to the path or name of the dataset.rzThe following provided arguments will be overriden because they are hardcoded when using load_default_dataset: {override_config_key}.cSsi|] \}}|dkr||qS)rIr$)r"kvr$r$r% sz6TaskProcessor.load_default_dataset..)r4r5rJrKrLrM)r(rrgr`r)riror+rcrnr1r2updaterbrOrr) r.rJrKrLrMrrrI common_keyskwargsr$r$r%load_default_datasets8   z"TaskProcessor.load_default_datasetr )NN)NNFFNFN)FFNF)r! __module__ __qualname__r r __annotations__r strrrr r Preprocessorr r/r,rr:rr>rCrFrHboolintrOr}r$r$r$r%r)s                   $   Wr)#__doc__r1r<rjabcrrtypingrrrrrr r r r r transformersr#transformers.image_processing_utilsroptimum.utils.import_utilsrrrArrr get_loggerr!rcrrr$r$r$r%s 0