o diP @sdZddlZddlmZddlmZmZddlmZm Z m Z m Z e e Z ddeeefd ed ed efd d Z  ddeeefdeeefded efddZdS)z"Utilities related to saving files.N)Path)ListUnion)AutoFeatureExtractorAutoImageProcessor AutoProcessor AutoTokenizerFsrc_name_or_path subfoldertrust_remote_codereturncCsg}z |tj|||dWn tyYnwz |tj|||dWn ty/Ynwz |tj|||dWn tyFYnwz|tj|||dW|Sty_Y|Sw)Nr r )appendrfrom_pretrained Exceptionrrr)r r r preprocessorsrf/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/optimum/utils/save_utils.pymaybe_load_preprocessorssF    rdest_dir src_subfoldercCs@t|ts t|}|jddt|||dD]}||qdS)a Saves the tokenizer, the processor and the feature extractor when found in `src_dir` in `dest_dir`. Args: src_dir (`Union[str, Path]`): The source directory from which to copy the files. dest_dir (`Union[str, Path]`): The destination directory to copy the files to. src_subfolder (`str`, defaults to `""`): In case the preprocessor files are located inside a subfolder of the model directory / repo on the Hugging Face Hub, you can specify the subfolder name here. trust_remote_code (`bool`, defaults to `False`): Whether to allow to save preprocessors that is allowed to run arbitrary code. Use this option at your own risk. T)exist_okrN) isinstancermkdirrsave_pretrained)r rrr preprocessorrrrmaybe_save_preprocessorsAs   r)r F)__doc__loggingpathlibrtypingrr transformersrrrr getLogger__name__loggerstrboolrrrrrrs6    )