o diA@sdZddlZddlZddlZddlmZddlmZmZm Z ddl m Z ddl m Z    dde eefd ed ed eed ee eefd ef ddZ     dde eefdeded edee eefd ee eefd eed eefddZdS)zPUtility functions related to both local files and files on the Hugging Face Hub.N)Path)ListOptionalUnion)HfApi)http_user_agentmodel_name_or_pathfilename subfolderrevisiontokenreturncCsJtj|rtjtj|||Stt|djtj|||||dS)zl Checks that the file called `filename` exists in the `model_name_or_path` directory or model repo.  user_agentr )r repo_idr r )ospathisdirisfilejoinrr file_exists)r r r r r rf/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/optimum/utils/file_utils.pyvalidate_file_existss  r**/*pattern glob_patternuse_auth_tokenc s|durtdt|durtd|}t|trt|n|}t|t j |r>t| |}fdd|D}|St t|dj|||d} fdd| D}|S) a Scans either a model repo or a local directory to find filenames matching the pattern. Args: model_name_or_path (`Union[str, Path]`): The name of the model repo on the Hugging Face Hub or the path to a local directory. pattern (`str`): The pattern to use to look for files. glob_pattern (`str`, defaults to `"**/*"`): The pattern to use to list all the files that need to be checked. subfolder (`str`, defaults to `""`): In case the model files are located inside a subfolder of the model directory / repo on the Hugging Face Hub, you can specify the subfolder name here. use_auth_token (`Optional[Union[bool,str]]`, defaults to `None`): Deprecated. Please use the `token` argument instead. token (`Optional[Union[bool,str]]`, defaults to `None`): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `huggingface-cli login` (stored in `huggingface_hub.constants.HF_TOKEN_PATH`). token (`Optional[Union[bool, str]]`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `transformers-cli login` (stored in `~/.huggingface`). revision (`Optional[str]`, defaults to `None`): Revision is the specific model version to use. It can be a branch name, a tag name, or a commit id. Returns: `List[Path]` NznThe `use_auth_token` argument is deprecated and will be removed soon. Please use the `token` argument instead.zLYou cannot use both `use_auth_token` and `token` arguments at the same time.cs g|] }tt|r|qSr)researchstr.0prrr c z/find_files_matching_pattern..r)r r cs g|] }t|rt|qSr)rmatchrr"r%rrr&hr')warningswarn FutureWarning ValueError isinstancerr!rcompilerrrglobrrlist_repo_files) r rrr rr r model_pathfiles repo_filesrr%rfind_files_matching_pattern1s&% r4)rNN)rrNNN)__doc__rrr)pathlibrtypingrrrhuggingface_hubrtransformers.utilsrr!boolrr4rrrrs\