uj4ddlZddlZddlmZddlmZmZmZddlm Z m Z m Z ddl m Z ddlmZddlmZe ZGd d ZGd d ZdS) N)Optional)GitErrorInvalidParameterNotLoginException)DEFAULT_DATASET_REVISIONDEFAULT_REPOSITORY_REVISIONMASTER_MODEL_BRANCH) get_logger)GitCommandWrapper) get_endpointc"eZdZdZedddfdededeedeedeedeef d Zd Zd Z ddedefdZ defdZ eedfdedeedeedee fdZ efdededeefdZefdededeefdZdS) Repositoryz8A local representation of the model git repository. N model_dir clone_fromrevision auth_tokengit_pathendpointc(||_||_tj||_tj||_|sd}t|ddl m }|r||_ n| |_ t} | stdt||_tj|jd||} tj|jr8|} |j| } | r| | krdS|j|j|j | |j|| r| |j|j|j|j|j r'|j|j|j dSdS)aInstantiate a Repository object by cloning the remote ModelScopeHub repo Args: model_dir (str): The model root directory. clone_from (str): model id in ModelScope-hub from which git clone revision (str, optional): revision of the model you want to clone from. Can be any of a branch, tag or commit hash auth_token (str, optional): token obtained when calling `HubApi.login()`. Usually you can safely ignore the parameter as the token is already saved when you login the first time, if None, we will use saved token. git_path (str, optional): The git command line path, if None, we use 'git' endpoint (str, optional): The ModelScope endpoint URL. If None, use default endpoint. Raises: InvalidParameter: revision is None. 0a non-default value of revision cannot be empty.rModelScopeConfigz)git lfs is not installed, please install.Texist_okN) _endpointrospathdirnamemodel_base_dirbasenamemodel_repo_namermodelscope.hub.apirr get_tokenr is_lfs_installedloggererror git_wrappermakedirs_get_model_id_urllistdir_get_remote_urlremove_token_from_urlclonegit_lfs_install add_user_infoconfig_auth_token) selfrrrrrrerr_msgrr(url remote_urls i/lsinfo/ai/hellotax_ai/data_center/backend/venv/lib/python3.11/site-packages/modelscope/hub/repository.py__init__zRepository.__init__s ."" gooi88!w// :: ,HG"7++ +777777  ;(DOO.88::DO')) ++-- F LLD E E E,X66 DNT2222$$Z00 :dn % % --//J)?? KKJ jC// t2DOS#3X ? ? ?  ' ' ) ) 8  ' ' 7 7 7 &&t':'+'; = = = ? P   . .t~t O O O O O P PcN|jr|jn t}|d|d}|S)N/.gitrr )r2model_idrr4s r6r*zRepository._get_model_id_urlTs3%)^G4>>**H*** r8cl |j|j}n#t$rd}YnwxYw|SN)r(get_repo_remote_urlrrr2remotes r6r,zRepository._get_remote_urlYsJ %99$.IIFF   FFF  " 11originmasterrBbranchcJ|j|j||dS)zPull remote branch Args: remote (str, optional): The remote name. Defaults to 'origin'. branch (str, optional): The remote branch. Defaults to 'master'. )rBrFN)r(pullr)r2rBrFs r6rHzRepository.pull`s* dnVFKKKKKr8file_name_suffixc tjd|dtj|jddS)zAdd file suffix to lfs list. Args: file_name_suffix (str): The file name suffix. examples '*.safetensors' z printf ' z( filter=lfs diff=lfs merge=lfs -text '>>z.gitattributesN)rsystemrjoinr)r2rIs r6 add_lfs_typezRepository.add_lfs_typeisP   rw||DN4?KKK &&t':'+'; = = =224>BB T^t<<< ??? ^/%'  ) ) ) ) )r8tag_namemessagerefc||dkrd}t|||dkrd}t||j|j|||dS)aCreate a new tag. Args: tag_name (str): The name of the tag message (str): The tag message. ref (str, optional): The tag reference, can be commit id or branch. Raises: InvalidParameter: no commit message. NzFWe use tag-based revision, therefore tag_name cannot be None or empty.z=We use annotated tag, therefore message cannot None or empty.)rWrcrdre)rr(tagr)r2rcrdrerbs r6rhzRepository.tags}  x2~~ZC"3'' ' ?gmmQC"3'' ' ^      r8cv|||||j|j|dS)zCreate tag and push to remote Args: tag_name (str): The name of the tag message (str): The tag message. ref (str, optional): The tag ref, can be commit id or branch. Defaults to MASTER_MODEL_BRANCH. )rWrcN)rhr(push_tagr)r2rcrdres r6 tag_and_pushzRepository.tag_and_pushs> 7C((( !!4>H!MMMMMr8)rDrE)__name__ __module__ __qualname____doc__rr]rr7r*r,rHrMr^rar rhrkr8r6rrs ,G-1+/+/ <P<P<P <P$C=<P&c] <P $C= <P $C= <P<P<P<P| LL3L3LLLL PS P P P P,G,G%* -)-) -)#C=-)%SM-)TN -)-)-)-)d"5#:+> N N" N! N#3- N N N N N Nr8rceZdZdZedddfdededeedeedeedeef d Zd efd Zed fd edeedee fdZ dZ dZ dS)DatasetRepositoryzEA local representation of the dataset (metadata) git repository. N repo_work_dir dataset_idrrrrc||_||_|rt|tsd}t ||d|_|jsd}t |tj |j|_ tj |j|_ |sd}t |||_ ddlm}|r||_n||_t%||_tj|jd|| |_d S) a Instantiate a Dataset Repository object by cloning the remote ModelScope dataset repo Args: repo_work_dir (str): The dataset repo root directory. dataset_id (str): dataset id in ModelScope from which git clone revision (str, optional): revision of the dataset you want to clone from. Can be any of a branch, tag or commit hash auth_token (str, optional): token obtained when calling `HubApi.login()`. Usually you can safely ignore the parameter as the token is already saved when you login the first time, if None, we will use saved token. git_path (str, optional): The git command line path, if None, we use 'git' endpoint (str, optional): The ModelScope endpoint URL. If None, use default endpoint. Raises: InvalidParameter: parameter invalid. z"dataset_work_dir must be provided!r:z%dataset_work_dir can not be root dir!rrrTr)rtN)rrtr\r]rrstriprsrrr repo_base_dirr! repo_namerr#rrr$r r(r) _get_repo_urlrepo_url) r2rsrtrrrrr3rs r6r7zDatasetRepository.__init__sA0"$ ,J}c$B$B ,:G"7++ +*11#66! ,=G"7++ +W__T-?@@))$*<== ,HG"7++ +  777777  ;(DOO.88::DO,X66 D&6666**j*AA r8returnctj|jr=|}|j|}|r ||jkrdStd |j|j |j |j |j|j |j|jS)NrgzCloning repo from {} )rr+rsr,r(r-rzr&infoformatr.rwrrxr)r2r5s r6r.zDatasetRepository.clones :d( ) ) --//J)?? KKJ jDM99r +224=AABBB t14?#}dndm M M M!!r8FrNrFrQctjdtd |t|tsd}t |t|t st d|jstd|j |j |j|j |j |j|}|j |}|j |j |j |j d|j |j ||j |j |j||| dSrS)rYrZr[r\r]rr^rrr(r1rsr0rwrxr,r-rHr_r`ra)r2rNrFrQrbr5s r6razDatasetRepository.pushs   T          !NC)H)H !4C"3'' '%&& 9"#788 8 O#$MNN N **4+=tOOO &&t'94>JJJ))++ %;;JGG  d0111 T/4@@@  2NCCC '/  " " " " "r8cJ|jr|jn t}|d|dS)Nz /datasets/r;r<)r2rtrs r6ryzDatasetRepository._get_repo_urlCs/%)^G4>>66j6666r8cl |j|j}n#t$rd}YnwxYw|Sr?)r(r@rsrrAs r6r,z!DatasetRepository._get_remote_urlGsK %99$:LMMFF   FFF  rC) rlrmrnrorr]rr7r.r^raryr,rpr8r6rrrrs ,D-1+/+/ 0B0B #0B 0B$C=0B&c] 0B $C= 0B $C= 0B0B0B0Bd "s " " " " &>%*0"0" 0"c]0"TN0"0"0"0"d777r8rr)rrYtypingrmodelscope.hub.errorsrrrmodelscope.utils.constantrrr modelscope.utils.loggerr gitr utils.utilsr r&rrrrpr8r6rs3 OOOOOOOOOO<<<<<<<<<</.....""""""%%%%%% xNxNxNxNxNxNxNxNvr8