ujJ0dZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z ddlmZmZddlmZddlmZmZmZdd lmZmZdd lmZdd lmZeZd d gZedGddZ edGddZ!edGddZ"edGddZ# d#de e e$efde#fdZ%dede&fdZ'd$dede$de&fd Z(d!ede$de e"fd"Z)dS)%zr&r'c*t|jS)zi (property) Size of the blob file as a human-readable string. Example: "42.2K". rrr"s r$size_on_disk_strzCachedFileInfo.size_on_disk_strH%T%6777r'N) __name__ __module__ __qualname____doc__str__annotations__rintfloatpropertyr%r)r,r'r$rrs&NNNOOOOOO 9999X99999X98#888X888r'rceZdZUdZeed<eed<eed<ee ed<e ed<e defdZ e defd Z e defd Zd S) CachedRevisionInfoaFrozen data structure holding information about a revision. Args: commit_hash (`str`): Hash of the revision (unique). Example: `"9338f7b671827df886678df2bdd7cc7b4f36dffd"`. snapshot_path (`Path`): Path to the revision directory in the `snapshots` folder. It contains the exact tree structure as the repo on the Hub. files: (`FrozenSet[CachedFileInfo]`): Set of [`~CachedFileInfo`] describing all files contained in the snapshot. size_on_disk (`int`): Sum of the blob file sizes that are symlink-ed by the revision. last_modified (`float`): Timestamp of the last time the revision has been created/modified. commit_hash snapshot_pathrfiles last_modifiedr c*t|jS)z (property) Timestamp of the last time the revision has been modified, returned as a human-readable string. Example: "2 weeks ago". rr=r"s r$last_modified_strz$CachedRevisionInfo.last_modified_strl  2333r'c*t|jSzn (property) Sum of the blob file sizes as a human-readable string. Example: "42.2K". r+r"s r$r,z#CachedRevisionInfo.size_on_disk_strvr-r'c*t|jS)zC (property) Total number of files in the revision. )lenr<r"s r$nb_fileszCachedRevisionInfo.nb_filess 4:r'N)r.r/r0r1r2r3rr4rrr5r6r@r,rFr7r'r$r9r9Rs" ^ $$$$ 43444X48#888X8#Xr'r9ceZdZUdZeed<eed<eed<eed<eed<ee ed<e ed<e ed <e d efd Z e d efd Z e d efd ZdS)CachedRepoInfoaFrozen data structure holding information about a cached repository. Args: repo_id (`str`): Repo id of the repo on the Hub. Example: `"damo/bert-base-chinese"`. repo_type (`Literal["dataset", "model"]`): Type of the cached repo. repo_path (`Path`): Local path to the cached repo. size_on_disk (`int`): Sum of the blob file sizes in the cached repo. nb_files (`int`): Total number of blob files in the cached repo. revisions (`FrozenSet[CachedRevisionInfo]`): Set of [`~CachedRevisionInfo`] describing all revisions cached in the repo. last_accessed (`float`): Timestamp of the last time a blob file of the repo has been accessed. last_modified (`float`): Timestamp of the last time a blob file of the repo has been modified/created. repo_id repo_type repo_pathrrF revisions last_accessedr=r c*t|jS)z (property) Last time a blob file of the repo has been accessed, returned as a human-readable string. Example: "2 weeks ago". )rrMr"s r$last_accessed_strz CachedRepoInfo.last_accessed_strrAr'c*t|jS)z (property) Last time a blob file of the repo has been modified, returned as a human-readable string. Example: "2 weeks ago". r?r"s r$r@z CachedRepoInfo.last_modified_strrAr'c*t|jSrCr+r"s r$r,zCachedRepoInfo.size_on_disk_strr-r'N)r.r/r0r1r2r3rr4rr9r5r6rOr@r,r7r'r$rHrHs*LLLNNNOOOMMM+,,,, 43444X443444X48#888X888r'rHcreZdZUdZeed<eeed<ee ed<e de fdZ de fdZ dS) ModelScopeCacheInfoaFrozen data structure holding information about the entire cache-system. This data structure is returned by [`scan_cache_dir`] and is immutable. Args: size_on_disk (`int`): Sum of all valid repo sizes in the cache-system. repos (`FrozenSet[CachedRepoInfo]`): Set of [`~CachedRepoInfo`] describing all valid cached repos found on the cache-system while scanning. warnings (`List[CorruptedCacheException]`): List of [`~CorruptedCacheException`] that occurred while scanning the cache. Those exceptions are captured so that the scan can continue. Corrupted repos are skipped from the scan. rreposwarningsr c*t|jS)zp (property) Sum of all valid repo sizes in the cache-system as a human-readable string. r+r"s r$r,z$ModelScopeCacheInfo.size_on_disk_strs %T%6777r'cdtdtdttfdgd}fdt |jdD}t || S) aGenerate a detailed table from the [`ModelScopeCacheInfo`] object. Returns a table with a row per repo and revision (thus multiple rows can appear for a single repo), with columns "repo_id", "repo_type", "revision", "size_on_disk", "nb_files", "last_modified", "local_path". Example: ```py >>> from modelscope.hub.cache_manager import scan_cache_dir >>> ms_cache_info = scan_cache_dir() ModelScopeCacheInfo(...) >>> print(ms_cache_info.export_as_table()) REPO ID REPO TYPE REVISION SIZE ON DISK NB FILES LAST_MODIFIED LOCAL PATH ---------------------- --------- ---------- ------------ -------- ------------- ------------------------------------------------------------- damo/bert-base-chinese model master 2.7M 5 1 week ago ~/.cache/modelscope/hub/models--damo--bert-base-chinese/... damo/structured-bert model master 8.8K 1 1 week ago ~/.cache/modelscope/hub/models--damo--structured-bert/... damo/t5-base model master 893.8M 4 7 months ago ~/.cache/modelscope/hub/models--damo--t5-base/... ``` Returns: `str`: The table as a string. reporevisionr c |j|j|jd|j|j|j|jt|j gS)zHFormat a single repo and revision into a list of strings for tabulation.z{:>12}) rIrJr:formatr,rFrOr@r2rK)rXrYs r$format_repo_revisionzAModelScopeCacheInfo.export_as_table..format_repo_revisionsN $ 566 &&DN##  r')zREPO IDz REPO TYPEREVISIONz SIZE ON DISKzNB FILES LAST_ACCESSED LAST_MODIFIEDz LOCAL PATHc\g|](}t|jdD]}||)S)c|jSN)r:)rYs r$z@ModelScopeCacheInfo.export_as_table...s X5Ir'key)sortedrL).0rXrYr\s r$ z7ModelScopeCacheInfo.export_as_table..sj   "$I$IKKK   ! x 0 0    r'c|jSrb)rI)rXs r$rcz5ModelScopeCacheInfo.export_as_table..sDLr'rd)rowsheaders)rHr9rr2rfrTr)r#column_headers table_datar\s @r$export_as_tablez#ModelScopeCacheInfo.export_as_tables2 ~ += BFs)            tz/H/HIII   "    r'N)r.r/r0r1r4r3rrHrr r6r2r,rnr7r'r$rSrSs  ^ $$$$*++++ 8#888X8< < < < < < < r'rS cache_dirr c|t}t|}|st d|d||rtd|dt}g}|dz }|dz }|rW| rCt|t\}}| || ||rW| rCt|t\}}| || |t|td \} } | | | | tt!|t#d |D| S) aScan the entire ModelScope cache-system and return a [`ModelScopeCacheInfo`] structure. Use `scan_cache_dir` to programmatically scan your cache-system. The cache will be scanned repo by repo. If a repo is corrupted, a [`~CorruptedCacheException`] will be thrown internally but captured and returned in the [`~ModelScopeCacheInfo`] structure. Only valid repos get a proper report. ```py >>> from modelscope.hub.utils import scan_cache_dir >>> ms_cache_info = scan_cache_dir() ModelScopeCacheInfo( size_on_disk=3398085269, repos=frozenset({ CachedRepoInfo( repo_id='damo/t5-small', repo_type='model', repo_path=PosixPath(...), size_on_disk=970726914, nb_files=11, revisions=frozenset({ CachedRevisionInfo( commit_hash='master', size_on_disk=970726339, snapshot_path=PosixPath(...), files=frozenset({ CachedFileInfo( file_name='config.json', size_on_disk=1197 file_path=PosixPath(...), blob_path=PosixPath(...), ), CachedFileInfo(...), ... }), ), CachedRevisionInfo(...), ... }), ), CachedRepoInfo(...), ... }), warnings=[ CorruptedCacheException("Snapshots dir doesn't exist in cached repo: ..."), CorruptedCacheException(...), ... ], ) ``` Args: cache_dir (`str` or `Path`, `optional`): Cache directory to scan. Defaults to the default ModelScope cache directory. Raises: `CacheNotFound`: If the cache directory does not exist. `ValueError`: If the cache directory is a file, instead of a directory. Returns: a [`ModelScopeCacheInfo`] object. NzCache directory not found: zQ. Please use `cache_dir` argument or set `MODELSCOPE_CACHE` environment variable.)roz1Scan cache expects a directory but found a file: modelsdatasetsrJT)rJinplacec3$K|] }|jV dSrb)r)rgrXs r$ z!scan_cache_dir..s%==t*======r')rTrrU)rr expanduserresolveexistsr is_file ValueErrorsetis_dir _scan_dirrupdateextendrrS frozensetsum) rorTrU model_dir dataset_dir model_reposmodel_warnings dataset_reposdataset_warnings other_reposother_warningss r$scan_cache_dirr#sG~,.. Y**,,4466I       G) G G G      ] ] ] ]   "%E.0HH$Ij(K(i..00(&/ '2'2'2# ^ [!!!'''* 2 2 4 4**3 #4+6+6+6' ' ]###()))#,_d#<#<#<K LL OON### ==u=====   r'dirc|sdS|sdS|rdS|jtvrdSdS)z+Check if a directory is valid for scanning.FT)ryr} is_symlinknameFILES_TO_IGNORE)rs r$ _is_valid_dirrsZ ::<<u ::<<u ~~u x?""u 4r'FrJrtct}g}|D]}|r |jdvrt|s|D]g}t|s t ||}|||<#t $r}||Yd}~`d}~wwxYw||fS)zWScan a directory for cached repos and return a set of [`~CachedRepoInfo`] and warnings.)rqrrhubrsN)r|iterdirrr_scan_cached_repoaddr append) rrJrtrTrU owner_dirname_dirinfoes r$r~r~s EEEH[[]]##  y~)FFF Y''  !))++ # #H **  #(YGGG#IIdOOO* # # #"""""""" # # (?s*(B B<B77B<rKc ^|std| tt|}|j}|j}|dd}|dkrdSn$#t$r}td|d}~wwxYwit}|D]}tj ||d} | dd } tj | s^t| } | | <|t%tj |d| | | j| | j| j d } |r8d |vr4|d d dd} |rt1fd|D} n|j} t3| t5|t7fd|D|| }rWt1dD}t1dD}n"|}|j}|j}t;t=|||t5|gt7dD||S)zScan a single cache repo and return information about it. Any unexpected behavior will raise a [`~CorruptedCacheException`]. zRepo path is not a directory: ___.unknownNz"Failed to load cache information: rRevision)rrrrrrrmasterz Revision:,rc3<K|]}|jjVdSrb)rst_mtimergfile blob_statss r$rvz$_scan_cached_repo..sI%D%D)-&0%?%H%D%D%D%D%D%Dr'c3<K|]}|jjVdSrb)rst_sizers r$rvz$_scan_cached_repo..sA88!$DN3;888888r')r:r<rr;r=c3$K|] }|jV dSrb)st_atimergstats r$rvz$_scan_cached_repo..$ O O4 O O O O O Or'c3$K|] }|jV dSrb)rrs r$rvz$_scan_cached_repo..rr'c3$K|] }|jV dSrb)rrs r$rvz$_scan_cached_repo..s$FF$FFFFFFr')rFrIrKrJrLrrMr=)r}r rr2 cached_filescached_model_revision get_model_idreplace Exceptionr|ospathjoingetryrrrrbasenamerrrsplitmaxr9rrvaluesrHrE)rKrJcacherrrIrcached_files_info cached_filerrr revision_hashrevision_last_modifiedcached_revisionrepo_last_accessedrepo_last_modified repo_statsrs @r$rrs     :% 8Y 8 8:: : P$S^^44) % ;$$&&..uc:: i  4 PPP%&N1&N&NOOOPJ$ GLLK,?@@ (__Z<<w~~i((  OO ) 0 0 9  '**;v+>??##5' 2:##-i#8#A#-i#8#A       M / / /177 DDQGMMM;!$%D%D%D%D1B%D%D%D"D"D"+!1!1!:)!)**8888%688888, O1 O O:;L;L;N;N O O OOO O O:;L;L;N;N O O OOO^^%% '0'0 Z_-..FF*2C2C2E2EFFFFF((    sAB B%B  B%rb)F)*r1r dataclassesrpathlibrtypingrrrrr r r modelscope.hub.errorsr r modelscope.hub.utils.cachingrmodelscope.hub.utils.utilsrrrmodelscope.utils.constantrrmodelscope.utils.file_utilsrmodelscope.utils.loggerrloggerrrr9rHrSr2rboolrr~rr7r'r$rsBB !!!!!!GGGGGGGGGGGGGGGGGGHHHHHHHH======DDDDDDDDDDHHHHHHHH@@@@@@......  - $8888888888888888v $11111111h $;8;8;8;8;8;8;8;8| $Y Y Y Y Y Y Y Y z15ppE#t),-p9Lppppf t     4C$.\\!$\)1.)A\\\\\\r'