uj7 2 d dl Z d dlZd dlZd dlZd dlZd dlmZmZ d dlm Z d dl
m Z m Z d dl
mZ d dlmZ e Z ej e d dk Z G d d
e Z G d d e ZdS )
N)movermtree)Dict) FILE_HASH)MODELSCOPE_ENABLE_DEFAULT_HASH_VALIDATION) compute_hash)
get_loggerFalsetruec h e Zd ZdZdZdZdZ defdZd Z d Z
d Z d
Z d Z
d Zd
Zd Zd Zd ZdS )FileSystemCachez.mscz.mdlidz.mvcache_root_locationc t j | _ t j |d || _ | dS )zBase file system cache interface.
Args:
cache_root_location (str): The root location to store files.
kwargs(dict): The keyword arguments.
Texist_okN) threadingRLock _cache_lockosmakedirsr
load_cache)selfr kwargss l/lsinfo/ai/hellotax_ai/data_center/backend/venv/lib/python3.11/site-packages/modelscope/hub/utils/caching.py__init__zFileSystemCache.__init__ sH %?,,
'$7777#6 c | j S N)r r s r get_root_locationz!FileSystemCache.get_root_location1 s ''r c g | _ t j | j t
j }t j | r t |d 5 }t j
| }d d d n # 1 swxY w Y t |t r || _ d S t d|t | j d S # t" $ rW}t d|| t j ||dz n# t& $ r Y n
w xY wY d }~d S Y d }~d S d }~ww xY wd S )Nrbz1Cache index %s has unexpected type %s, resetting.uk Failed to load cache index %s: %s. Resetting — already-downloaded files will be re-validated on next run.
.corrupted) cached_filesr pathjoinr r
KEY_FILE_NAMEexistsopenpickleload
isinstancelistloggerwarningtype__name__ ExceptionreplaceOSError)r cache_keys_file_pathfdataes r r zFileSystemCache.load_cache4 s !w||D,D,;,I K K
7>>. / /
.55 *!;q>>D* * * * * * * * * * * * * * * dD)) -(,D%%%NNK, T
+- - - - -
_(!- - -J 3 3l BD D D D DD D D D D D DDDDD
sf C! 'B< C! B C! B C! 1.C! !
E+ D=D! D=!
D.+D=-D..D==Ec t j | j t j }t
j d| j \ }} t j |d 5 }t j
t | j | |
t j | ddd n # 1 swxY w Y t j || dS # t" $ r[ t j | n# t& $ r Y nw xY wt j | rt j | w xY w)z8Write .msc atomically. Caller must hold ``_cache_lock``.z.tmp)suffixdirwbN)r r& r' r r
r( tempfilemkstempfdopenr+ dumpr. r% flushfsyncfilenor4 r3 closer5 r) unlink)r r6 fd
temp_filenamer7 s r _save_cached_files_unlockedz+FileSystemCache._save_cached_files_unlockedN s !w||D,D,;,I K K $ ,t 7 9 9 9M
2t$$
% D!2 3 3Q777 $$$
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
J}&: ; ; ; ; ;
w~~m,,
) -((( sU C4 $A"C C4 CC4 CC4 4
E?DE
D! E D!!8Ec n | j 5 | ddd dS # 1 swxY w Y dS )a
Save cache metadata in order to verify that the cached content is consistent with the remote content.
Example of the cached content:
[{'Path': 'configuration.json', 'Revision': 'f01dxxx'}, {'Path': 'model.bin', 'Revision': '1159xxx'}, ...]
N)r rI r s r save_cached_filesz!FileSystemCache.save_cached_filesc s
/ / , , . . . / / / / / / / / / / / / / / / / / /s *..c dS )zCheck the key is in the cache, if exist, return the file, otherwise return None.
Args:
key(str): The cache key.
Raises:
None
N r keys r get_filezFileSystemCache.get_filem s
r c dS )zPut file to the cache.
Args:
key (str): The cache key
location (str): Location of the file, we will move the file to cache.
Raises:
None
NrM )r rO locations r put_filezFileSystemCache.put_filex s
r c x || j v r0| j | | dS dS )zvRemove cache key in index, The file is removed manually
Args:
key (dict): The cache key.
N)r% removerK rN s r
remove_keyzFileSystemCache.remove_key sI $# # # $ $S ) ) ) " " $ $ $ $ $ $ #r c , | j D ] }||k r dS dS )NTF)r% )r rO
cache_files r r) zFileSystemCache.exists s2 + JS tt ! ur c V t | j | dS )zRemove all files and metadata from the cache
In the case of multiple cache locations, this clears only the last one,
which is assumed to be the read/write one.
N)r r r r s r clear_cachezFileSystemCache.clear_cache s+
t'((( r c r t j | S r )hashlibsha256encode hexdigestrN s r hash_namezFileSystemCache.hash_name s& ~cjjll++55777r N)r2
__module__ __qualname__r( MODEL_META_FILE_NAMEMODEL_META_MODEL_IDMODEL_VERSION_FILE_NAMEstrr r! r rI rK rP rS rV r) rZ r` rM r r r
r
s M! # "( ( ( 4 */ / /
% % % 8 8 8 8 8r r
c r e Zd ZdZd fd Zd Zd ZdefdZd Z d Z
d
Z d Z d Z
d
Zd Zd Zd Z xZS )ModelFileSystemCachezLocal cache file layout
cache_root/owner/model_name/individual cached files and cache index file '.mcs'
Save only one version for each file.
Nc ||St t j | | nlt t j ||| t j |d|i| _ | |
| _ dS )a\ Put file to the cache
Args:
cache_root(`str`): The modelscope local cache root(default: ~/.cache/modelscope/hub)
owner(`str`): The model owner.
name('str'): The name of the model
Returns:
Raises:
None
model_id = {owner}/{name}
N/) superr r r& r' load_model_metar
rd
model_metasave_model_metaload_model_versioncached_model_revision)r
cache_rootownername __class__s r r z ModelFileSystemCache.__init__ s =DL GG RW\\* 5 5 6 6 6 " " " " GG RW\\*eT B B C C C 3tt5L DO
" " "%)%<%<%>%>"""r c p t j | j t j }t j | r t |d 5 }t j | }d d d n # 1 swxY w Y t |t r || _ d S t d| n]# t $ rP}t d|| t j ||dz n# t" $ r Y nw xY wY d }~nd }~ww xY wt j di| _ d S )Nr# z-Model meta %s has unexpected type, resetting.z,Failed to load model meta %s: %s. Resetting.r$ unknown)r r& r' r r
rc r) r* r+ r, r- dictrm r/ r0 r3 r4 r5 rd )r meta_file_pathr7 r8 r9 s r rl z$ModelFileSystemCache.load_model_meta s d&>&5&JL L
7>>. ) )
.$// *1!;q>>D* * * * * * * * * * * * * * * dD)) &*DOF N -/ / / /
M -q2 2 2J~~ /LMMMM D
+> Jsf C B5 C BC B C *C
D D-DD
DDDDD c B t j | j t j }t j | rPt |d 5 }| cd d d S # 1 swxY w Y d S d S )Nr)
r r& r' r r
re r) r* readstrip)r model_version_file_pathr7 s r ro z'ModelFileSystemCache.load_model_version s "$',, $o&M#O #O
7>>1 2 2 -s33
(qvvxx~~''
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
( 4s &BBB
revision_infoc ^ t j | j t j }t
|d 5 }t |t r*d|d d|d }| | n| | d d d d S # 1 swxY w Y d S )Nwz Revision:Revisionz ,CreatedAt: CreatedAt)
r r& r' r r
re r* r- rw write)r r~ r} r7 version_info_strs r save_model_versionz'ModelFileSystemCache.save_model_version s "$',, $o&M#O #O
)3
/
/ '1-..
'
'!*---}[/I/I$K ())))
&&&
' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' 's AB""B&)B&c 0 | j t j S r )rm r
rd r s r get_model_idz!ModelFileSystemCache.get_model_id s BCCr c t j | j t j }t
|d 5 }t j | j | d d d d S # 1 swxY w Y d S )Nr= )
r r& r' r r
rc r* r+ rA rm )r rx r7 s r rn z$ModelFileSystemCache.save_model_meta s d&>&5&JL L
.$
'
' ,1 K + + + , , , , , , , , , , , , , , , , , ,s A((A,/A,c | j D ]q}||d k rct j | j |d }t j | r|c S | | rdS )zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
Returns:
path: the full path of the file.
PathN)r% r r& r' r r) rV )r file_path cached_filecached_file_paths r get_file_by_pathz%ModelFileSystemCache.get_file_by_path s , 1 1KK ///#%7<<0H0;F0C$E $E 7>>"233 1++++OOK000tr c d | j D ]}||d k r|d | s| |d rct j | j |d }t j | r|c S | | dS )zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
commit_id (str): The commit id of the file
Returns:
path: the full path of the file.
r r N)r%
startswithr r& r' r r) rV )r r commit_idr r s r get_file_by_path_and_commit_idz3ModelFileSystemCache.get_file_by_path_and_commit_id s , 1 1KK ///J'229== 0AJAUAUVablVmAnAn 0#%7<<0H0;F0C$E $E 7>>"233 1++++OOK000tr c | | }| j D ]m}||k ret j | j |d }t j | r|c S | | nndS )zCheck if exist cache file.
Args:
model_file_info (ModelFileInfo): The file information of the file.
Returns:
str: The file path.
r N)$_ModelFileSystemCache__get_cache_keyr% r r& r' r r) rV )r model_file_info cache_keyr orig_paths r get_file_by_infoz%ModelFileSystemCache.get_file_by_info s (( 99 , Ki'' GLL)A)4V)< > > 7>>),, $$$$OOK000E ( tr c ( |d |d d}|S )Nr r )r r rM )r r r s r __get_cache_keyz$ModelFileSystemCache.__get_cache_key2 s& #F+'
3
r c | | }d}|d }t j | j |d }| j D ]}|d |k r|d |d s!|d |d ro|t }|`t j | rAt rt | }n|}||k rd} n t d| d |r6t j | rdS |
| dS )a Check the file is cached or not. Note existence check will also cover digest check
Args:
model_file_info (CachedFileInfo): The cached file info
Returns:
bool: If exists and has the same hash, return True otherwise False
Fr r NTzFile [z?] exists in cache but with a mismatched hash, will re-download.)r r r& r' r r% r r r) enable_default_hash_validationr r/ inforV ) r r rO is_existsr cache_file_path
cached_key
expected_hashcache_file_sha256s r r) zModelFileSystemCache.exists9 sr ""?33 K ',,t'?'6v'>@ @ + J&!Y.. z*55c*oFF /:11*Z2HII / !0 :
, '2) 2), 6 :,8 ,I,I)),9)$(999$( oY o o o %w~~o..
%t#% % %ur c * | j D ]}|d |d k rv| | t j | j |d }t j | rt j | dS dS )zWe in cache, remove it.
Args:
model_file_info (ModelFileInfo): The model file information from server.
r N)r% rV r r& r' r r) rU )r r r r s r remove_if_existsz%ModelFileSystemCache.remove_if_existsb s , K6"of&=== ,,, GLL)A)4V)< > > 7>>),, )Ii(((
> r c | | | | }t j | j |d }t j | }t j | st j |d t || | j
5 | j | |
ddd n # 1 swxY w Y |S )a[ Put model on model_file_location to cache, the model first download to /tmp, and move to cache.
Args:
model_file_info (str): The file description returned by get_model_files.
model_file_location (str): The location of the temporary file.
Returns:
str: The location of the cached file.
r Tr N)r r r r& r' r dirnamer) r r r r% appendrI )r r model_file_locationr cache_full_pathcache_file_dirs r rS z ModelFileSystemCache.put_fileq s+
o...(( 99 ',,t'?'0'8: : 99w~~n-- 7 K 6 6 6 6
/222
/ / $ $Y / / / , , . . . / / / / / / / / / / / / / / / s /C<<D D )NN)r2 ra rb __doc__r rl ro r r r rn r r r r r) r rS
__classcell__)rt s @r rh rh s
? ? ? ? ? ?2K K K* ' ' ' ' 'D D D, , , ( , , ' ' 'R
r rh )r\ r r+ r> r shutilr r typingr modelscope.hub.constantsr r modelscope.hub.utils.utilsr modelscope.utils.loggerr r/ getenvr| lowerr objectr
rh rM r r r se
: : : : : : : : 3 3 3 3 3 3 . . . . . . BI7AAGGIIOOQQU[[ F8 F8 F8 F8 F8f F8 F8 F8Re e e e e ? e e e e e r