yj/ d Z ddlZddlZddlZddlZddlmZmZ ddlm Z ddl
m Z m Z m
Z
ddlmZ ddlmZ ej e d d
k Z G d d e Z G d
de ZdS )z
caching tools
N)movermtree)Dict) FILE_HASH'AISTUDIO_ENABLE_DEFAULT_HASH_VALIDATION CACHE_KEY ) compute_hash)logFalsetruec f e Zd ZdZdZdZdZdZ defdZ d Z
d Z d
Z d Z
d Zd
Zd Zd Zd ZdS )FileSystemCacheinfoz.mscz.mdlidz.mvcache_root_locationc h t j |d || _ | dS )zBase file system cache interface.
Args:
cache_root_location (str): The root location to store files.
kwargs(dict): The keyword arguments.
Texist_okN)osmakedirsr
load_cache)selfr s j/lsinfo/ai/hellotax_ai/data_center/backend/venv/lib/python3.11/site-packages/aistudio_sdk/utils/caching.py__init__zFileSystemCache.__init__ s8 '$7777#6 c | j S )location)r r s r get_root_locationz!FileSystemCache.get_root_location- s ''r c 8 g | _ t j | j t
j }t j | rDt |d 5 }t j
| | _ ddd dS # 1 swxY w Y dS dS )loadrbN) cached_filesr pathjoinr r
KEY_FILE_NAMEexistsopenpickler" )r cache_keys_file_pathfs r r zFileSystemCache.load_cache1 s !w||D,D,;,I K K
7>>. / / 3*D11
3Q$*KNN!
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3 3 3s &B
BBc 4 t j | j t j }t
j \ }}t |d 5 }t j
| j | ddd n # 1 swxY w Y t || dS )zSave cache metadata.wbN)
r r% r& r r r' tempfilemkstempr) r* dumpr$ r )r r+ fdfnr, s r save_cached_filesz!FileSystemCache.save_cached_files: s "w||D,D,;,I K K !##B
"d^^ .q K)1 - - - . . . . . . . . . . . . . . . R%&&&&&s A==BBc dS )zCheck the key is in the cache, if exists, return the file, otherwise return None.
Args:
key(str): The cache key.
Raises:
None
N r keys r get_filezFileSystemCache.get_fileD s
r c dS )zPut file to the cache.
Args:
key (str): The cache key
location (str): Location of the file, we will move the file to cache.
Raises:
None
Nr6 )r r8 r s r put_filezFileSystemCache.put_fileO s
r c x || j v r0| j | | dS dS )zvRemove cache key in index, The file is removed manually
Args:
key (dict): The cache key.
N)r$ remover4 r7 s r
remove_keyzFileSystemCache.remove_key[ sI $# # # $ $S ) ) ) " " $ $ $ $ $ $ #r c , | j D ] }||k r dS dS )checkTF)r$ )r r8
cache_files r r( zFileSystemCache.existse s2 + JS tt ! ur c V t | j | dS )zRemove all files and metadata from the cache
In the case of multiple cache locations, this clears only the last one,
which is assumed to be the read/write one.
N)r r r r s r clear_cachezFileSystemCache.clear_cachem s+
t'((( r c r t j | S )name)hashlibsha256encode hexdigestr7 s r hash_namezFileSystemCache.hash_nameu s& ~cjjll++55777r N)__name__
__module__ __qualname____doc__r' MODEL_META_FILE_NAMEMODEL_META_MODEL_IDMODEL_VERSION_FILE_NAMEstrr r r r4 r9 r; r> r( rC rJ r6 r r r r s JM! # ( ( (3 3 3' ' '
% % % 8 8 8 8 8r r c r e Zd ZdZd fd Zd Zd ZdefdZd Z d Z
d
Z d Z d Z
d
Zd Zd Zd Z xZS )ModelFileSystemCachezLocal cache file layout
cache_root/owner/model_name/individual cached files and cache index file '.mcs'
Save only one version for each file.
Nc ||St t j | | nlt t j ||| t j |d|i| _ | |
| _ dS )aU Put file to the cache
Args:
cache_root(`str`): The aistudio local cache root(default: ~/.cache/aistudio/)
owner(`str`): The model owner.
name('str'): The name of the model
Returns:
Raises:
None
model_id = {owner}/{name}
N/) superr r r% r& load_model_metar rP
model_metasave_model_metaload_model_versioncached_model_revision)r
cache_rootownerrE __class__s r r z ModelFileSystemCache.__init__ s =DL GG RW\\* 5 5 6 6 6 " " " " GG RW\\*eT B B C C C 3tt5L DO
" " "%)%<%<%>%>"""r c P t j | j t j }t j | rDt |d 5 }t j | | _
ddd dS # 1 swxY w Y dS t j di| _
dS )zget metar# Nunknown) r r% r& r r rO r( r) r* r" rY rP r meta_file_pathr, s r rX z$ModelFileSystemCache.load_model_meta s d&>&5&JL L
7>>. ) ) Ond++
1q"(+a..
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1 / BI NDOOOs BB
B
c B t j | j t j }t j | rPt |d 5 }| cddd S # 1 swxY w Y dS dS )zuse version inforN)
r r% r& r r rQ r( r) readstrip)r model_version_file_pathr, s r r[ z'ModelFileSystemCache.load_model_version s "$',, $o&M#O #O
7>>1 2 2 -s33
(qvvxx~~''
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
(
( 4s &BBB
revision_infoc | t j | j t j }t
|d 5 }t |t r9d|d d| d pd}|
| n|
| ddd dS # 1 swxY w Y dS )zsave file infowz Revision:Revisionz ,CreatedAt: CreatedAtra N) r r% r& r r rQ r)
isinstancedictgetwrite)r ri rh r, version_info_strs r save_model_versionz'ModelFileSystemCache.save_model_version s "$',, $o&M#O #O
)3
/
/ '1-..
'
'!*---}/@/@/M/M/ZQZ/Z$\ ())))
&&&
' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' 's A$B11B58B5c 0 | j t j S )rp )rY r rP r s r get_model_idz!ModelFileSystemCache.get_model_id s BCCr c t j | j t j }t
|d 5 }t j | j | ddd dS # 1 swxY w Y dS )saver. N)
r r% r& r r rO r) r* r1 rY rb s r rZ z$ModelFileSystemCache.save_model_meta s d&>&5&JL L
.$
'
' ,1 K + + + , , , , , , , , , , , , , , , , , ,s A((A,/A,c | j D ]q}||d k rct j | j |d }t j | r|c S | | rdS )zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
Returns:
path: the full path of the file.
PathN)r$ r r% r& r r( r> )r file_path cached_filecached_file_paths r get_file_by_pathz%ModelFileSystemCache.get_file_by_path s , 1 1KK ///#%7<<0H0;F0C$E $E 7>>"233 1++++OOK000tr c d | j D ]}||d k r|d | s| |d rct j | j |d }t j | r|c S | | dS )zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
commit_id (str): The commit id of the file
Returns:
path: the full path of the file.
ry rl N)r$
startswithr r% r& r r( r> )r rz commit_idr{ r| s r get_file_by_path_and_commit_idz3ModelFileSystemCache.get_file_by_path_and_commit_id s , 1 1KK ///J'229== 0AJAUAUVablVmAnAn 0#%7<<0H0;F0C$E $E 7>>"233 1++++OOK000tr c | | }| j D ]m}||k ret j | j |d }t j | r|c S | | nndS )zCheck if exist cache file.
Args:
model_file_info (ModelFileInfo): The file information of the file.
Returns:
str: The file path.
ry N)$_ModelFileSystemCache__get_cache_keyr$ r r% r& r r( r> )r model_file_info cache_keyr{ orig_paths r get_file_by_infoz%ModelFileSystemCache.get_file_by_info s (( 99 , Ki'' GLL)A)4V)< > > 7>>),, $$$$OOK000E ( tr c ( |d |d d}|S )Nr% sha)ry rl r6 )r r r s r __get_cache_keyz$ModelFileSystemCache.__get_cache_key s& #F+'.
r c | | }d}|d }t j | j |d }| j D ]}|d |k r|d |d s!|d |d ri|t }|Zt j | r;t rt | }n|}||k rd} nt j d| d |r6t j | rdS |
| dS ) a Check the file is cached or not. Note existence check will also cover digest check
Args:
model_file_info (CachedFileInfo): The cached file info
Returns:
bool: If exists and has the same hash, return True otherwise False
Fry r% rl NTzFile [z?] exists in cache but with a mismatched hash, will re-download.)r r r% r& r r$ r r r( enable_default_hash_validationr
r r r> ) r r r8 is_existsrz cache_file_path
cached_key
expected_hashcache_file_sha256s r r( zModelFileSystemCache.exists sp ""?33 K ',,t'?'6v'>@ @ + J&!Y.. z*55c*oFF /:11*Z2HII / !0 :
, '2) 2), 6 :,8 ,I,I)),9)$(999$( oY o o o %w~~o..
%t#% % %ur c * | j D ]}|d |d k rv| | t j | j |d }t j | rt j | dS dS )zWe in cache, remove it.
Args:
model_file_info (ModelFileInfo): The model file information from server.
ry r% N)r$ r> r r% r& r r( r= )r r r{ rz s r remove_if_existsz%ModelFileSystemCache.remove_if_exists4 s , K6"of&=== ,,, GLL)A)4V)< > > 7>>),, )Ii(((
> r c | | | | }t j | j |d }t j | }t j | st j |d t || | j
| | |S )a[ Put model on model_file_location to cache, the model first download to /tmp, and move to cache.
Args:
model_file_info (str): The file description returned by get_model_files.
model_file_location (str): The location of the temporary file.
Returns:
str: The location of the cached file.
ry Tr )
r r r r% r& r dirnamer( r r r$ appendr4 )r r model_file_locationr cache_full_pathcache_file_dirs r r; z ModelFileSystemCache.put_fileC s
o...(( 99 ',, $ f 99w~~n-- 7 K 6 6 6 6
/222 +++ r )NN)rK rL rM rN r rX r[ r rs ru rZ r} r r r r( r r;
__classcell__)r_ s @r rT rT z s
? ? ? ? ? ?2O O O
'
'
'
'
'D D D, , , ( , , ' ' 'R
r rT )rN rF r r* r/ shutilr r typingr aistudio_sdk.configr r r utilr
aistudio_sdkr getenvrg lowerr objectr rT r6 r r r sl
C C C C C C C C C C BI5w??EEGGMMOOSYY a8 a8 a8 a8 a8f a8 a8 a8H_ _ _ _ _ ? _ _ _ _ _ r