ujɥ'ddlZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZddlmZddlmZmZmZmZmZmZdd lmZdd lmZdd l m!Z!d d l"m#Z#m$Z$d dl%m&Z&d dl'm(Z(d dl)m*Z*m+Z+d dl,m-Z-m.Z.m/Z/d dl0m1Z1d dl2m3Z3m4Z4m5Z5m6Z6m7Z7eZ8dZ9dddddddddddddeddddfde:dee:dee:e dfdeeee:fdee;dee deee:ee:fdeee:ee:fdee:deeee:e:fd eeee:e:fd!ee<d"e:d#ee:d$ee;d%eee&d&ee:d'ee:d(e:f&d)Z=eddddddddddd*ddfd+e:dee:dee:e dfdee:deeee:fdee;dee deee:ee:fdeee:ee:fdeeee:e:fd eeee:e:fd$ee;d!ededddddddddd*dddd-d"e:d#ee:dee:dee:e dfdeeee:fdee;dee deee:ee:fdeee:ee:fdee:deeee:e:fd eeee:e:fd!ee:d?e:d#ee:dee:dee deee:ee:fdeee:ee:fdeeee:e:fd eeee:e:fd!e >@@ $-$9yy?W@ @  BGLLw77$GGGGGLLw!(e!>  ++-- ! !- 31+)#1!s)EEE dataset_idc| -ttjdd} | r||n t }tjtj|ddtj|d|dd}t|}nt}|5t|t||||||||| | | | | cdddS#1swxYwYdS) a Download raw files of a dataset. Downloads all files at the specified revision. This is useful when you want all files from a dataset, because you don't know which ones you will need a priori. All files are nested inside a folder in order to keep their actual filename relative to that folder. An alternative would be to just clone a dataset but this would require that the user always has git and git-lfs installed, and properly configured. Args: dataset_id (str): A user or an organization name and a dataset name separated by a `/`. revision (str, optional): An optional Git revision id which can be a branch name, a tag, or a commit hash. NOTE: currently only branch and tag name is supported cache_dir (str, Path, optional): Path to the folder where cached files are stored, dataset will be save as cache_dir/dataset_id/THE_DATASET_FILES. local_dir (str, optional): Specific local directory path to which the file will be downloaded. user_agent (str, dict, optional): The user-agent info in the form of a dictionary or a string. local_files_only (bool, optional): If `True`, avoid downloading the file and return the path to the local cached file if it exists. cookies (CookieJar, optional): The cookie of the request, default None. ignore_file_pattern (`str` or `List`, *optional*, default to `None`): Any file pattern to be ignored in downloading, like exact file names or file extensions. Use regression is deprecated. allow_file_pattern (`str` or `List`, *optional*, default to `None`): Any file pattern to be downloading, like exact file names or file extensions. allow_patterns (`str` or `List`, *optional*, default to `None`): If provided, only files matching at least one pattern are downloaded, priority over allow_file_pattern. For hugging-face compatibility. ignore_patterns (`str` or `List`, *optional*, default to `None`): If provided, files matching any of the patterns are not downloaded, priority over ignore_file_pattern. For hugging-face compatibility. enable_file_lock (`bool`): Enable file lock, this is useful in multiprocessing downloading, default `True`. If you find something wrong with file lock and have a problem modifying your code, change `MODELSCOPE_HUB_FILE_LOCK` env to `false`. max_workers (`int`): The maximum number of workers to download files, default 8. token (str, optional): The user token. Raises: ValueError: the value details. Returns: str: Local folder path (string) of repo snapshot Note: Raises the following errors: - [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError) if `use_auth_token=True` and the token cannot be found. - [`OSError`](https://docs.python.org/3/library/exceptions.html#OSError) if ETag cannot be determined. - [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) if some parameter value is invalid Nr>r?r@TrArCrD)r5r)r*r+r,r-r.r/r0r2r1r3r8r9) r%rFrGrHrrIrJrKrLr&rrMr)rTr)r*r0r+r,r-r.r/r1r2r6r3r8r9rNrOrPs rQdataset_snapshot_downloadrVsjH$ JNN5v > >@@ $-$9yy?W@ @  BGLLw77$GGGGGLLw!+!3!3C!?!?AA  ++-- ! '!- 31+)#s #C==DD)r5r)r*r+r,r-r.r/r0r1r2r3r7r8r9c |st}|tvrtd|dtt|| ||\}}||n t }|rXt |jdkrtdt d|z| Stj |ttjd}t d krXt#jd pt'}|r"td |||d <t'| }|||||}||}|tkr| r t"j| }nX|r+t#jj|g|dR}n+t#jj|dg|dR}t9d|d||||||}|d}dt"jvr i|ddi}ni|ddi}|j |j|d<tAtC|tC| }|"|||d|dn|||}|s@|>t d|d|"||dd|dn|||}tG|||| | }tI|||||dd||||d| || !d"|vrtK||}t"j&|rtd#ntd$|d% t#j't"j||d&n#tP$rBt d'|d(t"j|d"Yn5wxYwn/|tRkr#| r t"j| }nX|r+t#jj|g|dR}n+t#jj|d)g|dR}t9d*|tU|\}}|ptV}tAtC|tC| }|rd|znd}t9d+|d,tY||||||||| | - }t[||||||||||| || . |.|/| } | S)0Nr<r=)r0r*r5rzCannot find the requested files in the cached path and outgoing traffic has been disabled. To enable look-ups and downloads online, set 'local_files_only' to False.z6We can not confirm the cached file is for revision: %s)r+)z user-agentzsnapshot-identifierr?INTRA_CLOUD_ACCELERATION_REGIONz6Intra-cloud acceleration enabled for downloading from zx-aliyun-region-id)r8)r4r5r8rCmodelszDownloading Model from z to directory: )r)r-r9RevisionCI_TESTzsnapshot-ci-testTrueSnapshotcached_model_revision)r/r1TF)r(r)root recursive use_cookiesheadersr9zroot='zF' returned no model files, falling back to root=None for full listing.r/r.r1r2)r5r)r- pre_filteredr3r9r7.z3Target directory already exists, skipping creation.zCreating symbolic link [z].)target_is_directoryzFailed to create symbolic link z for datasetsz"Downloading Dataset to directory: zFetching file list (root: z)...)r8 root_pathr/r.r1r2) cachetemporary_cache_dirr4api dataset_name namespacerbr)r-r3r9r7) revision_info)/rrrrrlen cached_filesrEloggerwarningget_root_locationrget_user_agentstruuiduuid4rrFgetenvr!_get_internal_acceleration_domaininfoget_endpoint_for_read get_cookiesrJabspathrKsplitprintget_valid_revision_detailrGr^r"_normalize_patternsget_model_filesfilter_files_by_patterns_download_file_listsr#existssymlinkOSErrorrr$r_iter_dataset_file_pages_pipeline_download_datasetsave_model_version)!r4r5r)r*r+r,r-r.r/r0r1r2r3r7r8r9rjrirNrb region_id_api directoryrevision_detailsnapshot_headerextracted_root repo_filesmasked_directorygroup_or_ownernamerhfile_page_itercache_root_paths! rQrMrMsD& $# ))) ) +<+< >?? ?"F9 Y"P"P"P ) 599;S<<Lq u! " "a ' '<== = O!" # # #&&    +z D D D#&tz||#4#4   $v - - ;<<@88==??  : VWVV1:,-E"""  119E2CCH ?&&((G  ' ' >GOOI66  >GLK c8J8JKKK GLx>*1--*<*<>>> N(NN9NN P P P"<<(Gh=PPO&z2HBJ&&"MW"M1CV0L"M"EW"EV0D"E*6/4/J +-8#67I#J#J2>BBDDDN-- !#%,_EE''!.##J '."<C^CCCDDD"11$%")0g+%2'' 2#5$7- / 111J !##!!'!#5    "g~~#=w$($( 7>>),, KKMOOOOKK H9 H H HIII GOO,<==%0466666#rirrbgoo^nNoNorrr$+ + + >GOOI66  >GLK c8J8JKKK GLz>*1--*<*<>>> ByBB C C C#?#H#H ND&B*BO8#67I#J#J2>BBDDDN1?Gn,,CI >y>>> ? ? ?5##5$7- / 1 1 1N '$7!(!'!#5 7 7 7 7    ???1133s!4OAP#"P#rCc t dvrtdddd\} } | | \} t g f d}||}|s/|dkr)td|d|d}|S) awFetch and filter dataset repo files with pagination and server-side prefix filtering. Applies per-page pattern filtering to minimize memory usage. Falls back to root_path='/' if the extracted prefix yields no results. Args: _api: HubApi instance. repo_id: Dataset repo identifier (owner/name). revision: Git revision. endpoint: API endpoint URL. token: Authentication token. root_path: Server-side directory prefix filter. allow_file_pattern: Include patterns for client-side filtering. ignore_file_pattern: Exclude patterns for client-side filtering. allow_patterns: Additional include patterns (HF-compatible). ignore_patterns: Additional exclude patterns (HF-compatible). page_size: Number of files per API page request. Returns: List of filtered file entry dicts. rCInvalid repo_id: '', expected format 'owner/name'rrlrmr9r8c  d}g} |d|  }n7#t$r*}td|d|Yd}~npd}~wwxYw|sne r*t | }||n|d|Dt |krn|dz }|S) zFFetch all pages with the given root_path, applying per-page filtering.rT r4r)rhr` page_number page_sizer9r8dataset_hub_id#Error fetching dataset files (page ): Nrcc3LK|]}|ddk|V dS)r treeNrH.0fs rQ zAfetch_repo_files.._paginate_and_filter...sC"I"If 0G0GA0G0G0G0G"I"I)get_dataset_files Exceptionrqerrorrextendro)effective_root_pathrr dataset_filese page_filteredr_hub_idr/r1r9 has_patternsr.r2rr4r)r8s rQ_paginate_and_filterz.fetch_repo_files.._paginate_and_filter si  %   $ 6 6#%1" +'%#*!7 !, !,     M+MM!MMOOO  !  I 8!'9(;#1$3 !5!5!5 !!-0000!!"I"I,"I"I"IIII=!!I-- 1 KK% Ns' A AAz root_path='F' returned no results, falling back to root_path='/' for full listing.)rr~get_dataset_id_and_typeanyrqrr)rr4r)r9r8rhr/r.r1r2r_owner _dataset_name_rrrrs````` ````` @@rQfetch_repo_filesrsbD ' I I I IKK K#MM#q11FM--" .JGQ /L ,,,,,,,,,,,,,,,,^&%i00J /)s**JYJJJ K K K))#.. rpatterncR tj|dS#t$rYdSwxYw)NTF)recompile BaseException)rs rQ_is_valid_regexrDs> 7t uus  &&patternscRt|tr|g}| d|D}|S)NcFg|]}|ds|n|dzS)rC*)endswith)ritems rQ z'_normalize_patterns..Ps>   ?C c** :DDs    r) isinstanceru)rs rQrrLsF(C  :  GO    Orcd|-g}|D]&}t|r||'|SdSN)rappend)rregex_patternsrs rQ_get_valid_regex_patternrVsL , ,Dt$$ ,%%d+++trrcrct|}t|}t|}t|}t|}g}|D]ddkr |rtfd|Dr.|rtfd|DrL|rtfd|Drj|rtfd|Ds|rtfd|Dsn4#t$r'}td|zYd }~d }~wwxYw||S) aFilter repo file entries by include/exclude patterns. Skips 'tree' type entries. Applies fnmatch and regex pattern matching. Returns only file entries that pass all filter criteria. Args: repo_files: List of file entry dicts with 'Type', 'Path', 'Name' keys. allow_file_pattern: Include patterns (fnmatch). Files must match at least one. ignore_file_pattern: Exclude patterns (fnmatch). Matching files are skipped. allow_patterns: Additional include patterns (HF-compatible). ignore_patterns: Additional exclude patterns (HF-compatible). Returns: List of file entries that pass all filters. r rc3NK|]}tjd|V dSrNfnmatchrp repo_files rQrz+filter_files_by_patterns..sH'.'.OIf$5q99'.'.'.'.'.'.rc3NK|]}tjd|V dSrrrs rQrz+filter_files_by_patterns..sH+2+2OIf$5q99+2+2+2+2+2+2rc3RK|]!}tj|dduV"dSNameNrsearchrs rQrz+filter_files_by_patterns..sM,3,3Ia6!2334?,3,3,3,3,3,3rc3NK|]}tjd|V dSrrrs rQrz+filter_files_by_patterns..sH*-*-OIf$5q99*-*-*-*-*-*-rc3NK|]}tjd|V dSrrrs rQrz+filter_files_by_patterns..sH.1.1OIf$5q99.1.1.1.1.1.1rzInvalid file pattern: %sN)rrrrrqrrr) rr/r.r1r2ignore_regex_patternfilteredrrs @rQrras+.*/::O(88N-.ABB,-?@@34GHHH # # V  & &   3'.'.'.'.,'.'.'.$.$. " s+2+2+2+20+2+2+2(2(2 # ,3,3,3,31,3,3,3)3)3  c*-*-*-*-+*-*-*-'-'- ! #.1.1.1.1/.1.1.1+1+1     NN59 : : : HHHH   """" Os0!C8?C8C8;C8C88 D)D$$D)c # Kdvrtdddd\} } | | \} t g f d} d}||D]}d}|V |s+|dkr%t d |d |dD]}|Vt d S#t wxYw) aGenerator that yields filtered file pages from a dataset repo. Each yield is a non-empty list of file-entry dicts for one API page. Applies per-page pattern filtering to minimize memory usage. Falls back to root_path='/' if the extracted prefix yields no results. Args: _api: HubApi instance. repo_id: Dataset repo identifier (owner/name). revision: Git revision. endpoint: API endpoint URL. token: Authentication token. root_path: Server-side directory prefix filter. allow_file_pattern: Include patterns (fnmatch). ignore_file_pattern: Exclude patterns (fnmatch). allow_patterns: Additional include patterns (HF-compatible). ignore_patterns: Additional exclude patterns (HF-compatible). page_size: Number of files per API page request. Yields: List[dict]: Non-empty list of filtered file entries per page. rCrrrrc 3 Kd}d} |d|  }n8#t$r+}td|d|Yd}~dSd}~wwxYw|sdS rt | }n d |D}|t |z }|r|Vt d |d |d d dt |krdS|dz })z2Yield filtered file pages for the given root_path.rrTrrrNrccDg|]}|ddk|S)r rrrs rQrzE_iter_dataset_file_pages.._paginate_pages..s3!!!f 0G0GA0G0G0Grz Fetched z matching files (z pages)...)endflush)rrrqrrror)rr total_foundrrrrrr/r1r9rr.r2rr4r)r8s rQ_paginate_pagesz1_iter_dataset_file_pages.._paginate_pagess  /   $ 6 6#%1" +'%#*!7 !, !,     M+MM!MMOOO  !   8!'9(;#1$3 !5!5!5 !!,!!!  3}-- -K $#### ,{,,,,,      =!!I-- 1 K_/ s) A AAFTz root_path='rN)rr~rrr)rr4r)r9r8rhr/r.r1r2rrrrr yielded_anypagerrs````` ````` @@rQrrsF ' I I I IKK K$MM#q11FM--" .JGQ /L 4444444444444444n #OI..  DKJJJJ yC// EIEEE F F F',,    s ?ACC"c d} d}gtjfdtddd5t| 5}|D]}|D]}| dz } | _||r|dz }dR||d|||| }| t|||||| d| }| |ffd  d d d n #1swxYwYd d d n #1swxYwYrVd D}t tddd|Dz| |z tz }t!d| d|d|drdtdndzdzr"t#td| dd S)a_Pipeline consumer: download dataset files as pages are yielded. Consumes the page iterator from _iter_dataset_file_pages, submitting each file to a thread pool for concurrent download. Uses tqdm for real-time progress and thread-safe error collection. Args: file_page_iter: Iterator yielding List[dict] file pages. cache: ModelFileSystemCache instance for dedup. temporary_cache_dir: Temp staging directory. repo_id: Dataset repo identifier. api: HubApi instance. dataset_name: Dataset name component. namespace: Owner/namespace component. headers: HTTP request headers. revision: Git revision. cookies: HTTP cookies. max_workers: Thread pool concurrency. endpoint: API endpoint URL. progress_callbacks: Optional progress callback list. rc |n|#t$ro}5||fdddn #1swxYwYtd|ddd|Yd}~nd}~wwxYwddS#dwxYw)z8Done callback: update progress bar and collect failures.NzDownload failed for r?z: r)resultrrrqdebugrHupdate)futurerexc failed_itemslockpbars rQ_on_donez,_pipeline_download_dataset.._on_doneHs/  MMOOOO L L L 6 6##Y$4555 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 LLJy}}VS'A'AJJSJJ L L L L L L L L L KKNNNNNDKKNNNNsMB+ BB A B A B A 7B B+ BB++Cz filesF)totalunitdisable)r3rr file_namerlrmr)r9 disable_tqdmr7c||Sr)rrfrs rQz,_pipeline_download_dataset..xsBrNcg|]?\}}t|tr|ddnt|@S)rr)rdictrHrurrrs rQrz._pipeline_download_dataset..~sT   a&0d%;%; JDHHVS ! ! !T   r file(s) failed to download:  c3 K|] }d|V dSz - Nrrrs rQrz-_pipeline_download_dataset..s( B B B B B B B BrzDownload complete: z files found, z cached, z downloadedz, z failedrre# file(s) failed to download out of ) threadingLockr rrrefreshrrget_dataset_file_urlsubmitradd_done_callbackrqrrorKrr)rrirjr4rkrlrmrbr)r-r3r9r7r total_cachedexecutor page_filesrurlr failed_paths downloadedrrrrs @@@@rQrrsHKLL >  D        AHe 4 4 4#A K 8 8 8" AH,! A! A !+ A AI1$K!,DJLLNNN||I..!$)  A 22"+F"3%1"+!)!) 3++C&__%!+%*+=-  F,,%.?????AAAA? A! A" A" A" A" A" A" A" A" A" A" A" A" A" A" A" A#A#A#A#A#A#A#A#A#A#A#A#A#A#A#APD  '     L))IIIyy B B\ B B BBBC D D D|+c,.?.??J  <  < <  < <$. < < <0< D ,#l## , , , ," FHK LMMM<        s6D$B8D  D$ D D$D D$$D(+D(rirjrkrrrdc  |rwg}|D]prDtjd}td|d[|qnt|}t| } t| } t| } t| }g}|D]Pddkr |rtfd|Dr/| rtfd| DrM|rtfd|Drk| | rtfd | Ds| | rtfd | DsrEtjd}td|d |#t$r(}t d |zYd}~Jd}~wwxYwt|d d   f d}t|dkrtdt|d||}g}t|t rpt|dkr]|\}}|rVd|D}tt|ddd|Dztdt|dd|r1t't|dt|ddSdS)NrzFile z8 already in cache with identical hash, skip downloading!r rcFg|]}tjd|Srrrrrs rQrz(_download_file_lists..s:,,,#  &(97CC,,,rcFg|]}tjd|Srrrs rQrz(_download_file_lists..s:000#  &(97CC000rcJg|]}tj|ddu Srrrs rQrz(_download_file_lists..s?111# '9V+<==TI111rc3NK|]}tjd|V dSrrrs rQrz'_download_file_lists..sH;; '$OIf,=wGG;;;;;;rc3NK|]}tjd|V dSrrrs rQrz'_download_file_lists..sH?? '$OIf,=wGG??????rz The file pattern is invalid : %sFT)r3rfault_tolerantc   tkrt |d }nF tkr!|d }nt d dt t || d dS)Nr)r( file_pathr)r9rr<z, supported types: Fr)rr rrrrr)rrrkrir-r9rrbrr7r4r5r)rjs rQ_download_single_filez3_download_file_lists.._download_single_files  ' '' #F+!! ###CC + + +**#F+!(!! +##CC#WiWWDUWW        1 rrzGot z files, start to download ...clg|]1\}}t|tr|dnt|2Sr)rrrurs rQrz(_download_file_lists..sI   a%/tT$:$:IDLLD    rrrc3 K|] }d|V dSrrr s rQrz'_download_file_lists..s(AAq q AAAAAArzFinish downloading z files for repo ''r re)rrFrJbasenamerqrrrrrrrrrrorzrtuplerrKr)rrirjr4rkrrrbr5r)r-r.r/r1r2r3r9r7rdfiltered_repo_filesrrrr!download_resultrrrrs `````````` `` @rQrrsK*A6 # 2 2I||I&& G,,Yv->??  _I___  & &y 1 1 1 1 2.o>>,^<<12EFF01CDD78KLL #, 6, 6I F**) 6"s,,,,'6,,,((&30000':000,,'C1111';111--!-.-;;;;+9;;;;;!!%16H1????+=?????!!<< ** " 0 061B C CILLc ccc $**95555 G G GAAEFFFFFFFF G eDJJJ               JJ < !## J3*++ J J J L L L//0CDD ou - - C#o2F2F!2K2K-OA| C  #/     <((HHHiiAALAAAAABCCC  W#&9":": W WW W W W     0#|$$//*++///00 0/$#, 0 0s7'G3G3#G3G3!G3AG33 H%=H  H%)IrrFrr rvconcurrent.futuresr contextlibrhttp.cookiejarrpathlibrtypingrr r r r tqdm.autor modelscope.utils.constantrrrrrrmodelscope.utils.file_utilsrmodelscope.utils.loggerrmodelscope.utils.thread_utilsrrkrrcallbackr constantsrerrorsrr file_downloadrrr utils.cachingr! utils.utilsr"r#r$r%r&rqDEFAULT_DATASET_PAGE_SIZEruboolintrRrVrMrrrrrrrrrrrrQr=s   111111""""""$$$$$$44444444444444:::::::::::::::: A@@@@@......999999))))))))&&&&&&******77777777BBBBBBBBBB//////************** "(,-1',#';?:>#6:7;!%.'+7;"%xxxsmxS$_%xtSy)* x tn x i x"%T#Y"78x!sDI~!67x}xU49c>23xeDIsN34x#xx}xtnx T"234!x" C=#x$sm%x& 'xxxxz7(,#-1',#';?:>6:7;'+"aaasmaS$_%a} a tSy)* a tn ai a"%T#Y"78a!sDI~!67aU49c>23aeDIsN34atnaa C=asma  !aaaaN $4(,-1',#';?:>#6:7;7;"#NNN N}Nsm N S$_% N tSy)* NtnNi N"%T#Y"78N!sDI~!67N}NU49c>23NeDIsN34NNT"234N C=!N"sm#NNNNl 'hhhhVS%T#Y"7tCy/3/3*.+/ @@@T @!c+@"$s), @ T#Y' @ d3i( @ $Z@@@@P 'xxxxL$ oooov $4#';?:>6:7;"7;'Q0Q0S Q0 Q0Q0 Q0  Q0  Q0Q0}Q0smQ0i Q0"%T#Y"78Q0!sDI~!67Q0U49c>23Q0eDIsN34Q0 !Q0"sm#Q0$T"234%Q0&'Q0Q0Q0Q0Q0Q0r