vj hddlZddlmZddlmZmZddlZddl m Z m Z ddl m Z GddZdS)N)abstractmethod)ListUnion) read_configsnapshot_download)ConfigceZdZdefdZedeeeeeefdeefdZ defdZ e defd Z e d d Zd S)InferFrameworkmodel_id_or_dirc ^tj|r||_nt ||_||}t |j}|d}||p||}tj|jd}tj |rBtj |}|d}||p||}|std|dS)z Args: model_id_or_dir(`str`): The model id of the modelhub or a local dir containing model files. z model.typeNz config.json model_typez"Model accelerating not supported: ) ospathexists model_dirrmodel_type_supportedrsafe_getjoinisfiler from_file ValueError)selfr kwargsmodel_supportedconfigr config_files t/lsinfo/ai/hellotax_ai/data_center/backend/venv/lib/python3.11/site-packages/modelscope/pipelines/accelerate/base.py__init__zInferFramework.__init__ s5 7>>/ * * @,DNN.??DN33ODD$T^44__\22  !-1J1J22Ogll4>=AA 7>>+ & & %k22F66J%"1# T5N5N6 6  HF_FFHH H H Hpromptsreturnc dS)z Args: prompts(`Union[List[str], List[List[int]]]`): The string batch or the token list batch to input to the model. Returns: The answers in list according to the input prompt batch. N)rr rs r__call__zInferFramework.__call__)s  rr cdS)NFr#)rr s rrz#InferFramework.model_type_supported5sur major_versioncPtj\}}||kS)z%Check the GPU compatibility. )torchcudaget_device_capability)r&major_s rcheck_gpu_compatibilityz&InferFramework.check_gpu_compatibility8s&:3355q %%rvllmc b|dkrddlm}||fi|}||_|Std|)aInstantiate the model wrapped by an accelerate framework. Args: model_id_or_dir(`str`): The model id of the modelhub or a local dir containing model files. framework(`str`): The framework to use. Returns: The wrapped model. r.)VllmzFramework not supported: )r.r1 llm_frameworkr)clsr frameworkrr1r.s rfrom_pretrainedzInferFramework.from_pretrained?s\    " " " " " "422622D!*D KDDDEE ErN)r.)__name__ __module__ __qualname__strrrrrintr$r staticmethodr- classmethodr5r#rrr r sHHHHH8  d3id3i&@ A  "3i    ^  s&s&&&\& FFF[FFFrr )os.pathrabcrtypingrr torch.cudar( modelscoperrmodelscope.utils.configrr r#rrrCs55555555******CFCFCFCFCFCFCFCFCFCFr