o Q5iD=@sddlmZddlmZddlmZmZmZmZddl m Z ddl m Z m Z ddlmZddlmZmZmZmZmZGdd d ZGd d d Zd S) ) annotations) Semaphore)IterableIteratorOptionalUnion) EngineArgs) BatchHandler select_model)logger)ClassifyReturnTypeEmbeddingReturnTypeImageClassTypeModelCapabilitesRerankReturnTypec@seZdZdZ  dFdGdd ZedHd dZdIddZddZddZ ddZ ddZ ddZ dJddZ edJdd ZedKd"d#ZedLd$d%Z dMdNd+d,Zd-dd.dOd5d6Zd-d7dPd9d:Zdd;dQd>d?Zdd;dRdBdCZdDdEZdS)SAsyncEmbeddingEnginea An LLM engine that receives requests and embeds them asynchronously. This is the main worker of the infinity-emb library. It is responsible for handling the requests and embedding them asynchronously. Initialize via `from_args` method. NTmodel_name_or_path Optional[str]returnNonecKsT|rtd|dur||d<tdi||_d|_d|_t|j\|_|_|_ dS)zvCreating a Async EmbeddingEngine object. preferred way to create an engine is via `from_args` method. zaAsyncEmbeddingEngine() is deprecated since 0.0.25. Use `AsyncEmbeddingEngine.from_args()` insteadNrF) r warningr _engine_argsrunning_running_sepamorer _model_replicas_min_inference_t_max_inference_t)selfr_show_deprecation_warningkwargsrra/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/infinity_emb/engine.py__init__#s zAsyncEmbeddingEngine.__init__ engine_argsr'AsyncEmbeddingEngine'cCs*td||di|ddi}|S)zpcreate an engine from EngineArgs Args: engine_args (EngineArgs): EngineArgs object z'Creating AsyncEmbeddingEngine from `%s`rFNr)r debugto_dict)clsr#enginerrr! from_args<s zAsyncEmbeddingEngine.from_argsstrcCs$d|jd|j|jgd|jdS)NzAsyncEmbeddingEngine(running=z, inference_time=z, ))rrrrrrrr!__str__Ks  zAsyncEmbeddingEngine.__str__c s|jdur td|_|j4IdH9|js@d|_t|jj|j|jjtj dk|jj d|_ |j IdHWdIdHdSWdIdHdS1IdHsQwYdS)zstartup engineNT )max_batch_sizemodel_replicasvector_disk_cache_pathverboselengths_via_tokenize) rrrr r batch_sizerr2r levelr4_batch_handlerspawnr,rrr!astartRs"  .zAsyncEmbeddingEngine.astartc s|jdurdS|j4IdH%|jr)d|_|jIdHWdIdHdSWdIdHdS1IdHs:wYdS)z stop engineNF)rrr7shutdownr,rrr!astopcs .zAsyncEmbeddingEngine.astopc|IdHdSN)r9r,rrr! __aenter__lzAsyncEmbeddingEngine.__aenter__cr<r=)r;)rargsrrr! __aexit__or?zAsyncEmbeddingEngine.__aexit__cC||jSr=)_assert_runningr7overload_statusr,rrr!rDr z$AsyncEmbeddingEngine.overload_statusboolcCrBr=)rCr7 is_overloadedr,rrr!rGvrEz"AsyncEmbeddingEngine.is_overloadedcC|jSr=)rr,rrr! is_runningzzAsyncEmbeddingEngine.is_runningset[ModelCapabilites]cCs |jdjS)Nr)r capabilitiesr,rrr!rL~s z!AsyncEmbeddingEngine.capabilitiescCrHr=)rr,rrr!r#rJz AsyncEmbeddingEngine.engine_args sentences list[str]matryoshka_dim int | None'tuple[list['EmbeddingReturnType'], int]c,||jj||dIdH\}}||fS)a embed multiple sentences Kwargs: sentences (list[str]): sentences to be embedded matryoshka_dim (int): Length of matryoshka embedding Raises: ValueError: raised if engine is not started yet ModelNotDeployedError: If loaded model does not expose `embed` capabilities Returns: list["EmbeddingReturnType"]: embeddings 2D list-array of shape( len(sentences),embed_dim ) int: token usage )rMrON)rCr7embed)rrMrO embeddingsusagerrr!rS zAsyncEmbeddingEngine.embedF raw_scorestop_nquerydocsrXrY Optional[int]$tuple[list['RerankReturnType'], int]cs0||jj||||dIdH\}}||fS)arerank multiple sentences Kwargs: query (str): query to be reranked docs (list[str]): docs to be reranked raw_scores (bool): return raw scores instead of sigmoid top_n (Optional[int]): number of top scores to return after reranking if top_n is None, <= 0 or out of range, all scores are returned Raises: ValueError: raised if engine is not started yet ModelNotDeployedError: If loaded model does not expose `rerank` capabilities Returns: list[float]: list of scores int: token usage rZr[rXrYN)rCr7rerank)rrZr[rXrYscoresrUrrr!r_szAsyncEmbeddingEngine.rerankrX$tuple[list[ClassifyReturnType], int]crR)aclassify multiple sentences Kwargs: sentences (list[str]): sentences to be classified raw_scores (bool): if True, return raw scores, else softmax Raises: ValueError: raised if engine is not started yet ModelNotDeployedError: If loaded model does not expose `embed` capabilities Returns: list[ClassifyReturnType]: list of class encodings int: token usage rMrXN)rCr7classify)rrMrXr`rUrrr!rds zAsyncEmbeddingEngine.classifyrOimages)list[Union[str, 'ImageClassType', bytes]]crR)aEembed multiple images Kwargs: images (list[Union[str, ImageClassType]]): list of image urls or ImageClassType objects, to be embedded matryoshka_dim (int): Length of matryoshka embedding Raises: ValueError: raised if engine is not started yet ModelNotDeployedError: If loaded model does not expose `image_embed` capabilities Returns: list["EmbeddingReturnType"]: embeddings 2D list-array of shape( len(sentences),embed_dim ) int: token usage rfrON)rCr7 image_embed)rrfrOrTrUrrr!ris z AsyncEmbeddingEngine.image_embedaudioslist[Union[str, bytes]]crR)a(embed multiple audios Kwargs: audios (list[Union[str, Audiobytes]]): list of audio data, to be embedded matryoshka_dim (int): Length of matryoshka embedding Raises: ValueError: raised if engine is not started yet ModelNotDeployedError: If loaded model does not expose `audio_embed` capabilities Returns: list["EmbeddingReturnType"]: embeddings 2D list-array of shape( len(sentences), embed_dim ) int: token usage rjrON)rCr7 audio_embed)rrjrOrTrUrrr!rmrVz AsyncEmbeddingEngine.audio_embedcCs|jstddS)Nzgdidn't start `AsyncEmbeddingEngine` recommended use is via AsyncContextManager `async with engine: ..`)r ValueErrorr,rrr!rCs z$AsyncEmbeddingEngine._assert_running)NT)rrrr)r#rrr$)rr*rrF)rrK)rrr=)rMrNrOrPrrQ) rZr*r[rNrXrFrYr\rr])rMrNrXrFrrb)rfrgrOrPrrQ)rjrkrOrPrrQ)__name__ __module__ __qualname____doc__r" classmethodr)r-r9r;r>rArDrGpropertyrIrLr#rSr_rdrirmrCrrrr!rs@          % rc@seZdZdZd;ddZedddZ d?dd Z d!dd"d@d(d)Z d!d*dAd,d-Z dddBd0d1ZdCd5d6ZdddDd9d:ZdS)EAsyncEngineArrayz$sz,AsyncEngineArray.__init__..z$Engines must have unique model namescSsi|]}|jj|qSrryr{rrr! 'sz-AsyncEngineArray.__init__..)rnlenlistset engines_dict)rrwrrr!r" s zAsyncEngineArray.__init__engine_args_arrayIterable[EngineArgs]r'AsyncEngineArray'cCsttj|}|t|dS)z|create an engine from EngineArgs Args: engine_args_array (list[EngineArgs]): EngineArgs object )rw)maprr)tuple)r'rrwrrr!r))s zAsyncEngineArray.from_args Iterator['AsyncEmbeddingEngine']cCst|jSr=)iterrvaluesr,rrr!__iter__4szAsyncEngineArray.__iter__c$|jD] }|IdHqdS)zstartup enginesN)rrr9rr(rrr!r97zAsyncEngineArray.astartcr)z stop enginesN)rrr;rrrr!r;<rzAsyncEngineArray.astopNremodelr*rMrNrOr\rQc||j||dIdHS)a9embed multiple sentences Kwargs: model (str): model name to be used sentences (list[str]): sentences to be embedded matryoshka_dim (int): Length of matryoshka embedding Raises: ValueError: raised if engine is not started yet ModelNotDeployedError: If loaded model does not expose `embed` capabilities Returns: list["EmbeddingReturnType"]: embeddings 2D list-array of shape( len(sentences),embed_dim ) int: token usage reN)rS)rrrMrOrrr!rSAzAsyncEngineArray.embedrFcCstdd|jDS)Ncss|]}|jVqdSr=)rIr{rrr!r}Xsz.AsyncEngineArray.is_running..)allrrr,rrr!rIWszAsyncEngineArray.is_runningFrWrZr[rXrYr]cs||j||||dIdHS)aererank multiple sentences Kwargs: model (str): model name to be used query (str): query to be reranked docs (list[str]): docs to be reranked raw_scores (bool): return raw scores instead of sigmoid top_n (Optional[int]): number of top scores to return after reranking Raises: ValueError: raised if engine is not started yet ModelNotDeployedError: If loaded model does not expose `rerank` capabilities Returns: list[float]: list of scores int: token usage r^N)r_)rrrZr[rXrYrrr!r_ZszAsyncEngineArray.rerankrarbcr)a classify multiple sentences Kwargs: model (str): model name to be used sentences (list[str]): sentences to be classified raw_scores (bool): if True, return raw scores, else softmax Raises: ValueError: raised if engine is not started yet ModelNotDeployedError: If loaded model does not expose `embed` capabilities Returns: list[ClassifyReturnType]: list of class encodings int: token usage rcN)rd)rrrMrXrrr!rdwszAsyncEngineArray.classifyrf"list[Union[str, 'ImageClassType']]cr)atembed multiple images Kwargs: model (str): model name to be used images (list[Union[str, ImageClassType]]): list of image urls or ImageClassType objects, to be embedded matryoshka_dim (int): Length of matryoshka embedding Raises: ValueError: raised if engine is not started yet ModelNotDeployedError: If loaded model does not expose `image_embed` capabilities Returns: list["EmbeddingReturnType"]: embeddings 2D list-array of shape( len(sentences),embed_dim ) int: token usage rhN)ri)rrrfrOrrr!riszAsyncEngineArray.image_embed index_or_nameUnion[str, int]r$cCsxt|jdkrt|jdSt|trt|j|St|tr-||jvr-|j|Std|dt|j)zresolve engine by model name -> Auto resolve if only one engine is present Args: model_name (str): model name to be used r.rzEngine for model name `z'` not found. Available model names are ) rrrr isinstanceintr* IndexErrorkeys)rrrrr! __getitem__s   zAsyncEngineArray.__getitem__rjrkcr)aQembed multiple audios Kwargs: model (str): model name to be used audios (list[Union[str, bytes]]): list of audio data, to be embedded matryoshka_dim (int): Length of matryoshka embedding Raises: ValueError: raised if engine is not started yet ModelNotDeployedError: If loaded model does not expose `audio_embed` capabilities Returns: list["EmbeddingReturnType"]: embeddings 2D list-array of shape( len(sentences),embed_dim ) int: token usage rlN)rm)rrrjrOrrr!rmrzAsyncEngineArray.audio_embed)rwrx)rrrr)rr)rr*rMrNrOr\rrQro) rr*rZr*r[rNrXrFrYr\rr])rr*rMrNrXrFrrb)rr*rfrrOr\rrQ)rrrr$)rr*rjrkrOr\rrQ)rprqrrrsr"rtr)rr9r;rSrIr_rdrirrmrrrr!rvs*      rvN) __future__rasynciortypingrrrrinfinity_emb.argsrinfinity_emb.inferencer r infinity_emb.log_handlerr infinity_emb.primitivesr r rrrrrvrrrr!s