o Q5i@4@sddlZddlZddlZddlZddlmZmZddlmZddl m Z ddl m Z m Z ddlmZmZmZmZmZmZddlmZdefd d ZGd d d Zd dZdefddZejreeddlZddlZdZ ej!dkrzddl"Z"e#e"$dZ Wn e%yYnwe&Z'e'(de j)de j*de j+de j,de j-de j.e j/de j0de j1de j2de j3de j4de5e j6de j7de j8de j9e j:e j;e je j?fdedede@dedeAdedddeAdeAd d!d"eAd#ed$d%d&d'd(eAd)eAd*eAd+eAd,ed-ed.ed/e@d0d1f.d2dZBe'(d3ejCdVied4d5d6iejCdViedd5d7iejCdViedd5d8iejCdViedd5d9iejCdViedd5d:iejCdViedd5d;iejCdViedd5diejCdVied?d5d@iejCdVied"d5dAiejCdVied#d5dBiejCdVied$d5dCiejCdVied&d5dDiejCdVied(d5dEiejCdVied)d5dFiejCdVied*d5dGiejCdVied.d5dHiejCdVied/d5dIiejCdVied-edJdKejCdViedd5dLiejCdVied0d5dMiejCdVied+d5dNiejCdVied,d5dOiejCdViedPd5dQifd4eDedeDedeDe@deDedeDeAdeDedeDeAdeDeAd eDed?eDed"eDeAd#eDed$eDed&eDed(eDeAd)eDeAd*eAd.ed/e@d-eded0d1d+eAd,edPef2dRd3ZEdSdTZFeGdUkrdTeHvreFdSdSdS)WN) CHECK_TYPER CHECK_UVICORN) EngineArgs)MANAGER)UVICORN_LOG_LEVELSlogger)DeviceDeviceIDDtypeEmbeddingDtypeInferenceEngine PoolingMethod) create_serverpathcCstd|r|Std)z This regex matches: - An empty string or A single '/' - A string that starts with '/' and does not end with '/' z^$|^/$|^/.*[^/]$z2Path must start with '/' and must not end with '/')rematchtyper BadParameter)rr^/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/infinity_emb/cli.py validate_urls  rc@s4eZdZdZdefddZdefddZdd Zd S) AutoPaddingz:itertools.cycle with custom behaviour to pad to max lengthlengthcKs||_||_dS)N)rkwargs)selfrrrrr__init__*s zAutoPadding.__init__ iterationcCs^t|}t|ttfs |St|dkr|dSt||jkr"||Std|jdt|)zpad x to length of self.lengthrzExpected length z but got )typer_option_resolve isinstancelisttuplelenr ValueError)rxrrrr_resolve.s zAutoPadding._resolveccsDt|jD]}i}|jD] \}}|||||<q|VqdS)z9iterate over kwargs and pad them to length of self.lengthN)rangerritemsr%)rrrkeyvaluerrr__iter__:szAutoPadding.__iter__N)__name__ __module__ __qualname____doc__intrr%r*rrrrr's  rcGsHt|dkrt|ddrt|ddr|djS|dSdd|DS)z&returns the value or the default valuerrdefaultenvvarcss.|]}t|drt|dr|jn|VqdS)r0r1N)hasattrr0).0arrr Ks,z'typer_option_resolve..)r"r2r0)argsrrrrCs    rnamecCs ttt|dt|ddS)z@constructs the default entry and type hint for the variable name`)r0r1)dictgetattrrto_name)r7rrr _constructNsr<auto) uvloopv1model_name_or_pathserved_model_name batch_sizerevisiontrust_remote_coderedirect_slashenginer model_warmupvector_disk_cachedevicerlengths_via_tokenizedtypeembedding_dtyper pooling_methodr compilebettertransformer preload_onlypermissive_corsapi_key url_prefixhostport log_levelrcCs|rtd| tjks| tkstdtdtdid|gd|gd|gd|gd|gd |gd | gd | gd | gd |gd|gd| gd|gd|gdtjgd|d|d|d|d|d|d|d|dddS)uTInfinity API ♾️ cli v1 - deprecated, consider use cli v2 via `infinity_emb v2`.z9api_key is not supported in `v1`. Please migrate to `v2`.zKselecting embedding_dtype is not supported in `v1`. Please migrate to `v2`.zOCLI v1 is deprecated. Consider use CLI `v2`, by specifying `v2` as the command.model_idrCrDrErFrHrMrOrKrIrJrLrPrQrNrRrUrVrWrGrXrSrTproxy_root_pathNr)r#r float32 default_valuerwarningv2)rBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrrrrAnsv       r_rYhelpztHuggingface model repo id. Subset of possible models: https://huggingface.co/models?other=text-embeddings-inference&zBthe nickname for the API, under which the model_id can be selectedz maximum batch size for inferencez!huggingface model repo revision.zCif potential remote modeling code from huggingface repo is trusted.zWhich backend to use. `torch` uses Pytorch GPU/CPU, optimum uses ONNX on GPU/CPU/NVIDIA-TensorRT, `CTranslate2` uses torch+ctranslate2 on CPU/GPU.z=if model should be warmed up after startup, and before ready.zLIf hash(request)/results should be cached to SQLite for latency improvement.z3device to use for computing the model forward pass. device_idzcdevice id defines the model placement. e.g. `0,1` will place the model on MPS/CUDA/GPU 0 and 1 eachz`if True, returned tokens is based on actual tokenizer count. If false, uses len(input) as proxy.zdtype for the model weights.zQdtype post-forward pass. If != `float32`, using Post-Forward Static quantization.z5overwrite the pooling method if inferred incorrectly.zEEnable usage of `torch.compile(dynamic=True)` if engine relies on it.ziEnables varlen flash-attention-2 via the `BetterTransformer` implementation. If available for this model.zwIf true, only downloads models and verifies setup, then exit. Recommended for pre-caching the download in a Dockerfile.z#host for the FastAPI uvicorn serverz#port for the FastAPI uvicorn serverzeprefix for all routes of the FastAPI uvicorn server. Useful if you run behind a proxy / cascaded API.)callbackr`z"where to redirect `/` requests to.zconsole log level.z!whether to allow permissive cors.z(api_key used for authentication headers.rZz\Proxy prefix for the application. See: https://fastapi.tiangolo.com/advanced/behind-a-proxy/c Cs t|ddt| D}tdidt|d|d|d|d|d|d |d |d |d |d | d| d| d| d|d|d|}g}|D] }|tdi|qOt||||||||| \ }}}}}}}}}t||t ||d|||||d}t j ||||j dt ddS)uAInfinity API ♾️ cli v2. MIT License. Copyright (c) 2023-now Michael Feil Multiple Model CLI Playbook: - 1. cli options can be overloaded i.e. `v2 --model-id model/id1 --model-id model/id2 --batch-size 8 --batch-size 4` - 2. or adapt the defaults by setting ENV Variables separated by `;`: INFINITY_MODEL_ID="model/id1;model/id2;" && INFINITY_BATCH_SIZE="8;4;" - 3. single items are broadcasted to `--model-id` length, making `v2 --model-id model/id1 --model-id/id2 --batch-size 8` both models have batch-size 8. cSsg|]}t|qSr)r )r3drrr :szv2..rrBrDrErFrHrIvector_disk_cache_pathrKrarLrMrNrOrPrQrC)rVrW)engine_args_listrU doc_extrarGrRrSrTrZ httptools)rVrWrXhttploopNr)rsetLevelto_intrrr"appendrrr9uvicornrunr7loopname)rYrCrDrErFrHrIrJrKrarLrMrNrOrPrQrRrVrWrUrGrXrSrTrZdevice_id_typedpadder engine_argsrapprrrr_sf          cCsltttjdkstjddvr1ttjdks tjddvr1tdtjdtj ddt dS)Nr)rAr_r`--helpz--show-completionz--install-completion)rAr_r`ruzpError: No command given. Please use infinity with the `v2` command. This is deprecated since 0.0.32. You are on z. Usage: `infinity_emb v2 --model-id BAAI/bge-large-en-v1.5. defaulting to `v2` since 0.0.75. Please pin your revision for future upgrades.r_) r mark_requiredr"sysargvrerror infinity_emb __version__inserttprrrrcli~s r~__main__r)Iasynciorrwrzinfinity_emb._optional_importsrrinfinity_emb.argsrinfinity_emb.envrinfinity_emb.log_handlerrrinfinity_emb.primitivesrr r r r r infinity_emb.infinity_serverrstrrrrr< is_availablervrrnrp version_infor@set_event_loop_policyEventLoopPolicy ImportErrorTyperr}commandrYrCrDrErFrGrHrIrJrKrLrMr]rOrPrQrRrSrTrUrVrWrXr/boolrAOptionr r_r~r+localsrrrrs*               E !%)-048<AEFGLORUX J