o ~riHn@sddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl m Z mZmZddlmZddlmZddlmZddl mZddlmZmZmZmZmZddlZddlmZdd lm Z dd l!m!Z!dd l"m!Z#ddl$Z$dd l$m%Z%m&Z&m'Z'm(Z(dd l)m*Z*m+Z+dd l,m-Z-ddl.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4erddl$m5Z5m6Z6m7Z7ddl8m9Z9e0rddl:Z:e4rddl;mZ>ddl?m@Z@mAZAddlBmCZCddlDmEZEmFZFddlGmHZHddlImJZJddlKmLZLmMZMmNZNddlOmPZPddlQmRZRmSZSmTZTmUZUddlQmPZVddlWmXZXddlYmZZZm[Z[m\Z\m]Z]m^Z^m_Z_m`Z`maZambZbmcZcmdZdmeZemfZfmgZgmhZhddlimjZjddlkmlZlmmZmmnZnGdd d ejd!d"ZoGd#d$d$eXd!d"ZpGd%d&d&eJd!d"ZqemeoZremepZsemeqZthd'Zuhd(Zvhd)Zwe-xeyZzd*d+d,d-iZ{e|e{}Z~d.Zd/d0Zd1d2Zd3d4ZGd5d6d6ejZd7ed8e'd9e'fd:d;ZGdd?d?ZGd@dAdAZdBdCZGdDdEdEZGdFdGdGZGdHdIdIZdJe_eydKkreZdSdS)LN)Callable GeneratorIterable)asynccontextmanager) lru_cache)BytesIO)Thread) TYPE_CHECKING AnnotatedOptional TypedDictUnion)scan_cache_dir) DecodeStream)tqdm) AutoTokenizerBitsAndBytesConfigGenerationConfigPreTrainedTokenizerBase)LogitsProcessorListTextIteratorStreamer)logging)is_fastapi_availableis_librosa_availableis_openai_availableis_pydantic_availableis_uvicorn_availableis_vision_available)PreTrainedModelPreTrainedTokenizerFastProcessorMixin)ContinuousBatchingManager)Image)FastAPI HTTPException)CORSMiddleware) JSONResponseStreamingResponse) Transcription)TranscriptionCreateParamsBase)ChatCompletionChatCompletionMessageChatCompletionMessageParam)Choice)ChatCompletionChunk ChoiceDeltaChoiceDeltaToolCallChoiceDeltaToolCallFunction)CompletionCreateParamsStreaming)ResponseResponseCompletedEventResponseContentPartAddedEventResponseContentPartDoneEventResponseCreatedEvent ResponseErrorResponseErrorEventResponseFailedEventResponseInProgressEventResponseOutputItemAddedEventResponseOutputItemDoneEventResponseOutputMessageResponseOutputTextResponseTextDeltaEventResponseTextDoneEvent)ResponseCreateParamsStreaming) BaseModel TypeAdapterValidationErrorc@eZdZUdZeed<dS))TransformersResponseCreateParamsStreamingz OpenAI's ResponseCreateParamsStreaming with an additional field for the generation config (as a json string). generation_configN__name__ __module__ __qualname____doc__str__annotations__rQrQd/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/transformers/cli/serve.pyrHs  rHF)totalc@rG)+TransformersCompletionCreateParamsStreamingz OpenAI's CompletionCreateParamsStreaming with additional fields for the generation config (as a json string) and passing the request_id rINrJrQrQrQrRrUzrSrUc@s.eZdZUdZeed<eed<dZeed<dS)%TransformersTranscriptionCreateParamsz OpenAI's TranscriptionCreateParamsBase with an additional field for the generation config (as a json string). filerIFstreamN) rKrLrMrNbytesrPrOrXboolrQrQrQrRrVs rV> textuserstorepromptinclude reasoning background truncation tool_choice service_tier top_logprobsmax_tool_callsprevious_response_id>nstopr\audior]logprobsmetadata functions modalities predictionrcrdre function_callstream_optionsresponse_formatpresence_penaltyreasoning_effortweb_search_optionsparallel_tool_callsmax_completion_tokens>r^r_languagerrchunking_strategytimestamp_granularitiesqwenz z )startendz x-request-idcCsddl}||dSNr)torch manual_seed)_seedrrQrQrRset_torch_seedsrcCs$ddl}|jr|jdSdSr~)rcuda is_available empty_cache)rrQrQrRreset_torch_caches rcCsddl}||Sr~)r ones_like) _input_tensorrrQrQrRtorch_ones_likes rc@seZdZdZdZdZdZdS)ModalityLLMVLMSTTTTSN)rKrLrMrrrrrQrQrQrRrs rreqmodel_generation_configreturncKsX|ddurtd it|d}nt|}|jd i|}|D]\}}|dur3t|||q%|ddurBt |d|_ |ddurPt |d|_ |ddur^t |d|_ |ddurj|d|_ |ddurv|d|_|ddurt |d|_t |dd krd |_|d durt |d |_|d durt|d |S)a Creates a generation config from the parameters of the request. If a generation config is passed in the request, it will be used as a baseline for parameterization. Otherwise, we will use the model's default generation config. Other parameters in the request will be applied on top of the baseline. Args: req (`dict`): The request which may optionally contain generation parameters. model_generation_config (`GenerationConfig`): The model's default generation config. kwargs (`dict`): Additional parameters to set in the generation config. Returns: The prepared `GenerationConfig` object. rINmax_output_tokens max_tokensfrequency_penalty logit_biasri temperaturegFtop_pseedrQ)getrjsonloadscopydeepcopyupdateitemssetattrintmax_new_tokensfloatrepetition_penalty sequence_bias stop_stringsr do_samplerr)rrkwargsrInon_standard_kwargskvrQrQrR!create_generation_config_from_reqs6     rc@sreZdZdZdeegdfdefddZdededBfd d Z ded ededBfd d Z defddZ ddZ dS)DownloadAggregatoraCAggregates byte-progress across multiple concurrent download tqdm bars into a single SSE stream. huggingface_hub opens one tqdm bar per file shard. This class tracks them all and emits a single aggregate ``{"stage": "download", "progress": {"current": ..., "total": ...}}`` event whenever any bar updates. enqueueNmodel_id_and_revisioncCs||_||_i|_d|_dSN)rmodelbarslast_emitted_current)selfrrrQrQrR__init__.s zDownloadAggregator.__init__bar_idrTcCsd|f|j|<|dSr~r_emit)rrrTrQrQrRregister4 zDownloadAggregator.registercurrentcCs||f|j|<|dSrr)rrrrTrQrQrRr8rzDownloadAggregator.updatecCsdSrrQ)rrrQrQrRclose<szDownloadAggregator.closecCsptdd|jD}||jkrdS||_dd|jD}|r&t|nd}|d|jd||dddS) Ncss|]\}}|VqdSrrQ).0c_rQrQrR @sz+DownloadAggregator._emit..cSsg|] \}}|dur|qSrrQ)rrtrQrQrR Dsz,DownloadAggregator._emit..loadingdownloadrrTstatusrstageprogress)sumrvaluesrrr)r agg_currenttotals agg_totalrQrQrRr?s zDownloadAggregator._emit) rKrLrMrNrdictrOrrrrrrrQrQrQrRr&s rc@sJeZdZdZddZddZdddZd d Zd d Zd dZ ddZ dS) DownloadProxyzT Leverages the DownloadAggregator in order to have a coherent tqdm wrapper. cCs(||_t||_||_d|_|j|_dSr~) wrapped_baridrdownload_aggregatorrhrT)rrrrQrQrRrUs   zDownloadProxy.__init__cC t|j|SrgetattrrrnamerQrQrR __getattr__] zDownloadProxy.__getattr__cCsF|durd}|j|7_|j|j|jt|jd|j|j|S)NrrT)rhrrrrrrTrrhrQrQrRr`s   zDownloadProxy.updatecCs|j|j|jSr)rrrrrrQrQrRris zDownloadProxy.closecC|j|Srr __enter__rrQrQrRrm zDownloadProxy.__enter__cG |jj|Srr__exit__rarQrQrRrqrzDownloadProxy.__exit__c csBd}|jD]}|d7}|j|j|t|jd|j|VqdS)NrrrT)rrrrrrT)rcountitemrQrQrR__iter__ts zDownloadProxy.__iter__Nr) rKrLrMrNrrrrrrrrQrQrQrRrPs   rc@sReZdZdZddZddZddZdd d Zd d Zd dZ ddZ ddZ dS) WeightsProxyzf Wraps the weight-loading tqdm bar to have finer control over how we emit them to the clinet. cCs*||_d|_||_||_d|_|j|_dS)Nr)r last_emittedcallablerrhrT)rr _callablerrQrQrRrs  zWeightsProxy.__init__cCrrrrrQrQrRrrzWeightsProxy.__getattr__c CsF|j|jkrdS|j|_|d|jd|jt|jd|jdddS)NrweightsrTrr)rhrrrrrrTrrQrQrRrs zWeightsProxy._emitrcCs.|durd}|j|7_||j|SNr)rhrrrrrQrQrRrs  zWeightsProxy.updatecCs |jSr)rrrrQrQrRrs zWeightsProxy.closecCrrrrrQrQrRrrzWeightsProxy.__enter__cGrrrrrQrQrRrrzWeightsProxy.__exit__ccs.|jD]}|jd7_||VqdSr)rrhr)rrrQrQrRrs  zWeightsProxy.__iter__Nr) rKrLrMrNrrrrrrrrrQrQrQrRr}s    rcs&tGfdddt}|S)NcsBeZdZdZfddZd fdd ZfddZZS) z$set_tqdm_class..ProgressTqdmuEtqdm subclass that routes progress to the correct SSE stage. Bars with ``unit="B"`` are download bars (one per file shard) — they are aggregated into a single ``download`` stage stream via ``_DownloadAggregator``. All other bars are weight-loading bars emitted as ``weights`` stage events. csb|dpd|_d|d<tj|i|d|_d|_|jdkr/t||_|j|j dSdS)NunititTdisablerrB) rsse_unitsuperrrhrr_bar_idrrT)rargsr __class__rrQrRrs  z-set_tqdm_class..ProgressTqdm.__init__rcst|durd}|j|7_|jdkr|j|j|jdS|j|jkr8|j|_dd|j|jdddSdS)Nrrrrrr)rhrrrrTrrcallbackrmidrQrRrs   z+set_tqdm_class..ProgressTqdm.updatecs$|jdkr |jtdS)Nr)rrrrrrrQrRrs  z*set_tqdm_class..ProgressTqdm.closer)rKrLrMrNrrr __classcell__rQr)rrR ProgressTqdms  r)r base_tqdm)rrrrQrrRset_tqdm_classs +rc@s eZdZdZddZddZdS) ToolStatez7Lightweight class to keep track of the tool call state.cCs |dSr)resetrrQrQrRrrzToolState.__init__cCsd|_d|_d|_d|_dS)z>Reset the tool call state (assumes we're outside a tool call).FrN)inside_tool_callhas_tool_name_definedarg_nesting_levelbufferrrQrQrRrs zToolState.resetN)rKrLrMrNrrrQrQrQrRrs rc@sReZdZdZ ddddededdBfdd Zd d Zd d ZddZ ddZ dS) TimedModelz A class that holds a PreTrainedModel instance and its associated processor. Automatically deletes the instances after a specified timeout. Nrrtimeout_seconds processor)r rcCs>||_t|j|_||_||_t|j|j|_ |j dSr) rrO name_or_path _name_or_pathr r  threadingTimertimeout_reached_timerr|)rrr r rQrQrRrs  zTimedModel.__init__cCs*|jt|j|j|_|jdS)z2Reset the timer for the deletion of the instances.N)rcancelrrr rr|rrQrQrR reset_timer s zTimedModel.reset_timercCsLt|dr"|jdur$|`|`d|_d|_tt|jdSdSdS)z>Delete the wrapped model and processor and clean up resources.rN)hasattrrr gccollectrrrrrQrQrR delete_modelszTimedModel.delete_modelcCs4|jdkr|t|jd|jddSdS)Nrz was removed from memory after z seconds of inactivity)r rloggerwarningrrrQrQrRr!s zTimedModel.timeout_reachedcCst|d p |jduS)z)Check if the instances have been deleted.rN)rrrrQrQrR is_deleted(szTimedModel.is_deletedr) rKrLrMrNrr rrrrrrQrQrQrRr s     r c$@seZdZ               dwdeedBejd d fd eeejd d fd eedBejdd fdeeejdd fdeedBejdd fdeedBejdd fdeeejdd fdeeejdd fdeeejdd fdeeejdd fdeedBejdd fdeeejd d fd!eeejd"d fd#eedBejd$d fd%eeejd&d'd(fd)df d*d+Z d,d-Z d.d/Z d0d1Z d2e d3ed4d5d6efd7d8Zd2e fd9d:Zd2e fd;d<Zd2e fd=d>Z ?       dxd@edAedBdBedBdCedBdDedBdEedFdBdGedBdHedId)dJfdKdLZedMdNd)efdOdPZeedydQedBd)ee eeffdRdSZdTe d@ed)dUfdVdWZedydBdXd)efdYdZZed[efd\d]Z dTe d)dUfd^d_Z!dTe d)e"eddffd`daZ#dTe d)e fdbdcZ$dTe d)e"eddffdddeZ%dTe d)efdfdgZ&d)e'dBfdhdiZ(djed)efdkdlZ) dydmedne*e gdfdBfdodpZ+ dydmedne*e gdfdBd)e,dqfdrdsZ-dmed)e,dtfdudvZ.dS)zServeNautoF localhost@,rcontinuous_batchingz8Whether to use continuous batching for chat completions.)helpdevicezgDevice to use for inference; will default to `auto` and place the model on an accelerator if available.dtypezOverride the default `torch.dtype` and load the model under this dtype. If `'auto'` is passed, the dtype will be automatically derived from the model's weights.trust_remote_codez2Whether to trust remote code when loading a model.attn_implementationz&Which attention implementation to use. quantizationzAWhich quantization method to use. choices: 'bnb-4bit', 'bnb-8bit'hostz$Interface the server will listen to.portzPort the server will listen to. model_timeoutz@Time in seconds after which a model will be removed from memory. log_levelz8Logging level as a string. Example: 'info' or 'warning'. default_seedz1The default seed for torch, should be an integer. enable_corsztWhether to enable CORS. Some apps that make requests from external domains (e.g. Cursor) require CORS to be enabled.input_validationz+Whether to turn on strict input validation. force_modelzName of the model to be forced on all requests. This is useful for testing Apps that don't allow changing models in the request. non_blockingTz/Whether to run the server in a separate thread.)hiddenr"rcststd|_|_|_|_|_|_|_|_ | _ | _ | _ | _ | _|_|_| dur;t| td}|tj| td}|tj| i_d_i_i_i_t_d_d_ d_!j durjrdnd_ jr"j}|_!#|t$dt%ffdd }t%|d }j r|j&t'd gd d gd gd t()d ddl*m+}|,dd|dt-ffdd }|,ddt-ffdd }|,dd|ffdd }|.d|/dfdd}|/ddd }|,d!dt-ffd"d# }|0d$d|fd%d&}t1j2|jj d'd(}t13|_4jr95dSj46dS))NzaMissing dependencies for the serving CLI. Please install with `pip install transformers[serving]` transformersz+transformers.generation.continuous_batchingrr appcsdVdSr)reset_loaded_models)r3rrQrRlifespans z Serve.__init__..lifespan)r5*T) allow_originsallow_credentials allow_methods allow_headerszUCORS allow origin is set to `*`. This is not recommended for production environments.r)Requestz/v1/chat/completionsrequestbodycsHj|dtd|ddjjr||jjS|S)Nr<[Request received] Model: r, CB: ) validate_chat_completion_requestrrr!#continuous_batching_chat_completionstate request_idgenerate_chat_completion)r<r=rrQrRchat_completions  z'Serve.__init__..chat_completionz /v1/responsescs`j|dtd|ddj|dd}|s%|}t|S|}t|ddS) Nr>r?rr@rXTtext/event-stream media_type) validate_response_requestrrr!rgenerate_response_non_streamingr'generate_responser()r<rX response_objoutputrrQrR responsess     z!Serve.__init__..responsesz/v1/audio/transcriptionsc s|4IdH5}t|dIdH|dd}td|djd|djd|djddd WdIdHn 1IdHsDwYj|d  |}t |d d S) NrWr)rWrzReceived file: z ; MIME type: z; size: z.2fz KiBr>rGrH) formrVreadrdebugfilename content_typesizevalidate_transcription_requestgenerate_transcriptionr()r<rQparsed_requestrNrrQrRaudio_transcriptionss(  z,Serve.__init__..audio_transcriptionsz /v1/modelscstddS)Nlist)objectdata)r'get_gen_modelsrQrrQrRget_all_modelssz&Serve.__init__..get_all_modelsz/healthcSs tddiS)Nrok)r'rQrQrQrR healthchecks z#Serve.__init__..healthcheckz /load_modelcsF|d}|durtddd|fdd}t|ddS) Nrz*Missing `model` field in the request body. status_codedetailcs,t}jv}|oj }|r/|r/jdtddddVdSjvrOj | | IdH}|durK dS|Vq=|gj<t dt ffdd t fd d fd d }t|j< | IdH}|durdS|Vq) Ndata: readyTrrcached payloadcs0dt|dfdd}|dS)Nrfrjcs"jgD]}|qdSr)loading_subscribersr put_nowaitq)rmsgrrQrR broadcasts z_Serve.__init__..load_model..event_publisher..enqueue..broadcast)rdumpscall_soon_threadsafe)rkrq)looprr)rprRrszLServe.__init__..load_model..event_publisher..enqueuecs||i|}|dpt|ddpd}|dpt|dd}t|d|d}|dkr:t||t|dS|dkrDt|S|S) NdescrrrrTr)rzLoading weights)rrrrrr)factoryrrbarrurrT)rrrrQrRstreaming_tqdm_hooks  zXServe.__init__..load_model..event_publisher..streaming_tqdm_hookc st}z_z tjIdHWn)ty=}ztjdd|dddt|dWYd}~nd}~wwWt|fdd} |dSWt|fdd} |dSt|fdd} |w) NzFailed to load z: T)exc_infoerror)rrmessagecs0jgD]}|dqjddSr)rlpoprm loading_tasksrnrrrQrR_send_sentinel;s zeServe.__init__..load_model..event_publisher..run_load.._send_sentinel) r set_tqdm_hookasyncio to_threadload_model_and_processor ExceptionrrzrOrs) previous_hooker)rrtrrrxrQrRrun_load1s(      zMServe.__init__..load_model..event_publisher..run_load)rQueue loaded_modelsrrrrrr}rlappendrget_running_looprr create_task)queue model_loadedmodel_not_deletedrrr~)rrrtrxrRevent_publishers<    z;Serve.__init__..load_model..event_publisherrGrH)rr%process_model_namer()r=rrr)rrR load_models   Pz"Serve.__init__..load_modelhttpcs>|jtp tt}||j_||IdH}||jt<|Sr)headersr X_REQUEST_IDrOuuiduuid4rCrD)r< call_nextrDresponserQrQrRget_or_set_request_idLs  z-Serve.__init__..get_or_set_request_idinfo)r(r)r+)7serve_dependencies_available ImportErrorr&r!r#r'r$r%r(r)r*r+r,r-r.r/r0rr get_loggersetLevel log_levelslowerr#running_continuous_batching_managerrlr} model_locksrLockmodel_locks_guard last_messages last_kv_cache last_modelrrrr$add_middlewarer&r warning_oncefastapir;postroptionsr middlewareuvicornConfigServerserver start_serverrun)rr!r#r$r%r&r'r(r)r*r+r,r-r.r/r0transformers_logger cb_loggerrr5r3r;rFrOrZr_rarrconfigrQrrRr0s9          Y  zServe.__init__cs,fdd}tj|ddd_jdS)Ncs,t_tjjjdSr)rnew_event_loop_loopset_event_looprun_until_completerserverQrrQrR_run]s  z Serve.start_server.._runzuvicorn-threadF)targetrdaemon)rr_threadr|)rrrQrrRr\s zServe.start_servercCsR|jstd|jstdd|j_|jr%|jr'|jjdddSdSdS)NzHThe server cannot be killed as it was not launched in a separate thread.zThe server is already killed.Tr!)timeout)r ValueErroris_aliver should_exitjoinrrQrQrR kill_serverfs zServe.kill_servercCsP|jdurtd|jjdddd|_t|jD]}|qd|_dS)z+ Resets all loaded models. Nz*Resetting the continuous batching manager.Tr!blockr) rrrrir[rrrr)rrrQrQrRr4qs    zServe.reset_loaded_modelsr<schema validatorrE unused_fieldsc Cstd|t|}|j}||}|r(td|tdd|d|jrhz||Wnt yP}ztd| td| dd}~ww||@} | rjtd| tdd| ddSdS)a Validates the request against the schema, and checks for unexpected keys. Args: request (`dict`): The request to validate. schema (`TypedDict`): The schema of the request to validate. It is a `TypedDict` definition. validator (`TypeAdapter`): The validator to use to validate the request. Built from `schema`. unused_fields (`set`): Fields accepted by `schema`, but not used in `transformers serve`. Raises: HTTPException: If the request is invalid or contains unexpected or unused fields. zValidating request: z Unexpected keys in the request: rbrczValidation error: NzUnused fields in the request: ) rrSsetkeys__mutable_keys__rzr%r.validate_pythonrFerrors) rr<rrr input_keys possible_keysunexpected_keysrunused_fields_in_requestrQrQrR_validate_request}s.   zServe._validate_requestcC|j|tttddSN)r<rrr)rrHresponse_validatorUNUSED_RESPONSE_FIELDSrr<rQrQrRrJ  zServe.validate_response_requestcCrr)rrUcompletion_validatorUNUSED_CHAT_COMPLETION_FIELDSrrQrQrRrArz&Serve.validate_chat_completion_requestcCrr)rrVtranscription_validatorUNUSED_TRANSCRIPTION_FIELDSrrQrQrRrWrz$Serve.validate_transcription_requestrrDcontentrrole finish_reason tool_callsr1 decode_stream tokenizerrr/c Cs\|dur|dur|dur||j|}t|tt|tt|||dd|dgddd} | S)a Builds a chunk of a streaming OpenAI Chat Completion response. IMPORTANT: The serialized chunk won't contain empty fields (fields with `None`). Some downstream apps, like Cursor, assume that when the field exists, it has data. Args: request_id (`str`): The request ID. content (`str`, *optional*): Content of the response from the model. model (`str`, *optional*): The model that generated the content. role (`str`, *optional*): The role of the next content, until a new role is defined. finish_reason (`str`, *optional*): The reason the generation by the model has finished. tool_calls (`list[ChoiceDeltaToolCall]`, *optional*): Data about the tool calls, when they are triggered. Returns: `str`: The built chunk, a string containing a JSON string with the payload. N)rrrr)deltaindexrrzchat.completion.chunk)rcreatedrchoicessystem_fingerprintr\)step _tokenizerr/rtime ChoiceChunkr0) rrDrrrrrrrchunkrQrQrRbuild_chat_completion_chunks("  z!Serve.build_chat_completion_chunkrzChatCompletionChunk | BaseModelcCs8t|tr|dr |Sd|dSd|jdddS)a/ Builds an event of a streaming OpenAI Response model or a ChatCompletion chunk. IMPORTANT: The serialized chunk won't contain empty fields (fields with `None`). Some downstream apps, like Cursor, assume that when the field exists, it has data. Args: chunk (`BaseModel` or `ChatCompletionChunk`): The response to build an event from. One of the multiple OpenAI Response output types Returns: `str`: The built chunk, a string containing a JSON string with the payload. rfrjT exclude_none) isinstancerO startswithmodel_dump_json)rrQrQrRchunk_to_sse_elements zServe.chunk_to_sse_element cache_dirc sddlm}m}g}tdtt|jD]u}|jdkrq|j }| D]e\}}|j }t dd|Dd} | s9q%t | } t| trKd| vsLq%| d} ||tfd d| Drd |jvro|jd nd } |j|d kr|d |nd } || | d|jdq%q|S)z2 List LLMs and VLMs in the cache. r!MODEL_FOR_CAUSAL_LM_MAPPING_NAMES*MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMESz/Scanning the cache directory for LLMs and VLMs.rcss |] }|jdkr|jVqdS)z config.jsonN) file_name file_path)rfrQrQrRr)sz'Serve.get_gen_models..N architecturesc3s$|] }|gvr|VqdSrrQ)rarchllmsvlmsrQrRr7s"/rmain@)owned_byrr\r)&transformers.models.auto.modeling_autorrrrrrrepos repo_typerefsrfilesnextrropenrRrrranyrepo_idsplitr last_modified)rrrgenerative_modelsrepor ref revision_infor  config_pathrrauthor repo_handlerQrrRr^s>  zServe.get_gen_modelsrz StreamingResponse | JSONResponsec s|djk}_|r!jdur!jjdddd_\}}j||dtjkr.stream_chat_completionc svd}jr|durjj|dd}jr|dus |j}t|ttdtdt |ddddgd }|S) Nr)rDrchat.completionrr)rrrirr{r)rrr\rr) r is_running get_resultdecoder2r+rrr.r,) _request_idr6rchat_completion_result)rrrrQrRbuffer_chat_completions$   zIServe.continuous_batching_chat_completion..buffer_chat_completioncsxz td}||D]}|VtdIdHqWdStjy;j|t d|dYdSw)NFrRequest  was cancelled.) rtolistrrsleepCancelledErrorrr4rr)rBr_chunk)inputsrr;rQrRcancellation_wrapper_streams  zNServe.continuous_batching_chat_completion..cancellation_wrapper_streamcs@z|WStjyj|td|dYdSw)NrErF)rrIrr4rr)rB)rDrrQrRcancellation_wrapper_buffers   zNServe.continuous_batching_chat_completion..cancellation_wrapper_bufferrX)rDr streamingrGrHrapplication/json)rrrrirget_model_modalityrrrrrErrrrIrrinit_continuous_batchingrlogit_processorr|*get_processor_inputs_from_inbound_messagesapply_chat_templatertor# add_requestrr(rr') rrrDmust_discard_cacherr modalityprocessor_inputsrLrMr json_chunkrQ)rDrIrKrrr;rrRrBEsh      0    z)Serve.continuous_batching_chat_completionrcCsj|dur t|tr tjSddlm}m}|jj}|| vr#tj }|S|| vr.tj}|St d|)NrrzUnknown modality: ) rrrrrrrrrKrrr)rr rrmodel_classnamerXrQrQrRrPs   zServe.get_model_modalityrXc Cs~g}|D]}|dgd}|tjkrEt|dtr|d}n"t|dtr@g}|dD]}|ddkr:||dq+d|}||d<nr|tjkrt|dtr^|dd|ddnY|dD]T}|ddkrr|d|qb|ddkrd |dd vrt d d |dd }t t t |}tjd dd} | j} || jn|dd } |dd| dqb||q|S)Nrrrrtyper[ )r]r[ image_urlbase64urlz^data:image/.+;base64,rz.pngF)suffixdeleteimage)r]ra)rrrrOr[rrrresubr#r rr` b64decodetempfileNamedTemporaryFilersave) rrXrYr{parsed_messageparsed_contentr image_datardrWrarQrQrRrSs@            z0Serve.get_processor_inputs_from_inbound_messagesc s\jdur j|d<|d}|dddkrdS|djk}_\}j|d}||}dtD]}|jjd vrO|nq?|j |d | d d d d d }| j }| d dd } djjd vrxd} t|| d d} |r|sj} |djd| krj} n d} n |djd} d} t|jdi|| d | dfdd} | drttj| | ddSg}d}| | }d}|D]%}|jd}t|jddr||jj|jr|j}t|ddr|j}qtt t!!dt"dt#d$|dd|d g|d!}|j%d d"}t&|d#dS)$a Generates an OpenAI Chat Completion using `generate`. Args: req (`dict`): The request to generate an OpenAI Chat Completion for. Returns: `Generator[str, None, None]`: A generator that yields the OpenAI Chat Completion chunks. Nrrrrr)rrTr r!r"rDreq_0gptossFskip_special_tokens skip_promptr'r)streamerrIreturn_dict_in_generatepast_key_valuesc 3sd}d}djjdvrd}d}fdd}t|d}d }zSz |t}jd d Vd }d} |D]}| d 7} djjdvrQ|d }||7}|r_||vr^d}qfinal<|message|>cjdi|}|j_dSNrQgeneratervrrgenerate_outputrrrQrRgenerate_with_cache z[Serve.generate_chat_completion..stream_chat_completion..generate_with_cacherrrr)r*r <|return|>r|r}r)rDrrrz\"name\": \"(.*?)\")rfunction _tool_call)rrr]rz"arguments": {{}) arguments)rrr])rDrrr)rrr+r,rir-r.r/)rrrrr|rr removesuffixstrip_TOOL_CALL_TOKENSrrr rresearchgroupr1r2rrrrrrrr+rrrzrO)rtrB filter_cot cot_trace_endrthreadresults tool_stater6r5 tool_nametoolr8r9r:rrIgeneration_kwargsrrrDrtool_model_familyrQrRr;s             z>Serve.generate_chat_completion..stream_chat_completionrXrGrHrirusager<rr=r>)rrr\rrrrrO)'r/rrrrPrS_MODELS_WITH_TOOL_SUPPORTrrrrTrrUr#ris_continuationrget_seq_lengthshaperrIr(maprrrrrrrrr+rrr.r,r model_dumpr')rrrrWr rXrYsupported_model_familiesrKrqgeneration_streamerseq_lenrr;rr generatorrrchoicerCr6rQrrRrE2s             zServe.generate_chat_completionc sdjk}_\}tdtr6dvr)dddgng}|dddnUtdtrjdvreddddkrXdddgd}n3d}d|dd <n&d}n!tdtrdvr}dddgng}|dntd |j |d d d d d}| j } ddd }dj jdvrd}t||d d}tjd}d}r|sֈj} |jd| krֈj}|t|||d |dfdd} | |S)a  Generates an OpenAI Response using `generate`. Args: req (`dict`): The request to generate an OpenAI Response for. Returns: `Generator[str, None, None]`: A generator that yields the OpenAI Response events. rinput instructionssystemr\r\rrr%inputs should be a list, dict, or strTr!r#r$r%r'rgrnroFrprsNr)rKattention_maskrtrIrurvc3s\d}d}djjdvrd}d}fdd}t|d}d}d}d}zz|t} td |td | d d d ddiidggddddd d} |d7} | Vt d|td | dd d ddiidggddddd d} |d7} | Vt d||t ddddgdd} |d7} | Vt dd|||td d!gd"d#} |d7} | Vd!}|D]U}djjdvr|d$}||7}|r||vrd}d!}qtd%d||||gd&}|d7} |Vq|r#td%d||||gd&}|d7} |Vqtd'd||d|gd(}|d7} |Vtd)d|||td |jgd"d#}|d7}|d7} |Vtd*||t ddd+d|jggd,d}|d7}|d7} |Vtd-|td | d+d d ddii|jgdgddddd. d}|d7} |V|Wncty}zVtd/t|td0|t|d1}|d7} |Vtd2|td | d3d d ddiigdgdddtd4t|d5d6 d}|d7} |VWYd}~n d}~wwW|dSW|dS|w)7NFrorTrwcrxryrzr|r~rQrRrrzMServe.generate_response..stream_response..generate_with_cacherzresponse.createdresp_queuedrformatr]r[rrvrrl) r created_atrrrr[r\r rNrvrcrl)r]sequence_numberrrzresponse.in_progress in_progresszresponse.output_item.addedmsg_r{r))rr]rrr)r]r output_indexrzresponse.content_part.added output_textrr]r[ annotations)r]item_idrr content_indexpartrzresponse.output_text.delta)r]rrrrrrkzresponse.output_text.done)r]rrrrr[rkzresponse.content_part.donezresponse.output_item.done completedrr]rrrrzresponse.completed rrrrrr[rNr\r rvrcrlz"Exception in response generation: rz)r]rr{zresponse.failedfailed server_error)coder{) rrrrrr[rNr\r rvrcrlrz) rrrrr|rr8r4rrr<r=r?r6r@rrArBr7r[r>rr5rrrrrzrOr:r;r9)rtrBrrrrrrrrresponse_createdresponse_in_progressresponse_output_item_addedresponse_content_part_addedrr6response_output_text_deltaresponse_output_text_doneresponse_content_part_doneresponse_output_item_doneresponse_completedr error_eventresponse_failedrrrrrDrrQrRstream_responses                        %z0Serve.generate_response..stream_response)rrrrrOrr[r TypeErrorrTrUr#rrrrrrrIrrrrr) rrrWr rKrqrrIrrrrQrrRrLBsb       ozServe.generate_responsecCsP||d}||jk}||_||\}}t|dtr6d|vr)d|ddgng}|d|ddnUt|dtrjd|vre|ddddkrXd|ddg|d}n3|d}|d|dd <n&|d}n!t|dtrd|vr}d|ddgng}||dntd |j |d d d d d}| |j }| dd}d }d|j jdvrd}t||jd} d} ||r|s|j} |jd| kr|j} |j|t|| d | d} | j|_|j| j|dd} t}td|dddtd| gdggd}td||d|| dd d!d"ii|gd#g| d$dd%| d&d' }|jd d(S))a Generates an OpenAI Response in non-streaming mode (single JSON payload). Args: req (`dict`): The request to generate an OpenAI Response for. Returns: `dict`: The OpenAI `Response` serialized as a dict. rrrrr\r\rrrrTr!rr'rgrnroFrsNr)rKrrIrurvrqrr{rr)rrrrrr]r[rrvrrlrr) rrrrrOrr[rrrTrUr#rrrrrrIrrrrr{rrv batch_decode sequencesrr?r@r4r)rrrrWrr rKrDrqrIrrr} full_textrresponse_output_itemrrQrQrRrKus        z%Serve.generate_response_non_streamingc ststd||d}||\tjddd}t|jd}jj }t |d}t j ||dd\}}||dd jd jd <||dd fd d } | S)a Generates an OpenAI Transcription using the audio file. Args: req (`dict`): The request containing the audio file and model information. Returns: `Generator[str, None, None]`: A generator that yields the transcription result. z]Missing librosa dependency for audio transcription. Please install with `pip install librosa`rTrprsrW)srmonor!) sampling_rater$input_features)rtrIruc3sHjdi}j|jddd}t|d}|jddVdS)NTrr)r[rrQ)r{rrr)r) generated_idstranscription_text transcription audio_inputs audio_modelaudio_processorrrQrR_generate_transcriptions  z=Serve.generate_transcription.._generate_transcription)rrrload_audio_model_and_processorrrrrIfeature_extractorriorlibrosaloadrUr#r$) rrrrrImodel_sampling_rate audio_bytes audio_arrayrrrQrrRrXs2 zServe.generate_transcriptioncCsx|dp |d}d}|jdurd}n#t|jt|kr d}ntt|jD]}|j|||kr6d}nq'||_|S)aD Determines whether the current request is a continuation of the last request. In other words, if it is the same chat session. Args: req (`dict`): The request to check. Returns: `True` if the request is a continuation of the last request, `False` otherwise. rrTNF)rrlenrange)rrrreq_continues_last_messagesirQrQrRrs  zServe.is_continuationcCsP|jdkr tdddd}n |jdkrtdd}nd}|dur&td||S) z Returns the quantization config for the given CLI arguments. Returns: `Optional[BitsAndBytesConfig]`: The quantization config. zbnb-4bitTnf4) load_in_4bitbnb_4bit_quant_typebnb_4bit_use_double_quantzbnb-8bit) load_in_8bitNz0Quantization applied with the following config: )r'rrr)rquantization_configrQrQrRget_quantization_configs   zServe.get_quantization_configmodel_idcCs&|jdur|j}d|vr|S|dS)aR Applies the `force_model` CLI argument and canonicalizes the model name to the format "model_id@revision". If the model_id DOESN'T contain an @, it defaults to "model_id@main". Args: model_id (`str`): The model ID. Returns: `str`: The canonicalized model name to be used Nrz@main)r/)rrrQrQrRr4s  zServe.process_model_namerprogress_callbackc sddl}ddlm}m}}}|durt||nd}dtffdd } td| dd vr>d d \} } nd } } z |j | | |j d } Wn t ynz t j | | |j d } Wn t ykt d wYnw|j dvrw|j nt||j } |}| |j| |j|j |d}| d|j | fi|}tt|jd}|j | fd|i|}|jjduo|jjdk}|jjduo|jjdk}|s|rd|j_|| fS)a Generic method to load a model and a data processor from a model ID and revision, making use of the serve CLI arguments. Args: model_id_and_revision (`str`): The model ID and revision to load. model_cls (`type[PreTrainedModel]`): The model class to load. Returns: `tuple[PreTrainedModel, Union[ProcessorMixin, PreTrainedTokenizerFast]]`: The loaded model and data processor (tokenizer, audio processor, etc.). rN) AutoConfig AutoProcessorrcs durdSd|ddS)Nr)rrrrQ)rrrrQrR emit_progress_sz;Serve._load_model_and_data_processor..emit_progresszLoading r rrr)revisionr%zBFailed to load processor with `AutoProcessor` and `AutoTokenizer`.)rN)rr&r$ device_mapr%rr tqdm_classrP)rr2rrrrOrrrfrom_pretrainedr%OSErrorrr$rrr&r#rrIr max_length)rrrrrrrrrrrrdata_processorr$r model_kwargsr architecturerhas_default_max_lengthhas_short_max_new_tokensrQrrR_load_model_and_data_processorEsb     z$Serve._load_model_and_data_processor)rrcCs|j|j|t}Wdn1swY|X||jvs+|j|rL|j||d\}}t||j |d|j|<|durK|d|ddn|j| |j|j }|j|j }|durk|d|dd||fWdS1sywYdS)a\ Loads the text model and processor from the given model ID and revision into the ServeCommand instance. Args: model_id_and_revision (`str`): The model ID and revision to load. Returns: `tuple[PreTrainedModel, PreTrainedTokenizerFast]`: The loaded text model and processor. N)rr r rgFrhT) rr setdefaultrrrrrr r*rrr )rrrlockrr rQrQrRrs2       $zServe.load_model_and_processor)rr cCs|j|j|t}Wdn1swY|F||jvs+|j|rFtd|| |\}}t ||j |d|j|<n|j| |j|j }|j|j}||fWdS1sgwYdS)aU Loads the audio model and processor from the given model ID and revision into the ServeCommand instance. Args: model_id_and_revision (`str`): The model ID and revision to load. Returns: `tuple[PreTrainedModel, ProcessorMixin]`: The loaded audio model and processor. NzLoading model into cache: r)rrrrrrrrrrr r*rrr )rrrrrrQrQrRrs&     $z$Serve.load_audio_model_and_processor)NrrFNNrrr rNFFNF)rNNNNNNNr)/rKrLrMr rZtyperOptionrOrrrrr4rr rrrJrArWr[rr r staticmethodrrrr^rBrrPrSrErrLrKrXrrrrrrtuplerrrQrQrQrRr-s`    "%(./58 .    1    :(/+-5Z0 V (ra Run a FastAPI server to serve models on-demand with an OpenAI compatible API. Models will be loaded and unloaded automatically based on usage and a timeout.  The server will expose the following endpoints: - POST /v1/chat/completions: Generates chat completions. - POST /v1/responses: Generates responses. - POST /v1/audio/transcriptions: Generates transcriptions from audio. - GET /v1/models: Lists available models for 3rd party tools. Requires FastAPI and Uvicorn to be installed. __main__)rr`renumrrrrerhrrrcollections.abcrrr contextlibr functoolsrrrtypingr r r r r rhuggingface_hubrtokenizers.decodersrr tqdm.autorr2rrrrtransformers.generationrrtransformers.utilsrtransformers.utils.import_utilsrrrrrrrrr r0r"rPILr#rrrr$r%fastapi.middleware.corsr&fastapi.responsesr'r( openai.types.audio.transcriptionr).openai.types.audio.transcription_create_paramsr*openai.types.chatr+r,r-!openai.types.chat.chat_completionr.'openai.types.chat.chat_completion_chunkr/r0r1r2r*openai.types.chat.completion_create_paramsr3openai.types.responsesr4r5r6r7r8r9r:r;r<r=r>r?r@rArB-openai.types.responses.response_create_paramsrCpydanticrDrErFrHrUrVrrrrrrrrKrrr[rrrrrrEnumrrrrrrrrr rrNrrQrQrQrRs                    D     ;*-=14E