jv6VUddlZddlZddlmZddlmZddlZddlmZddl m Z m Z m Z ddl mZddlmZmZddlmZdd lmZeeZdaejdzed <d ed ejfd ZGddZGddeZGddeZGddZ da!d e fdZ"dS)N)Iterator)Any)Session)ExternalServiceErrorModelNotFoundErrorProviderNotConfiguredError) LocalModel)Model ModelProvider) get_logger)trace_llm_call_shared_http_clienttimeoutreturnct tjr*tj|tjddatS)N )max_connectionsmax_keepalive_connections)rlimits)r is_closedhttpx AsyncClientLimits)rs V/lsinfo/ai/hellotax_ai/base_platform/app/services/llm/backends/chat_backend_factory.py_get_shared_clientrsF"&9&C"#/EL_a$b$b$b    ceZdZ ddeeeefdedzded ed ed ed eee ffd Z deee fd efdZ deee fd eeeffdZ dS)BaseChatClientNffffff?FTmessagesmodel temperature max_tokensstreamenable_table_formatrcKtNNotImplementedError)selfr"r#r$r%r&r's rchat_completionzBaseChatClient.chat_completion!s"!r api_responsectr)r*r,r.s rextract_response_textz$BaseChatClient.extract_response_text,s!!rcddddS)Nr input_tokens output_tokens total_tokensr0s rget_usage_infozBaseChatClient.get_usage_info/s !AqIIIrNr r!FT) __name__ __module__ __qualname__listdictstrfloatintboolrr-r1r8r7rrrrs ! $( " "tCH~& "Tz " "  "  "" " c3h " " " ""$sCx."S""""J4S>Jd38nJJJJJJrrceZdZ ddedededefdZ dd ee e e fde dzdedede de de e e ffdZ dd ee e e fdedefdZde e e fde fdZde e e fde e efffd ZxZS)OpenAICompatibleChatClientr@providerr# max_retriesrc|j|_|jp|j|_|j|_||_|j p|j |_ ||_ ||_ dSr))name_provider_namebase_urldefault_base_url _base_url auth_type _auth_type get_api_key_api_keyremote_model_idcode _model_coderHr)r,rGr#rHrs r__init__z#OpenAICompatibleChatClient.__init__5sb'm!*Gh.G", ,,..  0>EJ& rNr r!FTr"r$r%r&r'rc K|j}|st|jddi}|j} |jdkr | r d| |d<n%|jdkr| r| |d<n|jdkr| r| |d <|d d } |p|j||||d d id} |p|j} d} t|jD]} t|j }t|jd| |5}| | || d{V}| |}||di}||digddidd|dd|ddd|cdddcS#1swxYwY3#t"j$ri}|} t&d|dzd |jd|||jdz kr#t+jd|zd{VYd}~Yd}~d}~wt"j$r}|} |jj}d |cxkrd!kr_nn\||jdz krNt&d"|d#|dzd |jt+jd|zd{VYd}~2t5|jd$|d|jj|d}~wt"j$ri}|} t&d%|dzd |jd|||jdz kr#t+jd|zd{VYd}~Yd}~d}~wt:$r(}t5|jt=|dd}~wwxYwt?| t"jrt5|jd&|j d't5|jd(|jd)| )*N Content-Typeapplication/jsonbearerBearer Authorizationapi_keyapi-key x_api_key X-API-Key//chat/completionsenable_thinkingFr#r"r$r%r&chat_template_kwargsz/chat)rJr#r"headersjsonusagechoicesrmessagecontent prompt_tokenscompletion_tokens)inputoutput)rqrizChat API timeout on attempt z: iiXzChat API server error z on attempt zHTTP z"Chat API request error on attempt zRequest timeout after sz Failed after z retries: ) rNrrKrRrPrstriprUrangerHrrr postraise_for_statusrhgetupdaterTimeoutExceptionloggerwarningasynciosleepHTTPStatusErrorresponse status_codertext RequestError Exceptionr? isinstance)r,r"r#r$r%r&r'rLrgr]urlpayload model_name last_errorattemptclient generationrresultriers rr-z*OpenAICompatibleChatClient.chat_completion@s> B,T-@AA A!#56- ?h & &7 &':':':GO $ $ _ ) )g )!(GI   _ + + +#*GK %%888.d. &$%6$>   .d.  T-..0 R0 RG/ R+DL99# /666$%" %+[[gG[%T%TTTTTTTH--///%]]__F!- & 7B 7 7"))#)::i"#>#>q#A#E#EiQS#T#T#X#XYbdf#g#g).?A)F)F*/))4G*K*K##*"#"""""""""""""""""""$)    cgkccDL\cc`accdddT-111!-7 333333333HHHH21111(    j4 ++++++++++$:JQ:N0N0NNNjjj'TU+jjX\Xhjj"-7 333333333HHHH*')Q)Q)Q )Q)Q%    ^1^^tGW^^[\^^T-111!-7 333333333HHHH21111 R R R*4+>AGGTQ R j%"8 9 9 &#%Mdl%M%M%M #  !Y1A!Y!YZ!Y!Y   sj3.G"!C%G G"G G"G G""N21AIN2'A4L!&LN2AM== N2 #N--N2cK|j}|st|jddi}|j}|jdkr |r d||d<n%|jdkr|r||d<n|jdkr|r||d <|d d }|j|||d d did}t|j} | d|||4d{V} | | 23d{V} | ds| tdd} | dkrdddd{VdS tj| } | dddi}|dd}|r|WV#tjt&t(f$rYwxYw6 dddd{VdS#1d{VswxYwYdS)NrXrYrZr[r\r]r^r_r`rarbTrcFrdPOSTrfzdata:z[DONE]rjrdeltarlrm)rNrrKrRrPrurUrrr&rx aiter_lines startswithlenstriprhloadsryJSONDecodeErrorKeyError IndexError)r,r"r$r%rLrgr]rrrrlinedata chunk_datarrs rasync_stream_completionz2OpenAICompatibleChatClient.async_stream_completions!> B,T-@AA A!#56- ?h & &7 &':':':GO $ $ _ ) )g )!(GI   _ + + +#*GK %%888% &$%6$>   $DL11==gG=LL       PX  % % ' ' '&2244       dw//CLLNN+11338##              !%D!1!1J&y1!488"EEE 99Y33D#" ,h CH54                              sD)G*G0AG AFGF>;G=F>>G G #G r.cv |dg}|rj|ddi}|dd}t|trdd|DS|pdSdS#t$r(}t d|Yd}~d Sd}~wwxYw) Nrjrrkrlrmc3lK|]/}t|t|ddV0dS)rrmN)rr>ry).0parts r zCOpenAICompatibleChatClient.extract_response_text..sT##15JtUYDZDZ#,,######rz$Sorry, I cannot generate a response.z!Failed to extract response text: z7Sorry, an error occurred while processing the response.)ryrr=joinrr|error)r,r.rjrkrlrs rr1z0OpenAICompatibleChatClient.extract_response_texts M"&&y"55G %!!*..B77!++i44gt,,77##9@###}"$99 M M M LL@Q@@ A A ALLLLLL MsA=BB B8B33B8c |di}|d|dd}|d|dd}|d||z}|||dS#t$rH}td |t |cYd}~Sd}~wwxYw) Nrirnr4rror5r6r3zFailed to get usage info: )ryrr|rsuperr8)r,r.rirnror6r __class__s rr8z)OpenAICompatibleChatClient.get_usage_infos 8 $$Wb11E!IIouyyQR7S7STTM % *=uyyZ[?\?\ ] ]  99^]EV5VWWL -!2 ,   8 8 8 LL9a99 : : :77)),77 7 7 7 7 7 7 8sBB C=CCC)rErFr9r r!)r:r;r<r r rAr@rVr=r>r?rBrr-rr1r8 __classcell__)rs@rrDrD3s]b  % .3 BE TY    ! $(V V tCH~&V TzV  V  V  V "V  c3hV V V V r[_''T#s(^,';@'TW''''RM$sCx.MSMMMM 84S> 8d38n 8 8 8 8 8 8 8 8 8 8rrDceZdZdefdZ ddeeeefd edzd ed e d e d e deee ffdZ ddeeeefd ed e de efdZ ddeeeefd ed e fdZdeee fdefdZdS)LocalAssetChatClient local_modelc||_dSr))r)r,rs rrVzLocalAssetChatClient.__init__s&rNr r!FTr"r#r$r%r&r'rc Kddlm}g}d} |D]S} | d} | dd} | dkr| r|| K| dkr| } T||jj|jj| ||d| pd } d d | d igid S)Nrlocal_chat_servicermrolerlsystemuser  identifier model_pathpromptr%r$ system_promptrk assistant)rrl)rjri) #app.services.llm.local_chat_servicerryappendgenerate_responserrrrr)r,r"r#r$r%r&r'r system_parts user_promptrkrrl response_texts rr-z$LocalAssetChatClient.chat_completions  KJJJJJ"$   & &G;;v&&Dkk)R00GxG##G,,,,% *<<'2'2!# ++l3399;;Ct =  #[]$S$STU   rc nddlm}g}d}|D]S}|d}|dd} |dkr| r|| K|dkr| }T||jj|jj|||d| pd S Nrrrmrrlrrrr) rrryrgenerate_response_streamrrrrr) r,r"r$r%rrrrkrrls rstream_completionz&LocalAssetChatClient.stream_completions KJJJJJ"$   & &G;;v&&Dkk)R00GxG##G,,,,% !::'2'2!# ++l3399;;Ct ;   rc Kddlm}g}d}|D]S}|d}|dd} |dkr| r|| K|dkr| }T||jj|jj|||d| pd } | D]!} | WVtj dd{V"dSr) rrryrrrrrrrr~r) r,r"r$r%rrrrkrrlr&chunks rrz,LocalAssetChatClient.async_stream_completions0 KJJJJJ#%   & &G;;v&&Dkk)R00GxG##G,,,,% #<<'2'2!# ++l3399;;Ct =   # #EKKKK-"" " " " " " " " " # #rr.cZ |ddddS#t$rYdSwxYw)Nrjrrkrlrm)rr0s rr1z*LocalAssetChatClient.extract_response_text9sC  *1-i8C C   22 s  **r9r)r:r;r<r rVr=r>r?r@rArBrr-rrrr1r7rrrrsw'J'''' ! $(  tCH~& Tz       "  c3h    D[_  T#s(^, ;@ TW #    0[_##T#s(^,#;@#TW####6$sCx.Srrc<eZdZdededefdZdedededzfdZ dS)ChatBackendFactorymodel_iddbrcd|ttj|k}|st ||j}|stdd||jdvrt||St|j d|j)NchatzProvider not found for model >openaiopenai_compatible)rGr#zUnsupported chat protocol: ) queryr filteridfirstrrGrprotocolrDrJ)r,rrr#rGs r get_clientzChatBackendFactory.get_clientBs&&ux8';<<BBDD /$X.. .> [&v/Yx/Y/YZZ Z   ? ? ?-xuMMM M"8=2cPXPa2c2cdddrrNc|ttj|ktjdktj }|sdSt|S)Nactive)rr rrstatus is_deletedrr)r,rrrs rget_local_asset_clientz)ChatBackendFactory.get_local_asset_clientMsn HHZ V%3!X-)) UWW  4#K000r) r:r;r<rArrrr?rrr7rrrr@su e3 eG e e e e e11#*1  $111111rrc:ttatSr)) _chat_factoryrr7rrget_chat_factoryras*,, r)#r~rhcollections.abcrtypingrrsqlalchemy.ormrapp.core.exceptionsrrrapp.models.local_modelr app.models.providerr r common_loggingr common_langfuser r:r|rr__annotations__r@rrrDrrrrr7rrrs $$$$$$ """"""dddddddddd------44444444%%%%%%****** H  15U&-444%*;JJJJJJJJ(k8k8k8k8k8k8k8k8\\\\\\>\\\~11111111< ,r