o ri#7 @sdZddlmZmZmZmZmZddlZddlm Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZeeZGd ddZGdddZdde dee defddZ!    dde dee dee"dee"de"def ddZ#dS)u< LangChain检索服务 实现高级检索策略和RAG应用 )ListDictOptionalAnyTupleN)Session)Document)ChatPromptTemplate)StrOutputParser)RunnablePassthrough) ChatOpenAI)get_vector_store)CustomEmbeddings)get_embedding_factoryc@s"eZdZdZddedeefddZ   dd ed ed ed e deede e ee ff ddZ d ed ed e deede e ee ff ddZd ed ede e ee ffddZd ed ed e deede de de e ee ffddZ   dd ed ed ed e deede ee ff ddZdS)RetrievalServiceu'检索服务 - 实现多种检索策略Ndbknowledge_base_idcCs$||_||_t|||_t|_dS)u 初始化检索服务 Args: db: 数据库会话 knowledge_base_id: 知识库ID N)rrr vector_storerembedding_factory)selfrrrL/lsinfo/ai/hellotax_ai/base_platform/app/services/rag/langchain_retrieval.py__init__s  zRetrievalService.__init__semanticffffff?querymodek thresholdmodel_idreturnc Ksz:|dkr|||||WS|dkr|||WS|dkr4|dd}|dd}|||||||WStd|tyO} z td | d } ~ ww) uV 检索相关文档 Args: query: 查询文本 mode: 检索模式 ('semantic', 'keyword', 'hybrid') k: 返回结果数量 threshold: 相似度阈值 model_id: 向量模型ID **kwargs: 其他参数 Returns: 检索结果列表 rkeywordhybridkeyword_weightg333333?semantic_weightrzUnsupported retrieval mode: zRetrieval failed: N)_semantic_search_keyword_searchget_hybrid_search ValueError Exceptionloggererror) rrrrrr kwargsr$r%errrretrieve&s$  zRetrievalService.retrievec Cs|jj||j|d}|std|jj|||d}g}|D]"\}} ||d|d|d|d|d| |d |d d d q|S) u 语义搜索textrr "Failed to generate query embedding)query_embeddingrrid document_idtitler2 chunk_index category_id model_namer9r:r5r6r7r2r8scoremetadata)rgenerate_embeddingrr*rsimilarity_searchappend) rrrrr r4resultsformatted_resultsdocr=rrrr&Ps4    z!RetrievalService._semantic_searchc Csddlm}dg}||d}|jdur|d|j|d<|dd |d }|j||}g}|D]} || j| j| j | j | j t | j | j| jd d q3|S) u关键词搜索r)r2z(setweight(to_tsvector('simple', COALESCE(kd.title, '')), 'A') || setweight(to_tsvector('simple', COALESCE(dv.chunk_text, '')), 'B')) @@ plainto_tsquery('simple', :query))rrNz)dv.knowledge_base_id = :knowledge_base_idras SELECT dv.id, dv.document_id, dv.chunk_index, dv.chunk_text, dv.model_name, kd.title, kd.category_id, ts_rank( setweight(to_tsvector('simple', COALESCE(kd.title, '')), 'A') || setweight(to_tsvector('simple', COALESCE(dv.chunk_text, '')), 'B'), plainto_tsquery('simple', :query) ) as score FROM document_vectors dv JOIN knowledge_documents kd ON dv.document_id = kd.id WHERE z AND z> ORDER BY score DESC LIMIT :k r;r<) sqlalchemyr2rrAjoinrexecuter5r6r7 chunk_textr8floatr=r9r:) rrrr2 where_clausesparams keyword_queryresultrCrowrrrr'{s2        z RetrievalService._keyword_searchr$r%c Cs|jj||j|d}|std|jj||||||d}g} |D]"\} } | | d| d| d| d| d| | d | d d d q | S) u 混合搜索r1r3)rr4rrr$r%r5r6r7r2r8r9r:r;r<)rr?rr*r hybrid_searchrA) rrrrr r$r%r4rBrCrDr=rrrr)s:    zRetrievalService._hybrid_searchc KszXddl}|}|jd |||||d|} |} | |} dd| D} | r0t| t| nd} | r8t| nd}| r@t| nd}|||||d|| t| | ||| ddWStym}z td |d}~ww) ug 召回测试 - 用于评估检索效果 Args: query: 查询文本 mode: 检索模式 k: 返回结果数量 threshold: 相似度阈值 model_id: 向量模型ID **kwargs: 其他参数 Returns: 测试结果，包含检索结果和统计信息 rNrrrrr cSsg|]}|dqS)r=r).0rrrr sz0RetrievalService.recall_test..)rrr ) total_results avg_score max_score min_score elapsed_time)rrrKrB statisticszRecall test failed: r) timer0sumlenmaxminr+r,r-)rrrrrr r.rZ start_timerBend_timerXscoresrUrVrWr/rrr recall_testsN zRetrievalService.recall_testN)rrrN)__name__ __module__ __qualname____doc__rrintrstrrIrrrr0r&r'r)rbrrrrrs * "+7 3 rc@seZdZdZ    ddedeedeedeedef d d Z  ddededede deedeede ee ffddZ dS) RAGServiceuRAG服务 - 检索增强生成N gpt-3.5-turborr llm_api_key llm_base_url llm_modelcCs<||_||_t|||_|rt|||dd|_dSd|_dS)u 初始化RAG服务 Args: db: 数据库会话 knowledge_base_id: 知识库ID llm_api_key: LLM API密钥 llm_base_url: LLM API基础URL llm_model: LLM模型名称 r)api_keybase_urlmodel temperatureN)rrrretrieval_servicer llm)rrrrlrmrnrrrr+s  zRAGService.__init__rrrquestionrrrr system_promptr!c szW|jstd|jj|||||d}|sdggdWSdddt|Dd}td |p0|fd g} fd d td | B|jBt B} | |} | dd|D|dWSt yl} z t d| d} ~ ww)uB 基于知识库回答问题 Args: question: 用户问题 mode: 检索模式 k: 检索结果数量 threshold: 相似度阈值 model_id: 向量模型ID system_prompt: 系统提示词 Returns: 回答结果 z*LLM not configured, cannot generate answerrPzCSorry, I could not find relevant information in the knowledge base.)answersourcesretrieved_docsz c Ss2g|]\}}d|dd|dd|dqS)z Document z (Source: r7z): r2r)rQirDrrrrSxs z.RAGService.answer_question..aYou are a professional knowledge base assistant. Please answer the user's question based on the provided document content. Requirements: 1. Only use the provided document content to answer questions 2. If there is no relevant information in the documents, clearly inform the user 3. Answers should be accurate, concise, and professional 4. If possible, cite specific document sourcessystem)humanz9Reference documents: {context} User question: {question}csSrcr)xcontextrrsz,RAGService.answer_question..)rrucSs6g|]}|d|d|dddd|ddqS)r6r7r2Nz...r=)r6r7r2r=r)rQrDrrrrSszRAG Q&A failed: N)rtr*rsr0rF enumerater from_messagesr r invoker+r,r-) rrurrrr rvrydefault_system_promptprompt rag_chainrwr/rrranswer_questionLsV    zRAGService.answer_questionNNNrk)rrrNN) rdrerfrgrrrhrirrIrrrrrrrrj(sJ $ rjrrr!cCs t||S)u获取检索服务实例)r)rrrrrget_retrieval_services rrkrlrmrncCst|||||S)u获取RAG服务实例)rj)rrrlrmrnrrrget_rag_servicesrrcr)$rgtypingrrrrrloggingsqlalchemy.ormrlangchain_core.documentsrLangChainDocumentlangchain_core.promptsr langchain_core.output_parsersr langchain_core.runnablesr langchain_openair )app.services.storage.vector_store_factoryr app.services.storage.milvusr3app.services.llm.backends.embedding_backend_factoryr getLoggerrdr,rrjrhrrirrrrrsH