jj UddlZddlZddlmZddlmZddlmZm Z m Z m Z ddl m Z ddlmZdaeje ed<eGdd eZdS) N) dataclass)create_ingest_doc_from_dict)BaseIngestDocBatchBaseSessionHandleBaseSingleIngestDocIngestDocSessionHandleMixin)logger) SourceNodesession_handleceZdZdededefdZdededej efdZ dedej ej eej effdZ dS) Readerdocingest_doc_dictreturnc|jjsi|jrP|jjr2t jd|jd|n | d}t j d|dtj na#t$rT}t jd|t j d|jjdtj Yd}~nd}~wwxYw|jr||jn||D] \}}|||< |jS)Nz File exists: z, skipping downloadT)redact_sensitivez Fetching z - PID: zfailed to print full doc: ) read_config re_downloadfilenameis_filestatst_sizer infoupdate_source_metadatato_jsondebugosgetpid Exceptionwarning __class____name__retry_strategyget_fileto_dictitems)selfrrserialized_docekvs m/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/unstructured/ingest/pipeline/source.py get_singlezReader.get_singles ,  $$&&  !!##+  KI III J J J  & & ( ( ( ( X!$d!C!C NNNNNOOOO X X X;Q??? V)?VVVVWWWWWWWW X" ##CL1111 KKMM'')) # #DAq!"OA  |s7AB99 DA DD doc_batchc|jr||jn||D] \}}|||< d|jDS)Ncg|] }|j S)r).0rs r, z$Reader.get_batch..6s>>> >>>)r# get_filesr%r& ingest_docs)r'r.rr*r+s r, get_batchzReader.get_batch/s   "    3 4 4 4 4    ! ! !%%''--// # #DAq!"OA  >> (=>>>>r4c t|}t|trt|jan t|_t|t r|||St|tr|||Stdt|d#t$r4}|j j rtjd|d|dYd}~dSd}~wwxYw) N)rr)r.rz type of doc (zF) is not a recognized type: BaseSingleIngestDoc or BaseSingleIngestDocz/failed to get data associated with source doc: z, T)exc_info)r isinstancerr _session_handlerr-rr7 ValueErrortyperpipeline_contextraise_on_errorr error)r'rrr)s r,runz Reader.run8s6 -o>>C#:;; 9!)&)%7NN*8C'#233 3PPPC!344 ~~_~UUU BDIIBBB   $3  LX/XXUVXX    44444 s$A*B9-+B9 B99 C7)C22C7N)r" __module__ __qualname__rdictstrr-rtListr7OptionalUnionrAr1r4r,r r s1DS4?#5??QRQWX[Q\????4AJqwsAF3K?O7P,Qr4r )rtypingrF dataclassesr&unstructured.ingest.connector.registryrunstructured.ingest.interfacesrrrrunstructured.ingest.loggerr 'unstructured.ingest.pipeline.interfacesr r rH__annotations__r r1r4r,rQs !!!!!!NNNNNN .----->>>>>>15 ,-444 ?????Z?? ???r4