jk+ ddlZddlmZmZmZmZddlZddlmZm Z m Z ddl m Z ddl mZddlmZgdZdgfd eeefd Zd eeed eefd Zd efd ZdedefdZdefdZdedefdZdgfded eeedeeefdZ ddee d eeededee fdZdS)N)IterableIteratorListOptional)DetectorFactory detect_langslang_detect_exception)Element)logger)TESSERACT_LANGUAGES_SPLITTER)~aframharaasmazeaze_cyrlbelbenbodbosbrebulcatcebceschi_sim chi_sim_vertchi_tra chi_tra_vertchrcoscymdandeudivdzoellengenmepoequesteusfaofasfilfinfrafrkfrmfryglagleglggrcgujhathebhinhrvhunhyeikuindislitaita_oldjavjpnjpn_vertkankatkat_oldkazkhmkirkmrkorkor_vertlaolatlavlitltzmalmarmkdmltmonmrimsamyanepnldnorocioriosdpanpolporpusqueronrussansinslkslvsndsnumspaspa_oldsqisrpsrp_latnsunswaswesyrtamtatteltgkthatirtonturuigukrurduzbuzb_cyrlvieyidyorr( languagescD|tdttdd|D}tt|}t |dkrt jd|ddStj |S) zf Entry point: convert languages (list of strings) into tesseract ocr langcode format (uses +) Nz`languages` can not be `None`c|duo|dkS)N)xs e/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/unstructured/partition/lang.pyz1prepare_languages_for_tesseract..satm/Rc,g|]}t|Sr)convert_language_to_tesseract.0langs r z3prepare_languages_for_tesseract..s! G G GT *4 0 0 G G Grrz@Failed to find any valid standard language code from languages: z, proceed with `eng` instead.r() ValueErrorlistfilterdictfromkeyslenr warningr join)rconverted_languagess rprepare_languages_for_tesseractrs8999 / / G GY G G G  t}}-@AABB 1$$ C# C C C   u ' ,-@ A AAr ocr_languagesc|dg}t|tstd|9|dgkrtdt |}t jd|S)zHandle `ocr_languages` and `languages`, defining `languages` to ['eng'] as default and converting `ocr_languages` if neededNr(zOThe language parameter must be a list of language codes as strings, ex. ['eng']z}Only one of languages and ocr_languages should be specified. languages is preferred. ocr_languages is marked for deprecation.zmThe ocr_languages kwarg will be deprecated in a future version of unstructured. Please use languages instead.) isinstancer TypeErrorr&convert_old_ocr_languages_to_languagesr r)rrs rcheck_languagesrsG i & &  ]      S  ?}MMI N0    rc6|tS)z Convert ocr_languages parameter to list of langcode strings. Assumption: ocr_languages is in tesseract plus sign format )splitr )rs rrrs   ; < <z0convert_language_to_tesseract..s BBB48BBBrz* is not a language supported by Tesseract.)PYTESSERACT_LANGSiso639LanguagematchlowerLanguageNotFoundErrorr rpart3(_get_all_tesseract_langcodes_with_prefixr rpart2bpart2t)r lang_iso639pytesseract_langs_3matched_langcodess rrrsK     o++DJJLL99  '$GGGHHHrr CB0ABBB///D[EVWW+01BCCC  2 2 2D[EWXX+01BCCC  2 2 2D[EWXX+01BCCC $JJJKKKrs1?&A)(A)prefixc*fdtDS)zb Get all matching tesseract langcodes with this prefix (may be one or multiple variants). c>g|]}||Sr) startswith)rlangcoders rrz<_get_all_tesseract_langcodes_with_prefix..s, V V V(:M:Mf:U:U VH V V Vr)r)rs`rrrs! W V V V%6 V V VVrctj|dd}|jS)zP Convert a language code to the standard internal language code format. Nr)rrrrr)rrs r_convert_to_standard_langcoders4 /''RaR(8(899K  rautotextct|tstd|ddks|dkrdSt jd|r(t |dkrdgSdt_ |rd|vr d |D}nt |d krtj d |d  t|}n1#tj$r}tj |Yd}~dSd}~wwxYwd |D}g}|D]}||vr|||S)z Detects the list of languages present in the text (in the default "auto" mode), or formats and passes through the user inputted document languages if provided. zOThe language parameter must be a list of language codes as strings, ex. ["eng"]rrNz^[\x00-\x7F]+$r(rc,g|]}t|Sr)rrs rrz$detect_languages..%s!SSS6t<<SSSrz9Since "auto" is present in the input languages provided (z]), the language will be auto detected and the rest of the inputted languages will be ignored.cg|]?}|jdrtdnt|j@S)zh)rrr)rlangobjs rrz$detect_languages..;sZ   |&&t,, = )$ / / /.w|<<   r)rrrstriprerrrrseedr rrr LangDetectExceptionappend)rr doc_languageslangdetect_resultelangdetect_langsrs rdetect_languagesrs i & &  ]   |rTZZ\\R//t x!4((S->->-B-BwO "+V9,,SSSSS y>>A   N-I---      ,T 2 2  $8    N1   44444   -     $ + +D=(($$T*** sC##D2D  DFelementsdetect_language_per_elementc#K|dg}|dgkr |Ed{VdSt|tst|}dd|D}t ||}|.t |dkr|dur|D]}||j_|VdS|D]9}t|d r#t |j |j_|V5|V:dS) zBDetect and apply metadata.languages to each element in `elements`.Nrr c3DK|]}t|d|jVdS)rN)hasattrr)rrs r z&apply_lang_metadata..ds3HHAWQ5G5GHHHHHHHr)rrrFr) rrrrrrmetadatarrr)rrr full_textdetected_languagesrs rapply_lang_metadatarKs:H RD h % %">>HHHHHHHI)yINNN&  NNa   '5 0 0  A#5AJ GGGG    Aq&!! '7'?'? $   r)F)rtypingrrrrr langdetectrrr unstructured.documents.elementsr unstructured.loggerr &unstructured.partition.utils.constantsr rstrrrrrrrrboolrrrrrs@ 555555555555 KKKKKKKKKK333333&&&&&&OOOOOODGLWBBxS /BBBBB0xS 28C=6=#====&&&&&&RWSWWWW'-XDD DS "Dd3iDDDDT).**w*S "*"&*g ******r