3 deOO@sddlZddlmZmZddlmZmZmZmZyddl m Z Wne k rXe Z YnXddl mZmZmZmZddlmZmZmZmZddlmZdd lmZmZdd lmZmZmZm Z m!Z!m"Z"ej#d Z$ej%Z&e&j'ej(d de)e*e*e+ee ee e,e,ed ddZ-dee*e*e+ee ee e,e,ed ddZ.d e e*e*e+ee ee e,e,ed ddZ/d!e e*e*e+ee ee e,edddZ0dS)"N)basenamesplitext)BinaryIOListOptionalSet)PathLike)coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDTOO_BIG_SEQUENCETOO_SMALL_SEQUENCETRACE) mess_ratio) CharsetMatchCharsetMatches)any_specified_encoding iana_nameidentify_sig_or_bom is_cp_similaris_multi_byte_encodingshould_strip_sig_or_bomZcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s皙?TF) sequencessteps chunk_size threshold cp_isolation cp_exclusionpreemptive_behaviourexplainreturnc1.Cs t|ttfs tdjt||r>tj}tjt tj t t |} | dkrtj d|rvtjt tj |prtjtt|dddgdgS|dk rtjt d d j|d d |D}ng}|dk rtjt d d j|dd |D}ng}| ||krtjt d||| d}| }|dkr:| ||kr:t| |}t |tk} t |tk} | rltjt dj| n| rtjt dj| g} |rt|nd} | dk r| j| tjt d| t}g}g}d}d}d}t}t|\}}|dk r| j|tjt dt ||| jdd| kr.| jdxv| tD]h}|rT||krTq:|rh||krhq:||krvq:|j|d}||k}|ot|}|d|dk o>t || k} | rTtjt d|tt |d }!t'|!d!}!d}"d}#g}$g}%x|D]}&|&|| d"krq||&|&|}'|r|dkr||'}'y|'j(||rd#nd$d%}(WnBt#k r(}z$tjt d&|t"||!}"d}#PWYdd}~XnX|r|&dkr||&d'krt)|d(})|r|(d|)|krxdt&|&|&d d=D]P}*||*|&|}'|r|dkr||'}'|'j(|d#d%}(|(d|)|kr|Pq|W|$j|(|%jt*|(||%d>|kr|"d7}"|"|!ks|r|dkrPqW|# r| r| ry|td)dj(|d$d%WnFt#k r}z(tjt d*|t"||j|w:WYdd}~XnX|%rt+|%t |%nd}+|+|ks|"|!krF|j|tjt d+||"t,|+d,d-d.|dd| gkr:|# r:t|||dg|},|| kr.|,}n|dkr>|,}n|,}q:tjt d/|t,|+d,d-d.|srt-|}-nt.|}-|-rtjt d0j|t"|-g}.|dkrx4|$D],}(t/|(d1|-rd2j|-nd}/|.j|/qWt0|.}0|0rtjt d3j|0||jt|||+||0||| ddgkrd|+d1krdtj d4||rVtjt tj |t||gS||kr:tj d5||rtjt tj |t||gSq:Wt |dk rX|s|s|rtjt d6|rtj d7|j1|j|nd| r|dk s(| r| r|j2|j2k s(|dk r>tj d8|j|n| rXtj d9|j|| r|tj d:|j3j1t |dn tj d;| rtjt tj ||S)?ae Given a raw bytes sequence, return the best possibles charset usable to render str objects. If there is no results, it is a strong indicator that the source is binary/not text. By default, the process will extract 5 blocs of 512o each to assess the mess and coherence of a given sequence. And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page but never take it for granted. Can improve the performance. You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that purpose. This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain' toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging. Custom logging format and handler can be set manually. z4Expected object of type bytes or bytearray, got: {0}rz]szfrom_bytes..zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.cSsg|]}t|dqS)F)r)r)r*r+r+r,r-hsz^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.r z>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.asciiutf_16utf_32z[Encoding %s wont be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.z2Encoding %s does not provide an IncrementalDecodergA)encodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.ignorestrict)errorszaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sgj@z^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d)ndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}g?,z We detected language {} using {}z.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.>r0r/r>)4 isinstance bytearraybytes TypeErrorformattypeloggerlevel addHandlerexplain_handlersetLevelrlendebug removeHandlerloggingWARNINGrrlogjoinintrrrappendsetrraddrrModuleNotFoundError ImportErrorstrUnicodeDecodeError LookupErrorrrangemaxdecodeminrsumroundr r r r r1 fingerprintbest)1rrr r!r"r#r$r%Zprevious_logger_levellengthZis_too_small_sequenceZis_too_large_sequenceZprioritized_encodingsZspecified_encodingZtestedZtested_but_hard_failureZtested_but_soft_failureZfallback_asciiZ fallback_u8Zfallback_specifiedresultsZ sig_encodingZ sig_payloadZ encoding_ianaZdecoded_payloadZbom_or_sig_availableZstrip_sig_or_bomZis_multi_byte_decodereZsimilar_soft_failure_testZencoding_soft_failedZr_Zmulti_byte_bonusZmax_chunk_gave_upZearly_stop_countZlazy_str_hard_failureZ md_chunksZ md_ratiosiZ cut_sequencechunkZchunk_partial_size_chkjZmean_mess_ratioZfallback_entryZtarget_languagesZ cd_ratiosZchunk_languagesZcd_ratios_mergedr+r+r, from_bytes%sZ                                                             rh) fprr r!r"r#r$r%r&c Cst|j|||||||S)z Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )rhread)rirr r!r"r#r$r%r+r+r,from_fpsrk) pathrr r!r"r#r$r%r&c Cs,t|d}t||||||||SQRXdS)z Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. rbN)openrk) rlrr r!r"r#r$r%rir+r+r, from_paths ro)rlrr r!r"r#r$r&c Cst|||||||}t|}tt|} t|dkrBtdj||j} | dd| j7<t djt |j |dj | d} | j | jWdQRX| S)zi Take a (text-based) file path and try to create another file next to it, this time using UTF-8. rz;Unable to normalize "{}", no encoding charset seems to fit.-z{}r(wbN)rorlistrrJIOErrorrCrar1rnrWreplacerPwriteoutput) rlrr r!r"r#r$rcfilenameZtarget_extensionsresultrir+r+r, normalize7s*    ry)rrrNNTF)rrrNNTF)rrrNNTF)rrrNNT)1rMos.pathrrtypingrrrrosrrVrWZcdr r r r ZconstantrrrrZmdrmodelsrrutilsrrrrrr getLoggerrE StreamHandlerrH setFormatter FormatterrArQfloatboolrhrkroryr+r+r+r,sb    Y