3 deG@s<ddlmZddlmZmZddlmZmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZGdddZGdd d eZGd d d eZGd d d eZGdddeZGdddeZGdddeZGdddeZGdddeZ ee!ee!e"dddZ#eddd#e!e$e"e$dd d!Z%d"S)$) lru_cache)ListOptional)COMMON_SAFE_ASCII_CHARACTERSUNICODE_SECONDARY_RANGE_KEYWORD)is_accentuatedis_asciiis_case_variableis_cjk is_emoticon is_hangul is_hiragana is_katakanais_latinis_punctuation is_separator is_symbolis_thai remove_accent unicode_rangec@sPeZdZdZeedddZeddddZddd d Ze e dd d Z dS) MessDetectorPluginzy Base abstract class used for mess detection plugins. All detectors MUST extend and implement given methods. ) characterreturncCstdS)z@ Determine if given character should be fed in. N)NotImplementedError)selfrrL/var/www/agendate/envp3/lib/python3.6/site-packages/charset_normalizer/md.pyeligibleszMessDetectorPlugin.eligibleNcCstdS)z The main routine to be executed upon character. Insert the logic in witch the text would be considered chaotic. N)r)rrrrrfeed$szMessDetectorPlugin.feed)rcCstdS)zB Permit to reset the plugin to the initial state. N)r)rrrrreset+szMessDetectorPlugin.resetcCstdS)z Compute the chaos ratio based on what your feed() has seen. Must NOT be lower than 0.; No restriction gt 0. N)r)rrrrratio1szMessDetectorPlugin.ratio) __name__ __module__ __qualname____doc__strboolrrr propertyfloatr!rrrrrs rc@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS) TooManySymbolOrPunctuationPluginN)rcCs"d|_d|_d|_d|_d|_dS)NrF)_punctuation_count _symbol_count_character_count_last_printable_charZ_frenzy_symbol_in_word)rrrr__init__;s z)TooManySymbolOrPunctuationPlugin.__init__)rrcCs|jS)N) isprintable)rrrrrrCsz)TooManySymbolOrPunctuationPlugin.eligiblecCsp|jd7_||jkrf|tkrft|r8|jd7_n.|jdkrft|rft|dkrf|jd7_||_dS)NrF) r-r.rrr+isdigitrr r,)rrrrrrFs   z%TooManySymbolOrPunctuationPlugin.feedcCsd|_d|_d|_dS)Nr)r+r-r,)rrrrr Xsz&TooManySymbolOrPunctuationPlugin.resetcCs0|jdkrdS|j|j|j}|dkr,|SdS)Nrgg333333?)r-r+r,)rZratio_of_punctuationrrrr!]s   z&TooManySymbolOrPunctuationPlugin.ratio) r"r#r$r/r&r'rrr r(r)r!rrrrr*:s r*c@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS)TooManyAccentuatedPluginN)rcCsd|_d|_dS)Nr)r-_accentuated_count)rrrrr/jsz!TooManyAccentuatedPlugin.__init__)rrcCs|jS)N)isalpha)rrrrrrnsz!TooManyAccentuatedPlugin.eligiblecCs(|jd7_t|r$|jd7_dS)Nr)r-rr4)rrrrrrqszTooManyAccentuatedPlugin.feedcCsd|_d|_dS)Nr)r-r4)rrrrr wszTooManyAccentuatedPlugin.resetcCs*|jdkrdS|j|j}|dkr&|SdS)Nrggffffff?)r-r4)rZratio_of_accentuationrrrr!{s  zTooManyAccentuatedPlugin.ratio) r"r#r$r/r&r'rrr r(r)r!rrrrr3is r3c@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS)UnprintablePluginN)rcCsd|_d|_dS)Nr)_unprintable_countr-)rrrrr/szUnprintablePlugin.__init__)rrcCsdS)NTr)rrrrrrszUnprintablePlugin.eligiblecCs@|jdkr.|jdkr.|dkr.|jd7_|jd7_dS)NFr)isspacer0r7r-)rrrrrrs   zUnprintablePlugin.feedcCs d|_dS)Nr)r7)rrrrr szUnprintablePlugin.resetcCs|jdkrdS|jd|jS)Nrg)r-r7)rrrrr!s zUnprintablePlugin.ratio) r"r#r$r/r&r'rrr r(r)r!rrrrr6s  r6c@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS)SuspiciousDuplicateAccentPluginN)rcCsd|_d|_d|_dS)Nr)_successive_countr-_last_latin_character)rrrrr/sz(SuspiciousDuplicateAccentPlugin.__init__)rrcCs|jot|S)N)r5r)rrrrrrsz(SuspiciousDuplicateAccentPlugin.eligiblecCst|jd7_|jdk rjt|rjt|jrj|jrJ|jjrJ|jd7_t|t|jkrj|jd7_||_dS)Nr)r-r=risupperr<r)rrrrrrs  z$SuspiciousDuplicateAccentPlugin.feedcCsd|_d|_d|_dS)Nr)r<r-r=)rrrrr sz%SuspiciousDuplicateAccentPlugin.resetcCs|jdkrdS|jd|jS)Nrgr1)r-r<)rrrrr!s z%SuspiciousDuplicateAccentPlugin.ratio) r"r#r$r/r&r'rrr r(r)r!rrrrr;s r;c@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS)SuspiciousRangeN)rcCsd|_d|_d|_dS)Nr)"_suspicious_successive_range_countr-_last_printable_seen)rrrrr/szSuspiciousRange.__init__)rrcCs|jS)N)r0)rrrrrrszSuspiciousRange.eligiblecCsx|jd7_|js&t|s&|tkr0d|_dS|jdkrD||_dSt|j}t|}t||rn|jd7_||_dS)Nr)r-r9rrrAr is_suspiciously_successive_ranger@)rrunicode_range_aunicode_range_brrrrs  zSuspiciousRange.feedcCsd|_d|_d|_dS)Nr)r-r@rA)rrrrr szSuspiciousRange.resetcCs.|jdkrdS|jd|j}|dkr*dS|S)Nrgr1g?)r-r@)rZratio_of_suspicious_range_usagerrrr!s zSuspiciousRange.ratio) r"r#r$r/r&r'rrr r(r)r!rrrrr?s r?c@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS)SuperWeirdWordPluginN)rcCs:d|_d|_d|_d|_d|_d|_d|_d|_d|_dS)NrF) _word_count_bad_word_count_foreign_long_count_is_current_word_bad_foreign_long_watchr-_bad_character_count_buffer_buffer_accent_count)rrrrr/szSuperWeirdWordPlugin.__init__)rrcCsdS)NTr)rrrrrr szSuperWeirdWordPlugin.eligiblecCs|jrdj|j|g|_t|r0|jd7_|jdkrt|dksNt|rt|dkrt|dkrt |dkrt |dkrt |dkrd|_dS|jsdS|j st |st|o|jr|jd7_t|j}|j|7_|dkr8|j|dkrd|_t|jdr8|jdjr8|jd7_d|_|dkr^|jr^|jd7_d|_|jr|jd7_|jt|j7_d|_d|_d|_d|_n6|dkr|jdkrt|rd|_|j|7_dS)NrFrFTg(\?r<>-=~|_rX>rQrTrSrWrUrRrV)r5joinrMrrNrKrr r rrrr9rrrGlenr-rJr>rIrHrLr2r)rrZ buffer_lengthrrrrsR           zSuperWeirdWordPlugin.feedcCs4d|_d|_d|_d|_d|_d|_d|_d|_dS)NrFFr)rMrJrKrHrGr-rLrI)rrrrr DszSuperWeirdWordPlugin.resetcCs$|jdkr|jdkrdS|j|jS)N rg)rGrIrLr-)rrrrr!NszSuperWeirdWordPlugin.ratio) r"r#r$r/r&r'rrr r(r)r!rrrrrEs 6 rEc@s^eZdZdZddddZeedddZeddd d Zddd d Z e e dd dZ dS)CjkInvalidStopPluginu GB(Chinese) based encoding often render the stop incorrectly when the content does not fit and can be easily detected. Searching for the overuse of '丅' and '丄'. N)rcCsd|_d|_dS)Nr)_wrong_stop_count_cjk_character_count)rrrrr/\szCjkInvalidStopPlugin.__init__)rrcCsdS)NTr)rrrrrr`szCjkInvalidStopPlugin.eligiblecCs4|dkr|jd7_dSt|r0|jd7_dS)N丅丄r>r_r`)r]r r^)rrrrrrcs zCjkInvalidStopPlugin.feedcCsd|_d|_dS)Nr)r]r^)rrrrr jszCjkInvalidStopPlugin.resetcCs|jdkrdS|j|jS)Ng)r^r])rrrrr!ns zCjkInvalidStopPlugin.ratio) r"r#r$r%r/r&r'rrr r(r)r!rrrrr\Vsr\c@sZeZdZddddZeedddZedddd Zddd d Ze e dd d Z dS)ArchaicUpperLowerPluginN)rcCs.d|_d|_d|_d|_d|_d|_d|_dS)NFrT)_buf_character_count_since_last_sep_successive_upper_lower_count#_successive_upper_lower_count_finalr-_last_alpha_seen_current_ascii_only)rrrrr/vsz ArchaicUpperLowerPlugin.__init__)rrcCsdS)NTr)rrrrrrsz ArchaicUpperLowerPlugin.eligiblecCs$|jot|}|dk}|r|jdkr|jdkrV|jdkrV|jdkrV|j|j7_d|_d|_d|_d|_|j d7_ d|_dS|jdkrt |dkrd|_|jdk r|j r|jj s|j r|jj r|jdkr|jd7_d|_qd|_nd|_|j d7_ |jd7_||_dS)NFr@rTr1) r5r rdr2rhrfrergrcr-r r>islower)rrZ is_concernedZ chunk_seprrrrs8      zArchaicUpperLowerPlugin.feedcCs.d|_d|_d|_d|_d|_d|_d|_dS)NrFT)r-rdrerfrgrcrh)rrrrr szArchaicUpperLowerPlugin.resetcCs|jdkrdS|j|jS)Nrg)r-rf)rrrrr!s zArchaicUpperLowerPlugin.ratio) r"r#r$r/r&r'rrr r(r)r!rrrrrbus  * rb)rCrDrcCs~|dks|dkrdS||kr dSd|kr4d|kr4dSd|ksDd|krHdSd|ksXd|krld|kshd|krldS|jd|jd}}x"|D]}|tkrq||krdSqW|dk|dk}}|s|rd |ksd |krdS|r|rdSd |ksd |kr"d |ksd |kr dS|d ks|d kr"dSd |ksJd |ksJ|dkrz|dkrzd |ks^d |krbdSd|ksvd|krzdSdS)za Determine if two Unicode range seen next to each other can be considered as suspicious. NTFZLatinZ EmoticonsZ Combining HiraganaKatakanaCJKZHangulz Basic LatinZ PunctuationZForms)rlrm)rlrm)rmrl)rmrl)splitr)rCrDZkeywords_range_aZkeywords_range_belZrange_a_jp_charsZrange_b_jp_charsrrrrBsR    rBi)maxsize皙?F)decoded_sequencemaximum_thresholddebugrc CsddtjD}t|d}d}|dkr0d}n|dkr>d}nd }x|t|d t|D]f\}}x |D]} | j|rd| j|qdW|d kr||d ks||dkrVtd d |D}||krVPqVW|rx|D]} t| j | j qWt |dS)zw Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier. cSsg|] }|qSrr).0Zmd_classrrr szmess_ratio..rgi iri rcss|] }|jVqdS)N)r!)rvdtrrr &szmess_ratio..) r__subclasses__rZziprangerrsumprint __class__r!round) rsrtruZ detectorslengthZmean_mess_ratioZ!intermediary_mean_mess_ratio_calcrindexdetectorr{rrr mess_ratios*      rN)rrF)& functoolsrtypingrrZconstantrrutilsrr r r r r rrrrrrrrrrr*r3r6r;r?rEr\rbr&r'rBr)rrrrrs" D"/%6ZM D