3j5,UdZddlZddlmZmZmZdZdZdZdZ dZ d Z d Z d ezZ eed <d ededefdZd ededzfdZd ededzfdZd ededzfdZdedefdZddededefdZy)aStage 1a+: UTF-16/UTF-32 detection for data without BOM. This stage runs after BOM detection but before binary detection. UTF-16 and UTF-32 encoded text contains characteristic null-byte patterns that would otherwise cause binary detection to reject the data. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)ASCII_TEXT_BYTESDETERMINISTIC_CONFIDENCEDetectionResulti gQ??gffffff?g333333?_NULL_SEPARATOR_ALLOWEDdata null_fracreturncD|tk\ry|jdt S)uReturn True if the data looks like ASCII with null byte separators. :param data: The raw byte sample to examine. :param null_frac: The positional null fraction for this UTF-16 candidate (i.e. fraction of null bytes in even positions for BE, or odd positions for LE) — not the total null fraction across all bytes. Checks two conditions: 1. The positional null fraction is below ``_NULL_SEPARATOR_MAX_FRACTION`` 2. Every non-null byte is printable ASCII or common whitespace When both conditions are met, the nulls are likely field separators (e.g. ``find -print0``), not UTF-16 encoding artifacts. FN)_NULL_SEPARATOR_MAX_FRACTION translater )r r s E/DATA/.local/lib/python3.12/site-packages/chardet/pipeline/utf1632.py_is_null_separator_patternr6s%00~~d$;< <<cn|dt}t|tkryt|}||St |S)aDetect UTF-32 or UTF-16 encoding from null-byte patterns. UTF-32 is checked before UTF-16 since UTF-32 patterns are more specific. :param data: The raw byte data to examine. :returns: A :class:`DetectionResult` if a strong pattern is found, or ``None``. N) _SAMPLE_SIZElen_MIN_BYTES_UTF16 _check_utf32 _check_utf16)r sampleresults rdetect_utf1632_patternsrJsD-< F 6{%%& !F    rc ttdzz }|tkryd||dz}tfdtdtdD}tfdtdtdD}||k(r8||z dkDr0 j d}t |rt dtdS tfd td tdD}tfd td tdD}||k(r9||z dkDr1 j d }t |rt d tdS yy#t$rYwxYw#t$rYywxYw)aCheck for UTF-32 encoding based on 4-byte unit structure. For valid Unicode (U+0000 to U+10FFFF = 0x0010FFFF): - UTF-32-BE: the first byte of each 4-byte unit is always 0x00 - UTF-32-LE: the last byte of each 4-byte unit is always 0x00 For BMP characters (U+0000 to U+FFFF), additionally: - UTF-32-BE: the second byte is also 0x00 - UTF-32-LE: the third byte is also 0x00 Nc34K|]}|dk(s dywrN.0ir s r z_check_utf32..tJ#9aT!W\#9 rc3:K|]}|dzdk(sdyw)r!rNr"r#s rr&z_check_utf32..vs#O$:qd1q5kQ>N$:srz utf-32-beencoding confidencelanguagec34K|]}|dk(s dywr r"r#s rr&z_check_utf32..sI"8QDGqLq"8r(c34K|]}|dk(s dywr r"r#s rr&z_check_utf32..r'r(z utf-32-le) r_MIN_BYTES_UTF32sumrangedecode_looks_like_textrrUnicodeDecodeError)r trimmed_len num_units be_first_nullbe_second_nulltext le_last_null le_third_nulls` rrr`sxd)s4y1}-K%%   Dq IJ5CIq#9JJMOE!SY$:OON !ny&@3&F ;;{+D%&(7!&I%3t9a"8IILJ5CIq#9JJMy ]Y%>%D ;;{+D%&(7!& )"   ""    s$-E -E EE E'&E'c&ttt}||dzz}|tkry|dz}t fdt d|dD}t fdt d|dD}||z }||z }g}|t k\r"td||s|jd|f|t k\r"td||s|jd|f|syt|dk(r<|dd} d|j|} t| rt|td S yd} d } |D]/\}} d|j|} t| } | | kDs,| } |} 1| | tk\rt| td Sy#t$rYywxYw#t$rYjwxYw) aCheck for UTF-16 via null-byte patterns in alternating positions. UTF-16 encodes each BMP character as two bytes. For characters whose code-point high byte is 0x00 (Latin, digits, basic punctuation, many control structures), one of the two bytes in each unit will be a null. Even for non-Latin scripts (Arabic, CJK, Cyrillic, etc.) a significant fraction of code units still contain at least one null byte. Non-UTF-16 single-byte encodings never contain null bytes, so even a small null-byte fraction in alternating positions is a strong signal. When both endiannesses show null-byte patterns (e.g., Latin text where every other byte is null), we disambiguate by decoding both ways and comparing text-quality scores. r1Nc34K|]}|dk(s dywr r"r#s rr&z_check_utf16..K#:ad1gl#:r(rc34K|]}|dk(s dywr r"r#s rr&z_check_utf16..rAr(r!z utf-16-lez utf-16-ber*)minrrrr3r4_UTF16_MIN_NULL_FRACTIONrappendr5r6rrr7 _text_quality_MIN_TEXT_QUALITY)r sample_lenr9 be_null_count le_null_countbe_fracle_frac candidatesr+r< best_encoding best_quality_qualitys` rrrs SY -J*q. J$$aIK5J#:KKMK5J#:KKMi'Gi'G*,J**3M [j74 ;01**3M [j74 ;01  :!a=#  $++H5D%&%7!&!%ML! !  $++H5D % \ !"L$M" \5F%F"/  5"   "   s$,0E5,F5 FF FFr<c`|sy|dd}td|D}|t|z tkDS)z9Quick check: is decoded text mostly printable characters.FNc3JK|]}|js|dvsdyw) r!N) isprintable)r$cs rr&z#_looks_like_text..sJv!AMAvs##)r3r_MIN_PRINTABLE_FRACTION)r<r printables rr6r6s8  $3ZFJvJJI s6{ "%< <10% control characters or >20% combining marks (category ``M*``). NrrCLr!MZsrVCg?g?r)r unicodedatacategoryord) r<r[rnlettersmarksspacescontrols ascii_lettersrXcatscores rrGrGs(&5\F F AAvG E FHM ""1% q6S= qLG1v|" Vs] QJE D[AM aKF Vs] MH!|c qy3 aKE ma3 &&E2v&1*   Lr)rT)__doc__rcchardet.pipelinerrrrr2rrErHrYrr bytes__annotations__floatboolrrrrstrr6intrGr"rrrvs XX   $")+;!;;=U=u==( % Od,B ,5u54!75pQuQ4!7Qh=3=4=99C9%9r