yj+ ddlZddlZddlmZddlmZmZddlmZddl m Z ddl m Z mZddlmZddlmZdd lmZdd lmZmZmZdd lmZejeZGd d eZGddZGddZ GddZ!GddZ"d&de#de$de%e#fdZ& d'dede#de#de#dzde%ef dZ'ej(dddfd ed!e#de$d"e)e#efdzd#ej*dzd$ejdzde%e)e#effd%Z+dS)(N)BytesIO)AnyProtocol) UploadFile)generate)Integerselect) AsyncSession)schemas)settings)FileProcessingErrorUnsupportedFileTypeErrorValidationExceptionMessagec.eZdZdedefdZdedefdZdS) FileProcessorcontentreturnc KdSN)selfrs 7/DATA/AppData/hermes/projects/honcho/src/utils/files.py extract_textzFileProcessor.extract_texts content_typecdSrrrrs rsupports_file_typez FileProcessor.supports_file_typesrN)__name__ __module__ __qualname__bytesstrrboolr rrrrrs<<%s  U%nnW55OO! U U U%&JKKQT T U$$&& 2 K::o..DD# K K K%&@AAs J Kzz$Uz333s( =8=A-- B 7BB Nr9rrrrErE:sX2s2t22224%4C444444rrEc:eZdZdZdedefdZdededzfdZdS)FileProcessingServicecbtttg|_dSr)r(r;rE processors)rs r__init__zFileProcessingService.__init__Ss$ NN OO OO0 rfilerc4K|d{V}|dd{V||jpd}|s)t d|jdd|jD||d{VS)z7Extract text from uploaded file without saving to disk.NrrHzUnsupported file type: z. Supported types: c&g|]}|jjSr) __class__r!).0ps r zBFileProcessingService.extract_text_from_upload..es PPPjkQRQ\QePPPr)readseek_get_processorrrrTr)rrVr processors rextract_text_from_uploadz.FileProcessingService.extract_text_from_upload[s ######iill''(9(?R@@  *B$*;BBPPoso~PPPBB ++G444444444rrNcL|jD]}||r|cSdSr)rTr )rrr`s rr_z$FileProcessingService._get_processorjs? ! !I++L99 !     !tr) r!r"r#rUrr%rarr_rrrrRrRRsl    5: 5# 5 5 5 53=43GrrR\r8 max_charsrct||kr|gSg}d}|t|kr||z}|t|kr|||dnj|}dD]3}||||}||kr|t|z}n4|||||}|t|k|S)z6Split text into chunks that fit within message limits.rN)r, z.  )lenr2rfind)r8rdchunks current_posend_pos break_pos delimiterlast_delimiters rsplit_text_into_chunksrpqs 4yyIv FK D ! ! ) c$ii   MM${||, - - -  2  I!ZZ ;HHN ++*S^^; ,  d;y01222 # D ! !& Mrdbworkspace_namefile_id session_namec .Kddlm}m}ddlm}t |||j|k||j d|k}|r||j |k}| ||j d t}||d{V}t|S)zAGet all messages for a specific document, ordered by chunk_index.r)and_funcrrs chunk_indexN) sqlalchemyrvrw src.modelsrr whererrjsonb_extract_path_textinternal_metadatartorder_bycastrexecutelistscalarsall) rqrrrsrtrvrwrqueryresults rget_file_messagesrs0&%%%%%%%"""""" 7OO ! !   "n 4  ( ()BI N N     EB G0L@AA NN $$W%> NNSS     E ::e$$ $ $ $ $ $ $F   $$&& ' ''rrVpeer_idmetadata configuration created_atcKt}g}||d{V}t||} t} t | D]\} } | pd} t j| ||||}| |j| t| |j |j | |zt| dz|zt|gd}| ||d|st|S)a Process an uploaded file and prepare message creation data. This function extracts text from a file, splits it into chunks, and prepares the data needed to create messages. Args: file: Uploaded file to process peer_id: ID of the peer creating the messages max_chars: Maximum characters per message chunk metadata: Optional metadata to associate with all messages created from this file configuration: Optional configuration to associate with all messages created from this file created_at: Optional created_at timestamp to use for all messages created from this file Returns: List of dictionaries containing message_create and file_metadata Raises: HTTPException: If file processing fails N)rdrH)rrrrrr+)rsfilenamerx total_chunksoriginal_file_sizerchunk_character_range)message_create file_metadata)rRrarpgenerate_nanoidr/r MessageCreaterrhsizerminr2r )rVrrdrrrfile_processorall_message_dataextracted_textrjrsichunkmessage_contentrrs r!process_file_uploads_for_messagesrsM:+,,N-/*BB4HHHHHHHHN$Ni H H HFGf%%   5+2!.#'!     KK"&) -I QUi'^)<)<==&    "0!.       $!### r)rcr),datetimeloggingiortypingrrfastapirnanoidrrryrr sqlalchemy.ext.asyncior srcr src.configr src.exceptionsr rr src.schemasr getLoggerr!loggerrr(r;rErRr%intrrprMAX_MESSAGE_SIZEdictMessageConfigurationrrrrrs ......&&&&&&&&//////    8 $ $AAAAAHAAA + + + + + + + + 7 7 7 7 7 7 7 7444444440>cD $ (((((* (  '] ((((H.&*9=+/ LL L LL38nt# L /$6 L !D( L $sCx.LLLLLLr