yjJ ddlmZddlmZddlmZddlmZddlmZ ddl m Z m Z m Z mZmZmZmZmZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd l m!Z!ddl"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+ddl,m-Z-ee.Z/deej0de1de2ej0fdZ3dede2e4e2ej0e2ej0fddfdZ5dede6de6de2e6fdZ7de2e ed e1de e4ej0fd!Z8dede6d"e2ej0d#e1de2e4e2ej0e2ej0ff d$Z9dede2ej:de6d%e6de2ej0f d&Z; dSde6d%e6d(e dTdede6d%e6d,e1d-e1dzd e1dzde2ej0fd.Z? dUdede6d%e6d/e1d0e1dzde2ej0f d1Z@dede6d%e6d2e1de1f d3ZAdede6d%e6d2e6dej0dzf d4ZBded5ejCde6d%e6d2e6dee2e6d:edzd;edzde2ej0f d?ZGdddd@dAdBdede6d%e6dzd8e2eEd9e2e6dzd:edzd;edzde1d#e1de2e4e2ej0e2ej0ffdCZHd@dAddddDde6d%e6dzd8e2eEde1d#e1dEe6d:edzd;edzdFe6dzde2e4e2ej0e2ej0ffdGZI dVde6d%e6dzdHe6de1d#e1dIe2eEdzdFe6dzde2e4e2ej0e2ej0ffdJZJ dWdede6d%e6dzdKe6de1d#e1d9e2e6dzde2e4e2ej0e2ej0ffdLZK dWde6d%e6dzdKe6de1d#e1dFe6dzde2e4e2ej0e2ej0ffdMZL dXdede6d%e6dzd:edzd;edzde1dPe6dFe6dzde2ej0fdQZM dYde6d%e6dzdHe6d:edzd;edzde1d#e1dIe2eEdzdFe6dzde2e4e2ej0e2ej0ffdRZNdS)Z)Sequence)datetime) getLogger)Any)generate) ColumnElementSelectand_funcor_selecttextupdate) AsyncSession)modelsschemas)settings) tracked_db)embedding_client)VectorStoreError)EmbeddingCallPurpose) apply_filter)ILIKE_ESCAPE_CHARescape_ilike_pattern)embedding_call_purpose) VectorRecordget_external_vector_store)get_or_create_sessionmessageslimitreturnct}g}|D]O}|j|vrD||j||t ||krnP|S)z:Deduplicate messages by public_id, preserving input order.)set public_idaddappendlen)r r!seenresultmsgs 8/DATA/AppData/hermes/projects/honcho/src/crud/message.py_deduplicate_messagesr-spUUD#%F = $ $ HHS] # # # MM#   6{{e## MdbsnippetsNct}|D]L\}}g||D]@}t|}||vr||||AMdS)zFDetach snippet messages from the session, guarding against duplicates.N)r$idexpunger&)r/r0r)matchescontextr+obj_ids r,_expunge_snippetsr7)sUUD$'W'w'  CWWF~~ JJsOOO HHV      r.workspace_name peer_namecKttjjjtjjj|ktjjj|k}| |d{V}d| DS)zGet all session names where a peer has any membership record. Any membership record (regardless of joined_at/left_at) grants visibility to all messages in that session. Ncg|] }|d S)r).0rows r, z*get_peer_session_names..Hs + + +sCF + + +r.) r rsession_peers_tablec session_namewherer8r9distinctexecuteall)r/r8r9stmtr*s r,get_peer_session_namesrH7s v)+899 v)+:nL M M v)+5B C C   ::d## # # # # # #F + +fjjll + + ++r.base_conditions token_limitcttjjt jtjjtjj dj | }ttj |tjj|j jk |j j|kS)a Helper function to apply token limit logic to a message query. Creates a subquery that calculates running sum of tokens for most recent messages and returns a select statement that joins with this subquery to limit results based on token count. Args: base_conditions: List of conditions to apply to the base query token_limit: Maximum number of tokens to include in the messages Returns: Select statement with token limit applied )order_byrunning_token_sum)r rMessager2r sum token_countoverdesclabelrCsubqueryjoinrArM)rIrJtoken_subquerys r,_apply_token_limitrWKs&  N  HV^/ 0 0 T6>,1133T 4 4 U& ' '     !  v~ nfn/>3C3FF G G ~1[@ A Ar.matched_messagescontext_windowcK|sgSi}|D]0}||jg|1i}|D]\}}|dg} |D]m} | j|z } | j|z} | r<| | dddzkr'| d\} }}| t || g|| f| d<T| | | | gfn| ||<d|D}ttj  tj j |k t| tj jtj j}||d{V}i}|D]0}||jg|1g}|D]J\}} ||g}| D],\}fd|D}|||f-K|S)a Group matched messages by session, merge overlapping context ranges, and fetch context. Takes a list of matched messages and builds conversation snippets by: 1. Grouping matches by session name 2. Sorting matches within each session by sequence number 3. Merging overlapping context windows to avoid duplicate context 4. Fetching the full context for each merged range from the database Args: db: Database session workspace_name: Name of the workspace matched_messages: List of messages that matched a search query context_window: Number of messages before/after each match to include Returns: List of tuples: (matched_messages_in_range, context_messages) Each tuple represents a snippet where context_messages includes all messages in the merged range (including the matched messages), ordered chronologically. c|jSNseq_in_session)ms r,z(_build_merged_snippets..s 1#3r.)keyrc |g|]9\}}ttjj|kt d|D:S)c3hK|]-\}}}tjj||V.dSr\)rrNr^between)r= start_seqend_seq_s r, z4_build_merged_snippets...sQ- 7AN199)WMMr.)r rrNrBr )r= sess_name merged_rangess r,r?z*_build_merged_snippets..si    %I} N '9 4 1>      r.Nc>g|]}|jcxkrknn|Sr<r])r=r+rgrfs r,r?z*_build_merged_snippets..sH    2====g========r.) setdefaultrBr'itemssortr^maxr rrNrCr8r rLascrEscalarsrFget)r/r8rXrYsession_matchesr+session_rangesrjr4rkmatchstartend prev_startprev_end prev_matchessession_predicates context_stmtcontext_result by_sessionr0all_context_messages range_matchescontext_messagesrgrfs @@r,_build_merged_snippetsrps4  79OEE""3#3R88??DDDDNPN-335522 7 33 444EG  < 9E&7C <-*;A*>*B!B!B5B25F2 Hl#&&*l*E*% b!! $$eS5'%:;;;;$1y!!  )7(<(<(>(>    v~ v~,> ? ? s&' ( (  N ' + + - - N ) - - / /  ::l33333333N24J%%''++--@@c.33::3????  %3$8$8$:$:?? =)~~i<<1> ? ? -Iw      /     OO],<= > > > >  ? Or.rBc &Kd|D}t|tj|||d{V|t dd{V|t d||dd{V|t tjj  tjj |ktjj |k tjj dd{Vpd }g}t!|d D]p\}}||z} tj||j|j|jpi|t)t+|j|j| } || q|||d{V t6jrLd |D} | rTt;t<jj |d 5tCj"| d{V} dddn #1swxYwYni} tG} t6j$j%dkpt6j$j& }g}i}|D]} | '| j(g}t!|D]^\}}tj)| j| j(||| jd|r|nd}t+|}||f||<||_g}|r;|||*d{Vd|D}|d{V| |r|tWtj) tj)j,-|.dt_j0d d{V|d{Vn| 1d|}g}t!|D]a\}}||\}}|j2d|}|tg|ti||j2|j |jdb|r | 5||d{V|r|tWtj) tj)j,-|.dt_j0d d{V|d{Vn#tl$rtn8d|r|tWtj) tj)j,-|.tj)j9dzt_j0d{V|d{VYntt$rtn;d|r|tWtj) tj)j,-|.tj)j9dzt_j0d{V|d{VYnwxYwn:#tt$r-tn;dt+|||YnwxYw|S)a> Bulk create messages for a session while maintaining order. Args: db: Database session messages: List of messages to create workspace_name: Name of the workspace session_name: Name of the session to create messages in Returns: List of created message objects c@i|]}|jtjSr<)r9rSessionPeerConfigr=messages r, z#create_messages..s& T T TW  9 ; ; T T Tr.)namepeers)sessionr8NzSET LOCAL lock_timeout = '5s'zPSELECT pg_advisory_xact_lock(hashtext(:workspace_name), hashtext(:session_name)))r8rBrr)rw) rBr9content h_metadatar8r%rP created_atr^chi|]/}|j |j"|j|j0Sr<)rstripr%rs r,rz#create_messages..sO   ? (/'<'<'>'> !7?   r.api)r8parent_categorypgvectorpending)r message_idr8rBr9 sync_state embeddingcg|] }|j Sr<)r2)r=embs r,r?z#create_messages..Os E E EC E E Er.synced)r last_sync_at sync_attemptsrrh)rrBr9)r2rmetadataz:Vector store unavailable; leaving message vectors unsynced)rrz*Unexpected error upserting message vectorszUFailed to generate message embeddings for %s messages in workspace %s and session %s.)0 1 1 U-?+|;Xfn388:: ; ; U1XX          -/O$XQ777 , ,!)F!2n%'O'-2)%''G344)1     {++++JJ ))++U  "L .  .      $+(7=#1$) ,<+G(,,&&&&&&N "$$=$?$? ! %*j86,55 ) @B IK ". < < +// 0ErJJ 1::1F1F < <-NI$*$; + 3#.#8'5%1"-"7#,/K"U))QU%%%M""344G7Ey6Q*73%,,];;;; < (*M  F ,---hhjj       E E3D E E E ))++       %, &**v677v69==mLLMM'/)-*+ ))++%%%%%%%2FF~ 68$-.?$@$@  LGS0Fw0O-NI#&> D DN D DI"))$(&*9oo.1n030@-0]&&    "..-.3??%~) ."$** &v'> ? ?!&v'>'A'E'Em'T'T!U!U!'/71523"(""""###%))++-------+...X) ."$** &v'> ? ?!&v'>'A'E'Em'T'T!U!U!'282I2W&'3(15"(""""###%))++-------$ . . .(()UVVV( ."$** &v'> ? ?!&v'>'A'E'Em'T'T!U!U!'282I2W&'3(15"(""""###%))++------- .     c           sj=<]9I  ] I$$]'I$(I ] CV]C]0]2C]]]]4^ ^Freversefiltersmessage_count_limitcKtjj|ktjj|kg}|t tjj|}t |tj|}|tjj  |}|r8|tjj }n|tjj }nH|t||}t |tj|}|r7|tjj }n|tjj }nt tjj|}t |tj|}|r7|tjj }n6|tjj }|S)a Get messages from a session. If token_limit is provided, the n most recent messages with token count adding up to the limit will be returned. If message_count_limit is provided, the n most recent messages will be returned. If both are provided, message_count_limit will be used. Args: workspace_name: Name of the workspace session_name: Name of the session reverse: Whether to reverse the order of messages filters: Filter to apply to the messages token_limit: Maximum number of tokens to include in the messages message_count_limit: Maximum number of messages to include Returns: Select statement for the messages ) rrNr8rBr rCrrLr2rRr!rqrW)r8rBrrrJrrIrGs r, get_messagesrs6 %7#|3O &+vfn%%+_=D&.'::}}V^.335566<<=PQQ  :==!2!7!7!9!9::DD==!2!6!6!8!899DD  !/;??D&.'::  :==!2!7!7!9!9::DD==!2!6!6!8!899DD,vfn%%+_=D&.'::  :==!2!7!7!9!9::DD==!2!6!6!8!899D Kr.start_idend_idcK|dks|||ks|dkrgStjj|ktjj|kg}|rJ|t tjj|ktjj|kn(|tjj|k|r5t||}|tjj}n!ttjj |}| |d{V}t| S)aO Get messages from a session by primary key ID range. If end_id is not provided, all messages after and including start_id will be returned. If start_id is not provided, start will be beginning of session. Note: list is *inclusive* of the end_id message and start_id message. Args: db: Database session workspace_name: Name of the workspace session_name: Name of the session start_id: Primary key ID of the first message to return end_id: Primary key ID of the last message (exclusive) Returns: List of messages rN)rrNr8rBr'r r2rWrLr rCrErrrrF) r/r8rBrrrJrIrGr*s r,get_messages_id_rangersH2!||*F0B0BfPQkk  %7#|3O> "h.0AF0J K K     v~0H<===>!/;??}}V^.//+vfn%%+_=::d## # # # # # #F   $$&& ' ''r.rfrgcK|dks|||krgStjj|ktjj|kg}|J|t tjj|ktjj|kn(|tjj|kttjj| tjj }| |d{V}t| S)a Get messages from a session by seq_in_session range. This is useful for getting the last N messages in a session. Args: db: Database session workspace_name: Name of the workspace session_name: Name of the session start_seq: Sequence number of the first message to return (inclusive) end_seq: Sequence number of the last message to return (inclusive) Returns: List of messages ordered by seq_in_session rN)rrNr8rBr'r r^r rCrLrqrErrrrF)r/r8rBrfrgrIrGr*s r,get_messages_by_seq_ranger's;,1}},W1D1D  %7#|3O  -:-8       v~< IJJJ v~  ! &./3355 6 6 ::d## # # # # # #F   $$&& ' ''r.rcKttjjtjj|ktjj|ktjj|k}||d{V}|t|ndS)a  Get the sequence number of a message within a session. Args: db: Database session session_name: Name of the session message_id: Primary key ID of the message Returns: The sequence number of the message (1-indexed) Nr) r rrNr^rCr8rBr2rint)r/r8rBrrGseqs r,get_message_seq_in_sessionrYs$ v~,-- v~,> ? ? v~*l: ; ; v~ J. / /  IIdOO++++++C3s888A-r.czKttjtjj|ktjj|ktjj|k}||d{V}|Sr\) r rrNrCr8rBr%rEscalar_one_or_none)r/r8rBrrGr*s r, get_messagerts v~ v~,> ? ? v~*l: ; ; v~':5 6 6  ::d## # # # # # #F  $ $ & &&r.rcKt||||d{V}|td|j |j|_|d{V|S)N)r8rBrz,Message not found or does not belong to user)r ValueErrorrrr)r/rr8rBrhoncho_messages r,update_messagers' %! N GHHH$$+$4! ))++ r.rBallowed_session_names after_date before_datequery_embeddingrrrcKt}|gS|d|}i} |r|| d<n |d|i| d<|dup|du} | rdnd} ||||| z| r| nddgd{V} | sgSi} | D]'}|jd}|r || vrd| |<(t | }|sgS|S) zQuery the external vector store and return ordered message IDs. Multiple vector records can map to the same message (chunked embeddings), so we oversample from the vector store and deduplicate by message_id. NrrBinr)top_krinclude_attributes)rrqueryrrsrkeys)r8rr!rBrrrrrvector_filtershas_date_filters oversamplevector_resultsr)vrmid message_idss r,_search_messages_externalrs[677$ %::9nUUI%'NG)5~&&  **.0E)F~& "-HD1H&-AJ066j "0:d(> 7N  Dkool++  3d??DItyy{{##K  r.rrrcJKttjtjj|tjj|k}|r(|tjj|k}|r(|tjj|k}||d{V}d| Dfd|DS)z7Fetch messages by ID, preserving the supplied ordering.Nci|] }|j| Sr<)r%)r=r+s r,rz*_fetch_messages_by_ids..sKKKScmSKKKr.c(g|]}|v|Sr<r<)r=rmessages_by_ids r,r?z*_fetch_messages_by_ids..s( P P PC#:O:ON3 :O:O:Or.) r rrNrCr%rr8rrErrrF)r/r8rrr fetch_stmtr*rs @r,_fetch_messages_by_idsrs v~ v~'++K88 9 9 v~,> ? ? O%%fn&?:&MNN P%%fn&?;&NOO ::j)) ) ) ) ) ) )FKKFNN4D4D4H4H4J4JKKKN P P P P; P P PPr. )rrrr!rYcKttjtjtjjtjjktjj|k tjj | |dz} |r)| tjj |k} n9|7| tjj |} |r(| tjj|k} |r(| tjj|k} || d{V} t#| |} t)||| |d{VS)z?Run semantic message search against pgvector-backed embeddings.rN)r rrNrUrr%rrCr8rLrcosine_distancer!rBrrrEr-rrrFr) r/r8rBrrrrr!rY match_stmtr*rXs r,_search_messages_pgvectorrs$ v~   # N $(?(J J   v&5G H H &)3CCOTT U U uqy   %%  # 0L @    *%%  # 0 4 45J K K  O%%fn&?:&MNN P%%fn&?;&NOO ::j)) ) ) ) ) ) )F,V^^-=-=-A-A-C-CUKK' N,n      r.)r!rYrrobserveroperation_namerc ,Kd} |rY|sWt|d4d{V} t| ||d{V} dddd{Vn#1d{VswxYwY| sgStjjdkrtjjrt ||||| ||d{V} | sgSt|4d{V} t| || ||d{Vd|} t| || |d{V} t| | | cdddd{VS#1d{VswxYwYt|4d{V} t| |||| |||| d{V} t| | | cdddd{VS#1d{VswxYwYdS)zRun semantic message search with optional temporal filters. When observer is provided and session_name is None, results are scoped to sessions the observer has any membership record in. Nz .peer_scoperrr)rrrrr!rY) rrHrrrrrrrr7r)r8rBrr!rYrrrrrr/rrXr0s r,_semantic_search_messagesr s $/3 <<<==       *@NH++%%%%%% !                           % I!Z//H4I4R/5   %"7!#          In--       ,") + uf  4N$4nH b( + + +                             .))       R2   +"7!#)           "h'''                              s6A  AA;A D D$'D$0F F F rrc K||}nRttjj|5t j|d{V}dddn #1swxYwYt |||||d|d{VS)a Search for messages using semantic similarity and return conversation snippets. Each result includes matched messages plus surrounding context. Overlapping snippets within the same session are merged to avoid repetition. Args: workspace_name: Name of the workspace session_name: Name of the session (optional) query: Search query text limit: Maximum number of matching messages to return context_window: Number of messages before/after each match to include embedding: Optional pre-computed embedding observer: When provided and session_name is None, scope results to sessions this peer belongs to Returns: List of tuples: (matched_messages, context_messages) Each snippet may contain multiple matches if they were close together. Context messages are ordered chronologically and include the matched messages. Nr8zmessage.search_messages)rr!rYrrrrSEARCH_MESSAGESrrembedr)r8rBrr!rYrrrs r,search_messagesrhs<# $ 0 6)    B B%5$:5$A$AAAAAAAO  B B B B B B B B B B B B B B B +'%0      AAArcDKt|}ttjtjj|ktjjd|dt tjj  |}|r)|tjj |k}n9|7|tjj |}||d{V} t!| } t'||| |d{VS)z5Internal implementation of exact-text message search.%)escapeN)rr rrNrCr8rilikerrLrrRr!rBrrErrrrFr) r/r8rBrr!rYr escaped_textrr*rXs r,_grep_messages_internalrs~(--Lv~ v~,> ? ?  N " ( ()<\)<)<)Ktd4d{V}d}|r/|s-t|||d{V}|sgcdddd{VSt|||||||d{V}t|||cdddd{VS#1d{VswxYwYdS)a6 Search for messages containing specific text (case-insensitive substring match). Unlike semantic search, this finds EXACT text matches. Useful for finding specific names, dates, phrases, or keywords. Args: workspace_name: Name of the workspace session_name: Name of the session (optional - searches all sessions if None) text: Text to search for (case-insensitive) limit: Maximum number of matching messages to return context_window: Number of messages before/after each match to include observer: When provided and session_name is None, scope results to sessions this peer belongs to Returns: List of tuples: (matched_messages, context_messages) Each snippet may contain multiple matches if they were close together. zmessage.grep_messagesN)r)rrHrr7) r8rBrr!rYrr/rr0s r, grep_messagesrs6122b $  L *@NH++%%%%%% !) 1      "7          "h''')s!B  -B  BBrRordercKd}|r|st|||d{V}|sgSttjtjj|k} |r)| tjj|k} n9|7| tjj|} |r(| tjj|k} |r(| tjj|k} |dkr7| tjj } n6| tjj } | |} | | d{V} t| S)ap Get messages within a date range. Args: db: Database session workspace_name: Name of the workspace session_name: Name of the session (optional - searches all sessions if None) after_date: Return messages after this datetime before_date: Return messages before this datetime limit: Maximum messages to return order: Sort order - 'asc' for oldest first, 'desc' for newest first observer: When provided and session_name is None, scope results to sessions this peer belongs to Returns: List of messages within the date range Nrq)rHr rrNrCr8rBrrrLrqrRr!rErrrrF) r/r8rBrrr!rrrrGr*s r,get_messages_by_date_rangers8! &< ' ' ! ! ! ! ! ! % I &. ! ! ' '(E(W X XDRzz&.5EFF  *zz&.599:OPPQQCzz&.3zABBDzz&.3{BCC ~~}}V^6::<<==}}V^6;;==>> ::e  D::d## # # # # # #F   $$&& ' ''r.c K||} nRttjj|5t j|d{V} dddn #1swxYwYt ||| ||||d| d{VS)a Search for messages using semantic similarity with optional date filtering. Combines the power of semantic search with time constraints. Use after_date to find recent mentions, or before_date to find what was said before a certain point. Args: workspace_name: Name of the workspace session_name: Name of the session (optional) query: Search query text after_date: Only return messages after this datetime before_date: Only return messages before this datetime limit: Maximum number of matching messages to return context_window: Number of messages before/after each match to include embedding: Optional pre-computed embedding for the query observer: When provided and session_name is None, scope results to sessions this peer belongs to Returns: List of tuples: (matched_messages, context_messages) Each snippet may contain multiple matches if they were close together. Nr z message.search_messages_temporal)rrrr!rYrrr ) r8rBrrrr!rYrrrs r,search_messages_temporalr-sB# $ 0 6)    B B%5$:5$A$AAAAAAAO  B B B B B B B B B B B B B B B +'%9          r)FNNN)rNN)rN)rrNN)rrN)NNrrRN)NNrrNN)Ocollections.abcrrloggingrtypingrnanoidrr sqlalchemyrr r r r r rrsqlalchemy.ext.asynciorsrcrr src.configrsrc.dependenciesrsrc.embedding_clientrsrc.exceptionsrsrc.telemetry.eventsrsrc.utils.filterrsrc.utils.formattingrrsrc.utils.typesrsrc.vector_storerrrr__name__rrNrrr-tupler7strrHrWr MessageCreaterbooldictrrrrr MessageUpdaterfloatrrrrrrrrrr<r.r,r6s} $$$$$$......SSSSSSSSSSSSSSSSSSSS//////''''''111111++++++555555))))))HHHHHHHH222222DDDDDDDD****** 8   v~& /2  &.       $U4+?fnAU+U%V W      ,,,, #Y ,,,,("-,-"*a a  %V^$d6>&:: ;< aaaaH\\7()\\ \  &. \\\\D!%)"&* ????D[?#s(^d " ? t ? t ? E&. !"????L" /(/(/(/(/( /( $J /( t /( &./(/(/(/(l /(/(/(/(/( /( 4Z /(  &. /(/(/(/(d.... .  ....6 ' ' ' ' '  ^d ' ' ' '   "     : $.2"&#'6666%[6 6 * 6  9t+ 646D6 #Y6666|#'#' QQQQQcQ 4 Q D Q &.QQQQ>/3"&#'/////*/ %[ /  9t+ /4/D/ // %V^$d6>&:: ;<////n"&#'EEEE*E%[ E  E  EE4EDEDjE %V^$d6>&:: ;<EEEEX$(111*1 1  1  1 E{T! 1Dj1 %V^$d6>&:: ;<1111r.2""""*"  "  "  " 9t+" %V^$d6>&:: ;<""""R ///*/ /  /  / Dj / %V^$d6>&:: ;<////l#'#'7(7(7(7(*7(4 7( D 7(  7( 7(Dj7( &.7(7(7(7(|#'#'$(666*6 64 6 D 6  66E{T!6Dj6 %V^$d6>&:: ;<666666r.