yj?dZddlZddlmZmZddlmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZmZddl m!Z!ddl"m#Z#edZ$de%fdZ&ddde'e$de(de(de'e$fdZ) d+de*de'e+de(de,e*efdzde'e*f dZ- d+de de'e*de,e*efdzde'ej.fd Z/ d+de de*de'e+de(de,e*efdzde'ej.f d!Z0de d"e'ej.de*d#e*de'ej.f d$Z1de d%e*d&ee2ej.de(de'ej.f d'Z3dd(d)d%e*de,e*efdzde(de'ej.fd*Z4dS),z Reciprocal Rank Fusion (RRF) utilities for combining search results. RRF is a method to combine multiple ranked lists by computing the reciprocal of each item's rank in each list, then summing these reciprocal ranks. N)AnyTypeVar)Selectand_funcor_select) AsyncSession)models)settings) tracked_db)embedding_client)ValidationException)session_peers_table)EmbeddingCallPurpose) apply_filter)ILIKE_ESCAPE_CHARescape_ilike_pattern)embedding_call_purpose)get_external_vector_storeTreturncPtjjdkptjj S)zGReturn True when semantic message search can stay entirely in Postgres.pgvector)r VECTOR_STORETYPEMIGRATED8/DATA/AppData/hermes/projects/honcho/src/utils/search.py_uses_pgvector_message_searchr!s% "j0V8M8V4Vr<)k ranked_listsr#limitc|sgSi}|D]7}t|dD]$\}}||vrd||<||xxd||zz z cc<%8t|dd}d|D}|d|S) aj Combine multiple ranked lists using Reciprocal Rank Fusion (RRF). RRF assigns a score to each item based on the formula: RRF_score = sum(1 / (k + rank_i)) for all lists where the item appears Where: - k is a constant (typically 60) that controls the impact of high-ranked items - rank_i is the rank of the item in list i (1-indexed) Args: *ranked_lists: Variable number of ranked lists to combine k: RRF constant parameter (default: 60) limit: Maximum number of results to return Returns: list of items ranked by RRF score (highest score first) gg?c|dS)Nr'r)xs r z(reciprocal_rank_fusion..Fs AaDrT)keyreversecg|]\}}|Srr).0item_s r z*reciprocal_rank_fusion..Is / / /wtQd / / /rN) enumeratesorteditems) r#r%r$ rrf_scores ranked_listrankr/ sorted_itemsresults r reciprocal_rank_fusionr:$s&  "$J$11 #K33 1 1JD$:%%#& 4 t   q4x 0 0      1***,,..$OOOL0 /, / / /F &5&>rworkspace_nameembedding_queryfilterscKt}|gS|d|}i}|rd|vr |d|d<d|vr |d|d<||||dz|r|nddg d{V}|sgSi}|D]'} | jd} | r | |vrd|| <(t |S) z?Query the external vector store and return ordered message IDs.Nmessage session_id session_namepeer_id peer_name message_id)top_kr=include_attributes)rget_vector_namespacequerymetadatagetlistkeys) r;r<r%r=external_vector_store namespacevector_filtersvector_resultsseen_message_idsr9rEs r !query_external_vector_message_idsrSNs@677$ %::9nUUI%'N= 7 " "-4\-BN> *   *1)* 7N  (* 00_((66  0*,<<<+/ Z (  %%'' ( ((rdb message_idscK|sgSttjtjj|}t |tj|}||d{V}d| Dfd|DS)z5Fetch messages by ID and preserve the input ordering.Nci|] }|j| Sr) public_id)r.msgs r z)fetch_messages_by_ids..sEEEs sEEErc(g|]}|v|Srr)r.msg_idmessagess r r1z)fetch_messages_by_ids..s( M M M&H:L:LHV :L:L:Lr) r r MessagewhererXin_rexecutescalarsall)rTrUr=stmtr9r]s @r fetch_messages_by_idsrexs  &. ! ! ' '(@(D(D[(Q(Q R RD fng 6 6D::d## # # # # # #FEEfnn.>.>.B.B.D.DEEEH M M M M; M M MMrcKtjj|}t tjtjtjjtjjk tjj d tjj |k}|r4| }||d<t|tj|}|||dz}||d{V}t#} g} |D]:} | j| vr/| | j| | ;| d|S)a Perform semantic message search using pgvector in Postgres. Args: db: Database session workspace_name: Name of the workspace to search in embedding_query: Pre-computed embedding for the search query limit: Maximum number of results to return filters: Optional filters to apply to the message query Returns: list of messages ordered by semantic similarity N workspace_id)r MessageEmbedding embeddingcosine_distancer r^joinrXrEr_isnotr;copyrorder_byr%rasetrbrcaddappend) rTr;r<r%r= distance_exprrdinternal_filtersr9seendedupedrYs r _semantic_search_pgvectorrws(+5EEoVVM v~   # N $(?(J J   v&066t<< = = v&5G H H D"<<>>+9(D&.2BCC == ' ' - -eai 8 8D::d## # # # # # #FUUD$&G~~##%%   = $ $ HHS] # # # NN3    6E6?rr]rCcRK|sgStttjj|ktjj|k}||d{V}|}i}|D]9}|j} | |vrg|| <||  |j |j f:g} |D]N} | j|vr || jD]4\} } | j | kr$| | j | kr| | n5O| S)a Filter messages by peer perspective (temporal session membership). Only keeps messages from sessions where the peer was a member at the time the message was created (between joined_at and left_at). Args: db: Database session messages: List of messages to filter workspace_name: Name of the workspace peer_name: Name of the peer whose perspective to use Returns: Filtered list of messages N) r rr_cr;rCrarcrArr joined_atleft_at created_at)rTr]r;rCsession_memberships_queryr9 membershipssession_windows membershiprAfiltered_messagesrYrzr{s r _filter_by_peer_perspectiversn*   "## "$3~E F F "$.); < < ::788 8 8 8 8 8 8F**,,K9;O!YY !.  . .,.OL ) %,,j.BJDV-WXXXX/1    ? 2 2 #2#2B"C   Iw~**3>W#<#<!((--- rrIrdc Kttjd|}t|}|rztjjd|dt}| | tjj }nGtjdtjjdtjd|}t#|tjjd|dt} | |  tjtjtjdtjjtjd|d tjj }||}||d{V} t-| S)a2 Perform full-text search using PostgreSQL FTS and ILIKE fallback. Args: db: Database session query: Search query stmt: Base SQL query conditions limit: Maximum number of results to return Returns: list of messages ordered by text search relevance z&[~`!@#$%^&*()_+=\[\]{};\':"\\|,.<>/?-]%)escapeenglishz@@rN)boolresearchrr r^contentilikerr_ror|descr to_tsvectoropplainto_tsqueryrcoalescets_rankr%rarLrbrc) rTrIrdr%has_special_chars escaped_querysearch_conditionfulltext_query fts_conditioncombined_conditionr9s r _fulltext_searchrs& ;UCC )//M  !>177 ):8  $455>> N % * * , ,   U(FN4JKKNNtTT  E 2 2  !  N " ( ($M$$$-> )    $677@@ M $Y0FGG(E::    dff N % * * , ,   $))%00N::n-- - - - - - -F   $$&& ' ''r )r=r%c  Kttj d rMdvrHd dDrdvrdvrt d t ttjjt j jktjj t j j ktjj t j j ktt j jdtjj t j jkt j j k t% tj drAdpd}t)|t*r| rdzndz d d t,jrt)t*r t1t2jjd 5t9jd{V dddn #1swxYwYn5#t<$r(}t d t,jj d |d}~wwxYwtCstE  d{V dtFdtHtjf f d }tKd4d{V}||d{V}|D]}|&||cdddd{VS#1d{VswxYwYdS)a Search across message content using a hybrid approach with Reciprocal Rank Fusion (RRF). This function combines semantic search and full-text search results using RRF when both are available, providing better search results than either method alone. Args: query: Search query to match against message content filters: Optional filters to scope search (must include workspace_id for semantic search). Special filter 'peer_perspective' will search across all messages from sessions that the peer is/was a member of, filtered by the time window when they were actually in the session. limit: Maximum number of results to return Returns: list of messages that match the search query, ordered by RRF relevance or individual search relevance Raises: ValidationException: If query exceeds maximum token limit for embeddings Npeer_perspectivec&i|]\}}|dk ||S)rr)r.r#vs r rZzsearch..[s)OOODAqq._run_searchs57  # 4>3// 4 +,.. )B #1$3(# ***$$$$$$  *? 4 :#***$$$$$$ % )D$") **$$$$$$   ! !"2 3 3 3!1!) " " "        ./// ~   " ")>GGG G ~  ! # #!!$VeV, , rzsearch.messages)'r r r^r4rrlrrrAryr;r|rzrr{is_r_rCrrKrrr rrrSEARCH_MESSAGESvaluerembed ValueError EMBEDDINGMAX_INPUT_TOKENSr!rSr rLr expunge)rIr=r%workspace_valueer managed_dbcombined_resultsr?rrrrrdr;s``` @@@@@@r rr:s4 &. ! !D)-J%00 '(: ;OOGMMOOOOO  ' ) ).>g.M.M%_  yy  +/B/D/QQ-1D1F1UU)-@-B-LL')155d;;N-1D1F1NN       %#%/3HH I I  fng 6 6D!%N-!++n55VEU9V9V os + + -,N"7FUQYYUQYN*.O-1:nc#B#B '$4:- % F F )9(>u(E(E"E"E"E"E"E"E  F F F F F F F F F F F F F F F    %^8J8[^^^   -.. )J- /$ ***$$$$$$ -\-d6>6J--------------^+,,       !,Z!8!8888888' ( (G   w ' ' ' '                               sN !I*I IIIII J'#J  J:-L:: MM)N)5__doc__rtypingrr sqlalchemyrrrrr sqlalchemy.ext.asyncior srcr src.configr src.dependenciesr src.embedding_clientrsrc.exceptionsr src.modelsrsrc.telemetry.eventsrsrc.utils.filterrsrc.utils.formattingrrsrc.utils.typesrsrc.vector_storerrrr!rLintr:rfloatdictrSr^rerwrtuplerrrrr rs 66666666666666//////''''''111111......******555555))))))HHHHHHHH222222666666 GCLLt=?'''$q''c's'tTUw''''\&* ')')')%[') ')#s(^d " ')  #Y ')')')')Z&*NNNcN#s(^d "N &. NNNN.&* 1111%[1  1 #s(^d " 1  &. 1111h776>"77 7  &. 7777t?(?( ?( v~& '?(  ?(  &. ?(?(?(?(J&* M M M M #s(^d "M   M  &. M M M M M M r