yjl.ddlmZddlmZddlmZddlmZmZmZddl m Z ddl m Z dedefd Z d eeeeezeeeefzdeefd ZGd d eZGddeZGddeZGddeZGddeZGddeZGddeeZGddeeZGddeeZGddeeZGd d!eZd"eeefd#edefd$Zd%S)&)Sequence)datetime)Any) BaseModelFieldfield_validator)models)parse_datetime_iso timestampreturnc0|ddS)z] Remove microseconds and timezone info from a datetime for stable string formatting. rN) microsecondtzinfo)replace)r s @/DATA/AppData/hermes/projects/honcho/src/utils/representation.py _strip_microseconds_and_timezoner s   4  8 88 message_idscg}|D]J}t|ttzr||5||Kt t |S)aV Flatten message_ids that may be in old tuple format or nested list format. This handles backwards compatibility with the old schema where message_ids was list[tuple[int, int]] representing ranges, and the new schema where it's list[int] representing individual message IDs. Args: message_ids: Either a flat list of ints, nested list, or list of tuples Returns: A flat list of unique message IDs, sorted Examples: [1, 2, 3] -> [1, 2, 3] [[1, 2], [3, 4]] -> [1, 2, 3, 4] [(105, 105)] -> [105] [[105, 105]] -> [105] ) isinstancelisttupleextendappendsortedset)rresultitems rflatten_message_idsrsn,F   dTE\ + + MM$     MM$     #f++  rcheZdZUeddZeed<eed<ee ed<dZ edzed<dS) ObservationMetadataz Document ID for this observation)default descriptionid created_atrN session_name) __name__ __module__ __qualname__rr%str__annotations__rrintr'rrr!r!4sbeB,NOOOBOOOc#L#*#####rr!c2eZdZUedZeed<dS)ExplicitObservationBasezThe explicit observationr$contentN)r(r)r*rr2r+r,r.rrr0r0;s05%?@@@GS@@@@@rr0ceZdZUedeZeeed<edeZeeed<edZ eed<d S) DeductiveObservationBasez7Document IDs of premise observations for tree traversalr$default_factory source_idsz'Human-readable premise text for displaypremiseszThe deductive conclusionr1 conclusionN) r(r)r*rrr7r+r,r8r9r.rrr4r4?s!EMJS  %=Hd3ie(BCCCJCCCCCrr4ceZdZUdZedeZeeed<edeZ eeed<edd Z eed <ed Z eed <edd Z eed<dS)InductiveObservationBasez\Base model for inductive observations - patterns, generalizations, and personality insights.z6Document IDs of source observations for tree traversalr5r7z&Human-readable source text for displaysourceszSType of pattern: 'preference', 'behavior', 'personality', 'tendency', 'correlation'pattern)r$r# pattern_typez'The inductive generalization or patternr1r9z)Confidence level: 'high', 'medium', 'low'medium confidenceN) r(r)r*__doc__rrr7r+r,r<r>r9r@r.rrr;r;Ksff!ELJS <GT#YiL#e(QRRRJRRRe?Jrr;ceZdZUdZedeZeeed<edeZ eeed<edZ eed <d S) ContradictionObservationBasezVBase model for contradiction observations - when user has made conflicting statements.z.Document IDs of the contradicting observationsr5r7z3Human-readable text of the contradicting statementsr<z Description of the contradictionr1r2N) r(r)r*rArrr7r+r,r<r2r.rrrCrCas``!EDJS IGT#Y5%GHHHGSHHHHHrrCceZdZUdZedeZeeed<e dde de de fd Z d S) PromptRepresentationz\ The representation format that is used when getting structured output from an LLM. zFacts LITERALLY stated by the user - direct quotes or clear paraphrases only, no interpretation or inference. Example: ['The user is 25 years old', 'The user has a dog named Rover']r5explicitbefore)modevr c|gS|S)zGConvert None to empty list - handles LLMs returning null instead of [].r.)clsrIs rconvert_none_to_empty_listz/PromptRepresentation.convert_none_to_empty_listys 9IrN) r(r)r*rArrrFr0r,r classmethodrrLr.rrrErEos/4eL///Hd*+ _Zh///33[0/rrEcFeZdZdZdefdZdefdZdefdZde de fdZ dS) ExplicitObservationz/Explicit observation with content and metadata.r c@dt|jd|jS)N[] )rr&r2selfs r__str__zExplicitObservation.__str__s%V3DODDVV VVVrcl|jr d|jdnd}|dt|jd|jS)LFormat with ID prefix for use by agents that need to reference observations.[id:rRr"rQ)r%rr&r2)rT id_prefixs r str_with_idzExplicitObservation.str_with_idsL*.'9&47&&&&r aa>tOOaaSWS_aaarcDt|j|j|jfS)zD Make ExplicitObservation hashable for use in sets. hashr2r&r'rSs r__hash__zExplicitObservation.__hash__s T\4?D4EFGGGrotherct|tsdS|j|jko|j|jko|j|jkS)z Define equality for ExplicitObservation objects. Two observations are equal if all their fields match. F)rrOr2r&r'rTr_s r__eq__zExplicitObservation.__eq__sR %!455 5 LEM ) 85#33 8!U%77 rN) r(r)r*rAr+rUrZr-r^objectboolrbr.rrrOrOs99WWWWWbSbbbb H#HHHH  F  t       rrOcReZdZdZdefdZdefdZdefdZdefdZ de de fdZ d S) DeductiveObservationzODeductive observation with multiple premises and one conclusion, plus metadata.r cdd|jD}dt|jd|jd|S)N c3 K|] }d|V dS - Nr..0premises r z/DeductiveObservation.__str__..+!R!R"47"4"4!R!R!R!R!R!RrrQrR)joinr8rr&r9rT premises_texts rrUzDeductiveObservation.__str__sP !R!RDM!R!R!RRR j3DODDjjjj[hjjjrc|jr d|jdnd}dd|jD}|dt|jd|jd|S)rWrXrRr"rhc3 K|] }d|V dSrjr.rls rroz3DeductiveObservation.str_with_id..rprrQ)r%rqr8rr&r9)rTrYrss rrZz DeductiveObservation.str_with_idsw*.'9&47&&&&r  !R!RDM!R!R!RRR uu>tOOuuSWSbuufsuuurcbdd|jD}|jd|S)Nrhc3 K|] }d|V dSrjr.rls rroz9DeductiveObservation.str_no_timestamps..rpr)rqr8r9rrs rstr_no_timestampsz&DeductiveObservation.str_no_timestampss: !R!RDM!R!R!RRR /44]444rcDt|j|j|jfS)zr Make DeductiveObservation hashable for use in sets. NOTE: premises are not included in the hash. r]r9r&r'rSs rr^zDeductiveObservation.__hash__s T_dot7HIJJJrr_ct|tsdS|j|jko|j|jko|j|jkS)z Define equality for DeductiveObservation objects. Two observations are equal if all their fields match -- NOTE: premises are not included in the equality check. F)rrfr9r&r'ras rrbzDeductiveObservation.__eq__sS %!566 5 Ou/ / 85#33 8!U%77 rN r(r)r*rAr+rUrZrxr-r^rcrdrbr.rrrfrfsYYkkkkkvSvvvv 535555K#KKKK  F  t       rrfcReZdZdZdefdZdefdZdefdZdefdZ de de fdZ d S) InductiveObservationzPInductive observation with sources, pattern type, and confidence, plus metadata.r cd}|jr)d|jD}dd|z}dt|jd|jd|j|S)Nr"cg|]}d|Srkr.rmsources r z0InductiveObservation.__str__..!III&-V--IIIrrhrQ] [rR)r<rqrr&r@r9rT sources_text source_liness rrUzInductiveObservation.__str__sq < :IIDLIIIL$))L"9"99L{3DODD{{{{\`\k{my{{{rc|jr d|jdnd}d}|jr)d|jD}dd|z}|dt|jd|jd|j|S)rWrXrRr"cg|]}d|Srr.rs rrz4InductiveObservation.str_with_id..rrrhrQr)r%r<rqrr&r@r9rTrYrrs rrZz InductiveObservation.str_with_ids*.'9&47&&&&r  < :IIDLIIIL$))L"9"99LGG>tOOGGTXTcGGgkgvGyEGG Grcd}|jr)d|jD}dd|z}d|jd|j|S)Nr"cg|]}d|Srr.rs rrz:InductiveObservation.str_no_timestamps..rrrhrQrR)r<rqr@r9rs rrxz&InductiveObservation.str_no_timestampss[ < :IIDLIIIL$))L"9"99LE4?EEdoE|EEErcDt|j|j|jfS)z3Make InductiveObservation hashable for use in sets.rzrSs rr^zInductiveObservation.__hash__sT_dot7HIJJJrr_ct|tsdS|j|jko|j|jko|j|jkS)z1Define equality for InductiveObservation objects.F)rr~r9r&r'ras rrbzInductiveObservation.__eq__sQ%!566 5 Ou/ / 85#33 8!U%77 rNr|r.rrr~r~sZZ|||||GSGGGGF3FFFFK#KKKK F t      rr~cReZdZdZdefdZdefdZdefdZdefdZ de de fdZ d S) ContradictionObservationz[Contradiction observation - notes when user has made conflicting statements, plus metadata.r cd}|jr)d|jD}dd|z}dt|jd|j|S)Nr"cg|]}d|Srr.rs rrz4ContradictionObservation.__str__..rrrhrQ] CONTRADICTION: )r<rqrr&r2rs rrUz ContradictionObservation.__str__sg < :IIDLIIIL$))L"9"99Ls3DODDssW[Wcseqsssrc|jr d|jdnd}d}|jr)d|jD}dd|z}|dt|jd|j|S)rWrXrRr"cg|]}d|Srr.rs rrz8ContradictionObservation.str_with_id..rrrhrQr)r%r<rqrr&r2rs rrZz$ContradictionObservation.str_with_ids*.'9&47&&&&r  < :IIDLIIIL$))L"9"99L~~>tOO~~bfbn~p|~~~rc~d}|jr)d|jD}dd|z}d|j|S)Nr"cg|]}d|Srr.rs rrz>ContradictionObservation.str_no_timestamps..rrrhzCONTRADICTION: )r<rqr2rs rrxz*ContradictionObservation.str_no_timestampssQ < :IIDLIIIL$))L"9"99L==|===rcDt|j|j|jfS)z7Make ContradictionObservation hashable for use in sets.r\rSs rr^z!ContradictionObservation.__hash__ sT\4?D4EFGGGrr_ct|tsdS|j|jko|j|jko|j|jkS)z5Define equality for ContradictionObservation objects.F)rrr2r&r'ras rrbzContradictionObservation.__eq__ sP%!9:: 5 LEM ) 85#33 8!U%77 rNr|r.rrrrseetttttS>3>>>>H#HHHH F t      rrc eZdZUdZedeZeeed<edeZ ee ed<edeZ ee ed<ed eZ eeed <d efd Zd efd Zd!dZ d"dddedzfdZd efdZd efdZd efdZd#ded efdZedeejd dfdZedddeededed df d Z dS)$Representationam A Representation is a traversable and diffable map of observations. At the base, we have a list of explicit observations, derived from a peer's messages. From there, deductive observations can be made by establishing logical relationships between explicit observations. In the future, we can add more levels of reasoning on top of these. All of a peer's observations are stored as documents in a collection. These documents can be queried in various ways to produce this Representation object. Additionally, a "working representation" is a version of this data structure representing the most recent observations within a single session. A representation can have a maximum number of observations, which is applied individually to each level of reasoning. If a maximum is set, observations are added and removed in FIFO order. zFacts LITERALLY stated by the user - direct quotes or clear paraphrases only, no interpretation or inference. Example: ['The user is 25 years old', 'The user has a dog']r5rFzConclusions that MUST be true given explicit facts and premises - strict logical necessities. Each deduction should have premises and a single conclusion. deductivezPatterns, generalizations, and personality insights inferred from multiple observations. Higher-level reasoning created by the Dreamer agent. inductivezxConflicting statements made by the user that need clarification. The dialectic agent should surface these when relevant. contradictionr ct|jdkoGt|jdko/t|jdkot|jdkS)z7 Check if the representation is empty. rlenrFrrrrSs ris_emptyzRepresentation.is_empty<sc   ! # -DN##q( -DN##q( -D&''1,  rct|jt|jzt|jzt|jzS)zP Return the total number of observations in the representation. rrSs rrzRepresentation.lenGsO   $.!! "$.!! "$$%% & rr_ct}fd|jD|_fd|jD|_fd|jD|_fd|jD|_|S)z Given this and another representation, return a new representation with only observations that are unique to the other. Note that this only removes literal duplicates, not semantically equivalent ones. c&g|] }|jv |Sr.rFrmorTs rrz6Representation.diff_representation..Xs%MMMqat}6L6L6L6L6Lrc&g|] }|jv |Sr.)rrs rrz6Representation.diff_representation..Y%PPP8O8O!8O8O8Orc&g|] }|jv |Sr.)rrs rrz6Representation.diff_representation..Zrrc&g|] }|jv |Sr.)rrs rrz6Representation.diff_representation..[s-   at7I.I.IA.I.I.Ir)rrFrrr)rTr_diffs` rdiff_representationz"Representation.diff_representationRs MMMMENMMM PPPPU_PPPPPPPU_PPP    *    rNmax_observationsctt|j|jz|_tt|j|jz|_tt|j|jz|_tt|j|jz|_|jd|jd|jd|jd|rV|j| d|_|j| d|_|j| d|_|j| d|_dSdS)a Merge another representation object into this one. This will automatically deduplicate explicit, deductive, inductive, and contradiction observations. This *preserves order* of observations so that they retain FIFO order. NOTE: observations with the *same* timestamp will not have order preserved. That's fine though, because they are from the same timestamp... c|jSNr&xs rz5Representation.merge_representation..qsr)keyc|jSrrrs rrz5Representation.merge_representation..r!,rc|jSrrrs rrz5Representation.merge_representation..srrc|jSrrrs rrz5Representation.merge_representation..tsalrN)rrrFrrrsort)rTr_rs rmerge_representationz#Representation.merge_representation`syS!?@@AA c$.5?"BCCDDc$.5?"BCCDD!#d&85;N&N"O"OPP 55666  6 6777  6 6777 $:$:;;;  H M+;*;*<*<=DM!^-=,=,>,>?DN!^-=,=,>,>?DN!%!35E4E4F4F!GD     H Hrc(g}|dt|jdD]\}}||d| |d|dt|jdD]\}}||d| |d|dt|jdD]\}}||d| |d|dt|jdD]\}}||d| |dd|S) a Format representation into a clean, readable string for LLM prompts. NOTE: we always strip subsecond precision from the timestamps. Returns: Formatted string with clear sections and bullet points including temporal metadata Example: EXPLICIT: 1. [2025-01-01 12:00:00] The user has a dog named Rover 2. [2025-01-01 12:01:00] The user's dog is 5 years old 3. [2025-01-01 12:05:00] The user is 25 years old DEDUCTIVE: 1. [2025-01-01 12:01:00] Rover is 5 years old - The user has a dog named Rover - The user's dog is 5 years old EXPLICIT: . r" DEDUCTIVE: INDUCTIVE: CONTRADICTION: rh)r enumeraterFrrrrqrTpartsi observations rrUzRepresentation.__str__|s& ]###' q99 0 0NA{ LLA.... / / / / R ^$$$':: 0 0NA{ LLA.... / / / / R ^$$$':: 0 0NA{ LLA.... / / / / R '((('(:A>> 0 0NA{ LLA.... / / / / Ryyrcg}|dt|jdD]1\}}||d|2|d|dt|jdD]1\}}||d|2|d|dt|jdD]1\}}||d|2|d|dt|jdD]1\}}||d|2|dd|S) a Format representation with observation IDs for agents that need to reference/delete observations. Returns: Formatted string with IDs included Example: EXPLICIT: 1. [id:abc123] [2025-01-01 12:00:00] The user has a dog named Rover 2. [id:def456] [2025-01-01 12:01:00] The user's dog is 5 years old DEDUCTIVE: 1. [id:ghi789] [2025-01-01 12:01:00] Rover is 5 years old - The user has a dog named Rover - The user's dog is 5 years old INDUCTIVE: 1. [id:jkl012] [2025-01-01 12:05:00] [high] User tends to be methodical - id:abc123 - id:def456 rrrr"rrrrh)rrrFrZrrrrqrs r str_with_idszRepresentation.str_with_idss& ]###' q99 > >NA{ LLA<<!8!8!:!:<< = = = = R ^$$$':: > >NA{ LLA<<!8!8!:!:<< = = = = R ^$$$':: > >NA{ LLA<<!8!8!:!:<< = = = = R '((('(:A>> > >NA{ LLA<<!8!8!:!:<< = = = = Ryyrcg}|dt|jdD]$\}}||d|j%|d|dt|jdD]1\}}||d|2|d|dt|jdD]1\}}||d|2|d|dt|jdD]1\}}||d|2|dd|S) a Format representation into a clean, readable string for LLM prompts... but without timestamps. Returns: Formatted string with clear sections and bullet points including temporal metadata Example: EXPLICIT: 1. The user has a dog named Rover 2. The user's dog is 5 years old 3. The user is 25 years old DEDUCTIVE: 1. Rover is 5 years old - The user has a dog named Rover - The user's dog is 5 years old INDUCTIVE: 1. [high] User tends to be methodical - id:abc123 - id:def456 rrrr"rrrrh) rrrFr2rrxrrrqrs rrxz Representation.str_no_timestampss* ]###' q99 8 8NA{ LLA66!466 7 7 7 7 R ^$$$':: D DNA{ LLABB!>!>!@!@BB C C C C R ^$$$':: D DNA{ LLABB!>!>!@!@BB C C C C R '((('(:A>> D DNA{ LLABB!>!>!@!@BB C C C C RyyrF include_idscg}|jrJ|d|jD]}|||d|jr|d|jD]}|r|jr d|jdnd}t |j}||d|d|j|jr7|d|jD]}|d||d|d|jr8|d |jD]}|r|jr d|jdnd}||d |j d |j|j r|d |j |j r|d |j ddD]}|d|t|j dkr.|dt|j dz d|d|d|j r|d|j D]}|r|jr d|jdnd}||d|j|j r7|d|j D]}|d||d|dd|S)a& Format a Representation object as markdown. NOTE: we always strip subsecond precision from the timestamps. Args: include_ids: If True, include observation IDs for use with get_reasoning_chain Returns: Formatted markdown string z## Explicit Observations r"z## Deductive Observations rXrRrQz Premises:z - z## Inductive Observations z **Pattern** [z]: z **Type**: z **Sources**:Nz - ... and z morez## Contradictions z **CONTRADICTION**: z **Conflicting statements**:rh)rFrrr%rr&r9r8rr@r>r<rrr2rq)rTrrobsrYr rnrs rformat_as_markdownz!Representation.format_as_markdowns =  LL5 6 6 6} ' ' X&&&& LL    >  LL6 7 7 7~ ! !1<OO-36----R   LL6 7 7 7~ ! !1<OO-36----R   SSSS3>SS#ELL!C1A!C!CDDD;RLL!2333"%+bqb/77 %5V%5%566663;''!++ %PS5E5E5I%P%P%PQQQ R    LL       LL. / / /) ! !1<OO-36----R  LLs{LLMMM;7LL!ABBB"%+77 %5V%5%56666 R    LL   yyr documentscp|d|Dd|Dd|Dd|DS)Nc g|]n}|jdk t|jt|j|j|jt|jdg|j oS)rFr)r%r&r2rr') levelrOr%_safe_datetime_from_metadatainternal_metadatar&r2rgetr'rmdocs rrz1Representation.from_documents..Is9 **$v;-s~   K 3-11-DD!!"%!1   +**rc\g|]}|jdk t|jt|j|j|jt|jdg|j |j p|jdg|jdgS)rr premise_idsr8)r%r&r9rr'r7r8) rrfr%rrr&r2rrr'r7rs rrz1Representation.from_documents..Xs 9 ++!%v;-s~   #{ 3-11-DD!!"%!1"~ D,00CC 266z2FF ,++rcg|]}|jdk t|jt|j|j|j|jdg|j|j p|jdg|jdg|jdd|jdd S) rrr7r<r>r=r@r?) r%r&r9rr'r7r<r>r@) rr~r%rrr&r2rr'r7rs rrz1Representation.from_documents..ks 9 ++!%v;-s~   #{ # 5 9 9- L L!$!1"~ C,00rBB155iDD!$!6!:!:>9!U!U"488xPP ,++rcBg|]}|jdk t|jt|j|j|j|jdg|j|j p|jdg|jdgS)rrr7r<)r%r&r2rr'r7r<) rrr%rrr&r2rr'r7rs rrz1Representation.from_documents..~s9//)v;-s~   K # 5 9 9- L L!$!1"~ C,00rBB155iDD   0//r)rFrrrr.)rKrs rfrom_documentszRepresentation.from_documentsFss% %!& %!&%mG G G G rprompt_representationrErr'r&cJ|fd|jDggS)z/Convert PromptRepresentation to Representation.c@g|]}t|jS))r2r&rr')rOr2)rmer&rr's rrz=Representation.from_prompt_representation..sH $I) +!- r)rFrrr)rKrrr'r&s ```rfrom_prompt_representationz)Representation.from_prompt_representations]s/7    r)r_rr rr)F)!r(r)r*rArrrFrOr,rrfrr~rrrdrr-rrrr+rUrrxrrMrr Documentrrrr.rrrrs$+0%@+++Hd&'-2Eq---It()-2Ed---It()5:EO555M401  $       S         GKHH%H9 >>sAA54A5N) collections.abcrrtypingrpydanticrrrsrcr src.utils.formattingr rrr-rrr!r0r4r;rCrErOrfr~rrdictr+rr.rrrs+$$$$$$666666666633333399X9999cT$s)_,tE#s(O/DD #YD$$$$$)$$$AAAAAiAAA D D D D Dy D D Dy, I I I I I9 I I I9&     13F   >" " " " " 35H" " " J& & & & & 35H& & & R& & & & & ;=P& & & RN N N N N YN N N b ?CH~?:B? ??????r