yj8RdZddlmZddlZddlmZddlZddlm Z m Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZmZejeZeGd d ZeGddZd.dZd/dZd0dZd0d Z d0d!Z!d1d%Z"d2d(Z#d3d*Z$d4d,Z%d4d-Z&dS)5z Surprisal-based observation sampling for dream processing. Computes geometric surprisal scores for observations using tree-based data structures, enabling targeted deductive reasoning on anomalous or novel observations. ) annotationsN) dataclass)funcselect) AsyncSession)models)settings)get_all_documents) tracked_db) SurprisalTree create_treec<eZdZUdZded<ded<ded<ded<d S) ObservationDataz?Plain data extracted from a Document ORM object (session-safe).stridcontentz str | Nonelevelznp.ndarray | list[float] embeddingN__name__ __module__ __qualname____doc____annotations__=/DATA/AppData/hermes/projects/honcho/src/dreamer/surprisal.pyrrsBII GGGLLL''''''rrc2eZdZUdZded<ded<ded<dS) SurprisalScorez/Container for observation with surprisal score.r observationfloat surprisal np.ndarrayrNrrrrrr%s<99    rrworkspace_namerobserverobservedreturnlist[SurprisalScore]c K td4d{V}t||||d{V}d|D}t|t|z }|rtd|ddddd{Vn#1d{VswxYwY|s%td|d|d|gSt jjjd z}t||kr0td t|d |d gSt|}|j d krt dgSt|} t||| } d| D} t| t| kr;tdt| t| z dt| } | ddt!dt| } t jjjdz}td|ddtd| dt'| d| dD]g\}}|jj}t|dkr |dddz}td |d!|jd"d#|jjd$|ht1| }td%t|dt|d&|dd'|rotd(d)|d*jd"d+zd,|d jd"d+zd-t3d.|Dt|z d"zntd/|S#t4$r+}t d0|d1gcYd}~Sd}~wwxYw)2ag Sample observations and compute surprisal scores. Workflow: 1. Fetch observations based on SAMPLING_STRATEGY (short DB scope) 2. Extract embeddings (already stored on objects, no DB needed) 3. Build tree structure using trees.create_tree() 4. Compute surprisal for each observation 5. Rank by surprisal (highest first) 6. Filter by threshold and take top N Args: workspace_name: Workspace identifier observer: Observer peer name observed: Observed peer name Returns: List of SurprisalScore objects, ranked by surprisal (highest first) zdream.surprisal.fetchN)dbr$r%r&cjg|]0}|j t|j|j|j|j1S)N)rrrr)rrrrr.0obss r z6sample_observations_with_surprisal..QsQ   =, vK)!m -,,rzSkipped z" observations with null embeddingszNo observations found for /zToo few observations (z < z!), skipping surprisal computationrzFailed to extract embeddingscvg|]6}tj|jtj|j4|7Sr)npisinfr"isnanr-ss rr/z6sample_observations_with_surprisal..{sM   RXak%:%: CE8AKCXCX    rz Filtered z invalid surprisal scoresc|jSNr")xs rz4sample_observations_with_surprisal..sQ[rT)keyreversedu0🎯 Surprisal computation complete. Taking top z.0f%zTop z, observations by normalized surprisal score:PMz...z #z [surprisal=z.3fz ] [level=z] z Selected: z observations (top z%)u📊 Filtered statistics: zmin=z, zmax=zmean=c3$K|] }|jV dSr9r:r6s r z5sample_observations_with_surprisal..s$<3:Q> |  / / / NNr\):):rr?Orrr   I)66 ?a   LL7 8 8 8I:&&+<TJJ      |  s6{{ * * NNVCKK#l*;*;;VVV    .l;; #8#8$GGGAs,--...*@3F UwUUUUVVV N5NNNOOO!"3FUF";Q??  HAu'/G7||b  !#2#,. KKgaggU_ggg5CTCZgg^egg    &&788 ^X ^ ^\):): ^ ^w ^ ^ ^ ^    L KK,7"/777786!.66667S#<<8<<<<O BOB)OAO$5OI;O P  PP P r*rlist[models.Document]cKtjjj}tjjj}tjjj}|dkrt ||||||d{VS|dkrt||||||d{VS|dkrt||||||d{VSt d|dt ||||||d{VS)a Fetch observations based on configured sampling strategy. Args: db: Database session workspace_name: Workspace identifier observer: Observer peer name observed: Observed peer name Returns: List of Document objects recent)r*r$r%r&limitlevelsNrandomallzUnknown sampling strategy: z, using 'recent') r rMrNSAMPLING_STRATEGY SAMPLE_SIZEINCLUDE_LEVELS_fetch_recent_observations_fetch_random_observations_fetch_all_observationsrKrL)r*r$r%r&strategy sample_sizerrs rrIrIs$~'9H.*6K ^ % 4F8/)            X  /)            U  ,)            OXOOOPPP/)           rrqintrr list[str]cKt||||rdd|iind|}||d{V}t|S)a Fetch most recent observations. Uses existing get_all_documents() query with level filtering. Args: db: Database session workspace_name: Workspace identifier observer: Observer peer name observed: Observed peer name limit: Maximum number of observations to fetch levels: Document levels to include Returns: List of Document objects ordered by created_at DESC rinN)r$r%r&filtersrq)r executelistscalarsrtr*r$r%r&rqrrstmtresults rrxrxs0 %-3=4.))    D::d## # # # # # #F   $$&& ' ''rcTKttjtjj|ktjj|ktjj|k}|r7|tjj|}| tj  |}| |d{V}t|S)a Fetch random sample of observations. Uses PostgreSQL's random() function for efficient random sampling. Args: db: Database session workspace_name: Workspace identifier observer: Observer peer name observed: Observed peer name limit: Maximum number of observations to fetch levels: Document levels to include Returns: List of Document objects in random order N)rrDocumentwherer$r%r&rin_order_byrrsrqrrrrtrs rryrys0 &/ " " ( (&.8 H, H,  D =zz&//33F;;<< == ' ' - -e 4 4D::d## # # # # # #F   $$&& ' ''rcpKttjtjj|ktjj|ktjj|ktjj  |}|r7|tjj |}| |d{V}t|S)a Fetch all observations up to limit. Orders by created_at DESC for consistency. Args: db: Database session workspace_name: Workspace identifier observer: Observer peer name observed: Observed peer name limit: Maximum number of observations to fetch levels: Document levels to include Returns: List of Document objects ordered by created_at DESC N)rrrrr$r%r&r created_atdescrqrrrrrrtrs rrzrz8s2 v  O *n < O $ 0 O $ 0  &/,1133 4 4 u =zz&//33F;;<<::d## # # # # # #F   $$&& ' ''rr_list[ObservationData]r#c|stjgSd|D}tj|tj}|S)z Extract embeddings from observations as numpy array. Args: observations: List of Document objects with embeddings Returns: np.ndarray of shape (N, 1536) containing embeddings cg|] }|j Sr)rr,s rr/z'_extract_embeddings..os===s}===r)dtype)r3arrayfloat32)r_embeddings_listembeddings_arrays rrPrPbsJ x||== ===OxrzBBB rrbr c|jdkr#ttjjjSttjjjtjjj}|||S)z Build tree structure from embeddings. Args: embeddings: np.ndarray of shape (N, embedding_dim) Returns: SurprisalTree configured per settings r) tree_typek)rQr r rMrN TREE_TYPErO batch_insert)rbrcs rrSrSusl!8>3=>>> .*4 . " )   D  j!!! Krrccg}t||dD]?\}}||}|t|||@|S)a) Compute surprisal score for each observation. Args: observations: List of ObservationData objects embeddings: np.ndarray of embeddings matching observations tree: Built SurprisalTree Returns: List of SurprisalScore objects (unfiltered, unsorted) F)strictr r"r)zipr"appendr)r_rbrcrdr.rr"s rrTrTsy $&FlJuEEE   YNN9--  ##        Mrrdc|sgSd|D}t|}t|}||kr d|DSg}|D]A}|j|z ||z z }|t |j||jB|S)z Normalize surprisal scores to [0, 1] range using min-max normalization. Args: scores: List of SurprisalScore objects with raw surprisal values Returns: List of SurprisalScore objects with normalized surprisal values cg|] }|j Srr:r6s rr/z%_normalize_scores..s444 444rcFg|]}t|jd|jS)g?r)rr rr6s rr/z%_normalize_scores..sD    MSAK      rr)rWmaxr"rrr r)rdsurprisal_values min_surprisal max_surprisal normalizedrjnormalized_values rrUrUs  54V444())M())M %%       (*J   !Om; M )   !-*/        rc|sgStjjj}t dt t ||z}|d|S)a Filter observations by top percentage. Assumes scores are already sorted by surprisal (highest first). Args: scores: List of SurprisalScore objects, sorted by surprisal DESC Returns: Filtered list of SurprisalScore objects (top N% by surprisal) rBN)r rMrNrXrr}rJ)rd top_percentcounts rr[r[sP  .*@K 3s6{{[011 2 2E &5&>r)r$rr%rr&rr'r() r*rr$rr%rr&rr'rn)r*rr$rr%rr&rrqr}rrr~r'rn)r_rr'r#)rbr#r'r )r_rrbr#rcr r'r()rdr(r'r()'r __future__rlogging dataclassesrnumpyr3 sqlalchemyrrsqlalchemy.ext.asynciorsrcr src.configr src.crud.documentr src.dependenciesr src.dreamer.treesr r getLoggerrrKrrrmrIrxryrzrPrSrTrUr[rrrrs+#"""""!!!!!!########////////////''''''88888888  8 $ $ ((((((( (  ~~~~B: : : : z!(!(!(!(H%(%(%(%(P'('('('(T&0@))))Xr