yjddlZddlmZddlmZddlmZmZmZej dZ de e e zdzde fdZd ed eee fddfd ZdS) N)settings)prometheus_metrics)DeriverComponentsDeriverTaskTypes TokenTypes o200k_basetextreturnc|sdSt|trd|} tt|S#t $rt|dzcYSwxYw)z@Estimate token count using tiktoken for text or list of strings.r ) isinstancelistjoinlen tokenizerencode Exception)r s 8/DATA/AppData/hermes/projects/honcho/src/utils/tokens.pyestimate_tokensrs q$yy9##D))*** 4yyA~s&AA65A6 task_type componentsc|D]G\}}tjjr1t j||jtjj|jHdS)z Helper method to track input token components for a given task type. Args: task_type: The type of task components: Dict mapping component names to token counts )countr token_type componentN) itemsrMETRICSENABLEDrrecord_deriver_tokensvaluerINPUT)rrr token_counts rtrack_deriver_input_tokensr$sm#-"2"2"4"4 ;   #   4!#/%+1#/     )tiktoken src.configr src.telemetryr src.telemetry.prometheus.metricsrrr get_encodingrstrrintrdictr$r%rr/s,,,,,, "H !, / /  #S /D0 S    &+, r%