jNdZddlZddlZddlZddlmZddlmZddl Z ddl m Z ddl mZ ddlmZd efd Zej&d k\r dd lmZd ZneZ ddlmZddZdZGdde j<ZGddZ GddZ!GddeZ"y#e$r ddlmZYqwxYw#e$r* ddlmZn#e$re j6deZYnwxYwY{wxYw)a caching.py ----------- Functions and classes that help with tracking changes in `numpy.ndarray` and clearing cached values based on those changes. You should really `pip install xxhash`: ``` In [23]: %timeit int(blake2b(d).hexdigest(), 16) 102 us +/- 684 ns per loop In [24]: %timeit int(sha256(d).hexdigest(), 16) 142 us +/- 3.73 us In [25]: %timeit xxh3_64_intdigest(d) 3.37 us +/- 116 ns per loop ``` N)wraps)sha256)log) is_sequence)MappingreturncHtt|jdS)N)int_sha256 hexdigestitems .get_cachedys Aw    4;;$$ $;;$$T* *$)&) KK ' 'w'EKK 1$$)EKK !"' $ r)rproperty)r8r9s` rcache_decoratorr;gs)$ 8_B J rcFeZdZdZdZddZedZejdZdZ fdZ fdZ fd Z fd Z fd Zfd Zfd ZfdZfdZfdZfdZfdZfdZfdZfdZfdZfdZfdZfdZfdZfdZfdZfdZ xZ!S)r#aE Subclass of numpy.ndarray that provides hash methods to track changes. General method is to aggressively set 'modified' flags on operations which might (but don't necessarily) alter the array, ideally we sometimes compute hashes when we don't need to, but we don't return wrong hashes ever. We store boolean modified flag for each hash type to make checks fast even for queries of different hashes. Methods ---------- __hash__ : int Runs the fastest available hash in this order: `xxh3_64, xxh_64, blake2b, sha256` cLd|_t|t|rd|_yy)z Sets a modified flag on every TrackedArray This flag will be set on every change as well as during copies and certain types of slicing. TN) _dirty_hash isinstancetype)r5objs r__array_finalize__zTrackedArray.__array_finalize__s&  c4: &"CO 'rcn|jr%tjj|||g|i|S|dS)zn Return a numpy scalar if array is 0d. See https://github.com/numpy/numpy/issues/5819 )ndimr ndarray__array_wrap__)r5out_arrcontextr3r4s rrGzTrackedArray.__array_wrap__s8 <<::,,T7GUdUfU Ur{rc |jdS)N WRITEABLE)r$r5s rmutablezTrackedArray.mutableszz+&&rc&||j_yN)r$r2)r5r7s rrMzTrackedArray.mutables$ rc|jst|dr |jSt|j d}||_d|_|S)z Return a fast hash of the contents of the array. Returns ------------- hash : long int A hash of the array contents. _hashedC)orderF)r>r/rQ hash_fasttobytes)r5hasheds r__hash__zTrackedArray.__hash__sMGD)$<<< 4<super __class____iadd__r5r3r4rZs rr[zTrackedArray.__iadd__s( T^^T3TDVDDrcDd|_t|j||i|SNT)r>rYrZ__isub__r\s rr_zTrackedArray.__isub__&T^^T3TDVDDrcDd|_t|j||i|Sr^)r>rYrZfillr\s rrbzTrackedArray.fill&T^^T/@@@rcDd|_t|j||i|Sr^)r>rYrZ partitionr\s rrezTrackedArray.partitions&T^^T4dEfEErcDd|_t|j||i|Sr^)r>rYrZputr\s rrgzTrackedArray.puts&T^^T.???rcDd|_t|j||i|Sr^)r>rYrZbyteswapr\s rrizTrackedArray.byteswapr`rcDd|_t|j||i|Sr^)r>rYrZitemsetr\s rrkzTrackedArray.itemset&T^^T2DCFCCrcDd|_t|j||i|Sr^)r>rYrZsortr\s rrnzTrackedArray.sort rcrcDd|_t|j||i|Sr^)r>rYrZsetflagsr\s rrpzTrackedArray.setflags r`rcDd|_t|j||i|Sr^)r>rYrZ__imul__r\s rrrzTrackedArray.__imul__r`rcDd|_t|j||i|Sr^)r>rYrZ__idiv__r\s rrtzTrackedArray.__idiv__r`rcDd|_t|j||i|Sr^)r>rYrZ __itruediv__r\s rrvzTrackedArray.__itruediv__&T^^T7HHHrcDd|_t|j||i|Sr^)r>rYrZ __imatmul__r\s rryzTrackedArray.__imatmul__&T^^T6GGGrcDd|_t|j||i|Sr^)r>rYrZ__ipow__r\s rr|zTrackedArray.__ipow__!r`rcDd|_t|j||i|Sr^)r>rYrZ__imod__r\s rr~zTrackedArray.__imod__%r`rcDd|_t|j||i|Sr^)r>rYrZ __ifloordiv__r\s rrzTrackedArray.__ifloordiv__)s&T^^T8$I&IIrcDd|_t|j||i|Sr^)r>rYrZ __ilshift__r\s rrzTrackedArray.__ilshift__-rzrcDd|_t|j||i|Sr^)r>rYrZ __irshift__r\s rrzTrackedArray.__irshift__1rzrcDd|_t|j||i|Sr^)r>rYrZ__iand__r\s rrzTrackedArray.__iand__5r`rcDd|_t|j||i|Sr^)r>rYrZ__ixor__r\s rrzTrackedArray.__ixor__9r`rcDd|_t|j||i|Sr^)r>rYrZ__ior__r\s rrzTrackedArray.__ior__=rlrcDd|_t|j||i|Sr^)r>rYrZ __setitem__r\s rrzTrackedArray.__setitem__ArzrcDd|_t|j||i|Sr^)r>rYrZ __setslice__r\s rrzTrackedArray.__setslice__ErwrrO)"r* __module__ __qualname____doc__rBrGr:rMsetterrWr[r_rbrergrirkrnrprrrtrvryr|r~rrrrrrrr __classcell__)rZs@rr#r#s& #'' ^^%%. EEAF@EDAEEEIHEEJHHEEDHIIrr#c\eZdZdZddZdZdZddZdZdZ d Z d Z d Z d Z d ZdZy)Cachezd Class to cache values which will be stored until the result of an ID function changes. c\||_t||_d|_d|_i|_y)z Create a cache object. Parameters ------------ id_function : function Returns hashable value force_immutable : bool If set will make all numpy arrays read-only Nr) _id_functionboolr. id_current_lockr-)r5 id_functionr.s r__init__zCache.__init__Ps.(#O4  rcZ||jvr|jj|dyy)z. Remove a key from the cache. N)r-popr5keys rdeletez Cache.deletees' $**  JJNN3 % rc X|jdk7ry|j}||jk7r{t|jdkDrTt j dt|jtt|jji|_||_yy)z Verify that the cached values are still for the same value of id_function and delete all stored items if the value of id_function has changed. rNz%d items cleared from cache: %s) rrrr0r-rdebugstrlistkeys)r5id_news rr,z Cache.verifyls ::? ""$ T__ $4::" 5 OTZZ__./0DJ$DO %rNc|i|_y|jjDcic] \}}||vs ||c}}|_ycc}}w)z Remove elements in the cache. Parameters ----------- exclude : list List of keys in cache to not clear. N)r-items)r5excludekvs rclearz Cache.clearsF ?DJ+/::+;+;+=N+=41ag!Q$+=NDJNs AAc|jj||jrV|jjD]9}t |dst |j dkDs)d|j_;|jy)zg Update the cache with a set of key, value pairs without checking id_function. r$rFN) r-updater.valuesr/r0r1r$r2id_set)r5rrs rrz Cache.updatesf %   ZZ&&(1g&3qww|j||jvSrOrrs r __contains__zCache.__contains__s djj  rcL|jt|jSrO)r,r0r-rLs r__len__z Cache.__len__s 4::rc.|xjdz c_yNr)rrLs r __enter__zCache.__enter__s a rcX|xjdzc_|j|_yr)rrr)r5r3s r__exit__zCache.__exit__s a ++-r)FrO)r*rrrrrr,rrrrrrrrrrDrrrrJsD *&%8 O . (*!.rrceZdZdZddZdZy) DiskCachez Store results of expensive operations on disk with an option to expire the results. This is used to cache the multi-gigabyte test corpuses in `tests/corpus.py` c||_tjjtjj ||_tj |jdy)a Create a cache on disk for storing expensive results. Parameters -------------- path : str A writeable location on the current file path. expire_days : int or float How old should results be considered expired. T)exist_okN) expire_daysospathabspath expandusermakedirs)r5rrs rrzDiskCache.__init__sA'GGOOBGG$6$6t$<=  DII-rc~t|jdj}tjj |j|}tjj |rltjtj|jz dz }||jkr&t|d5}|jcdddStjd|d|}t|d5}|j|ddd|S#1swYQxYw#1swY|SxYw)a Get a key from the cache or run a calculation. Parameters ----------- key : str Key to reference item with fetch : function If key isn't stored and recent run this function and store its result on disk. zutf-8g@rbNznot in cache fetching: ``wb)r encoderrrjoinisfiletimestatst_mtimeropenreadrrwrite)r5rfetchkey_hashrage_daysfraws rgetz DiskCache.gets3::g./99;ww||DIIx0 77>>$  bggdm&<&<<GH$***$%668&% ,SE34g $  GGCL &% sD& D2&D/2D<N))r*rrrrrrDrrrrs.&%rrceZdZdZdZdZdZdZedZ e jdZ dZ d Z d Z d Zd Zd ZdZdZy) DataStorez A class to store multiple numpy arrays and track them all for changes. Operates like a dict that only stores numpy.ndarray ci|_yrOdatarLs rrzDataStore.__init__-s  rc,t|jSrO)iterrrLs r__iter__zDataStore.__iter__0sDIIrc:|jj|dSrOrrrs rrz DataStore.pop3syy}}S$''rc<|jj|dyrOrrs r __delitem__zDataStore.__delitem__6s c4 rct|ddS)z Is data allowed to be altered or not. Returns ----------- is_mutable : bool Can data be altered in the DataStore _mutableT)getattrrLs rrMzDataStore.mutable9stZ..rct|}|jjD]}t|ts||_||_y)z Is data allowed to be altered or not. Parameters ------------ is_mutable : bool Should data be allowed to be altered N)rrrr?r#rMr)r5r7 is_mutablers rrMzDataStore.mutableEs>%[ !!#A!\*! $# rct|jdk(ry|jjD]?}t|rt|dk(ryyt t j |s?yy)z Is the current DataStore empty or not. Returns ---------- empty : bool False if there are items in the DataStore rTF)r0rrrrr isrealr5rs ris_emptyzDataStore.is_emptyXs^ tyy>Q !!#A1~q6Q; biil#$rci|_y)z5 Remove all data from the DataStore. NrrLs rrzDataStore.clearms  rc |j|SrOrrs rrzDataStore.__getitem__ssyy~rc |js tdt|tr|}nDt|tj t ttfr t|}n t||}t|dr|j|_||j|<y#t$rtd|dt|dwxYw)a Store an item in the DataStore. Parameters ------------- key A hashable key to store under data Usually a numpy array which will be subclassed but anything hashable should be able to be stored. z"DataStore is configured immutable!z unhashable `:rrN)rM ValueErrorr?r#r rFrsettupler'hash BaseExceptionr@r/r)r5rrr&s rrzDataStore.__setitem__vs||AB B dL )G rzz4e< =#D)G ET G 4 $"llGO  #! E <uAd4j\!CDD Es " B%Cc||jvSrOrrs rrzDataStore.__contains__sdiirc,t|jSrO)r0rrLs rrzDataStore.__len__s499~rctt|ts td|jD] \}}|||< y)Nz!Update only implemented for dicts)r?dictrr)r5rrr7s rrzDataStore.updates5&$'@A A ,,.JCDI)rc ttj|jj Dcgc])}|%t |drt |dkDr t|+c}tjjScc}w)z Get a hash reflecting everything in the DataStore. Returns ---------- hash : str hash of data in hexadecimal rrr) rTr r%rrr/r0rint64rUrs rrWzDataStore.__hash__s{ HH"YY--//}ga.Cs1vPQzG/ hh  gi  s.B N)r*rrrrrrrr:rMrrrrrrrrrWrDrrrr%sp(! / / ^^##$* #!J   rrrO)#rrsysr functoolsrhashlibrr numpyr constantsrutilrcollections.abcrrr version_inforrrxxhashrrTrrr'r;rFr#rrrrDrrr s, %(' .C.v+JM"5<4 niI2::iIXU.U.p@@FV V C('(. " "7 "  , " " "sGBB' B$#B$'C-B43C4C CCCC