Balancing Long-Term Reinforcement and Short-Term Inhibition
Christian Lebiere (cl@cmu.edu) Carnegie Mellon University Bradley Best (bjbest@adcogsys.com) Adaptive Cognitive Systems
Outline
• Rational analysis of memory
• Computational implementation
• Long-term impact of short-term effects • Revisiting the environment
• Combining reinforcement and inhibition • Internal dynamics mirror environment
8/1/09 Cognitive Science Conference 2009 2
Rational Analysis of Memory
• Humanmemoryhasadaptedthroughevolutionto the structure of its environment (Anderson & Schooler, 1991)
– Frequency effects – Recency effects
– Spacing effects
• Evenapparentfailuresserveafunctionalrole
– Forgetting as a way of managing scale of memory demands
• Givenresourceconstraintsonlong-termmemory, optimal behavior is making most available the memories that are most likely to be needed
8/1/09 Cognitive Science Conference 2009 3
Power Law of Memory Retention and practice functions for range of measures
8/1/09
Cognitive Science Conference 2009 4
P = AT−b
logP =logA−blogT
Power Law of the Environment
Need Odds follow power law in human environments: NYT headlines, CHILDES speech db, email addresses
8/1/09 Cognitive Science Conference 2009 5
Computational Implementation • DelayandPracticeareroughlyadditiveeffects
• Activationaslogoddsmodulatesrecallandlatency
P r o b ∑n
8/1/09 Cognitive Science Conference 2009 6
1−Prob=Odds=a tk−d 1
∑n Ai =B+ln tk
−d 1
In The Short Run
• “Apowerfunctionimpliesthattheperformance measure will go to infinity as time goes to zero.” p.398
• “Powerfunctionsforforgettingtendtobeobtained when we use measures that do not have upper bounds or do not approach their upper bounds.” p.398
• “Powerfunctionsseemtodescribememory performance from a few seconds to years.” p.398
• Whatarethelong-termimplicationsifperformance measures grow arbitrarily large under a few seconds?
8/1/09 Cognitive Science Conference 2009 7
Winner-Take-All Dynamics
• Reinforcementleads to feedback loop
• Strongconditionson retrieval can help…
• Buttheydetractfrom the use of activation
• Needpartialmatchingandunder-constrainedretrievals
• Canbecontrolledbymodelerwithtagsorfinsts
• Metacognitiveknowledgehardtospecifyathigh-level
• Needrobust,generalmodelsofopen-endedbehavior
8/1/09 Cognitive Science Conference 2009 8
Back to the Environment
• Languageaspreeminentsequentialstructuredenvt • Short-termdepressioninneedoddsatsmalllags
• Consistentacrossverydifferenttextcorpuses
8/1/09 Cognitive Science Conference 2009 9
Words and Groups
• Thepatternholdsforhighestfrequencywords
• Additiveeffectisobservedatalllevelsoffrequency
• Peakinoddsboostconsistent~lag5-10forallgroups
8/1/09 Cognitive Science Conference 2009 10
Access to Arithmetic Facts
• Asinoriginalanalysis,effectmustholdacrossdomains • Arithmeticdomaingeneratedfromvalidatedmodel
• Patternholdsincludingpeakandsizeofinhibition
• Applytootherenvironments,e.g.web,physicalspaces
8/1/09 Cognitive Science Conference 2009 11
Activation Inhibition
n −ds ∑−d t
B=log t −log1+ n ij j=1 ts
• Parameters:
– Inhibition scale ts – controls
period to peak reinforcement
– Inhibition decay td – specifies magnitude of inhibition effect
• Additive term integrates with other terms of activation equation: noise, spreading association, partial matching
8/1/09 Cognitive Science Conference 2009 12
Emergent Robustness
• Soft inhibition differs from pathological behavior of the default version and from the hard and fixed round-robin of the finst version
• Running the retrieval mechanism unsupervised leads to the gradual emergence of an internal power law distribution internally
8/1/09 Cognitive Science Conference 2009 13
Discussion
• Biologicalimplementationofshort-terminhibition
– Short-term depotentiation?
• Architecturalimplicationsofshort-terminhibition
– Working memory and refraction
• Contributionofothercognitivefactorsintask
– Grammatical rules, base-10 systems
– Could have evolved in response to cognitive limitations
• Integratecombinationofenvironmental,neuraland behavioral constraints in cognitive architectures
8/1/09 Cognitive Science Conference 2009 14