Transcript IJCNN05 Montreal Workshop NNs, Bio- and Neuro

IJCNN05 Montreal Workshop,
19:00-22:00 Thursday 04Aug05
NNs, Bio- and Neuro- Informatics
Junk DNA and Neural Networks:
conjecture on directions and implications
Mr. Bill Howell [email protected]
(use Menu selection “View->Notes” to see the notes which accompany many of the
slides in this MS Powerpoint presentation)
The views expressed in this presentation are
personal and speculative. They are in no way
related to the research, policies, viewpoints, and
programs of my current employer, the federal
government department “Natural Resources
Canada”.
To the best of my knowledge, there is no work
underway or planned in this area within the federal
government at this time.
Bill Howell
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Outline
1.Revolution beyond the "central dogma of biology"
2.
–
DNA: function beyond gene-protein through “junk”
–
Junk = non-protein coding, incl non-exon, epigenetics, other?
Related trends with NN architectures and processes
•
•
3.
Computational Neuro-Genetic Modelling (CNGM)
1.
2.
4.
Structure, function, messaging, dynamic transitions
Learning, control, planning, behaviour, goals
Bio- and Neuro- informatics relevance
Expanded approach to ANNs
Implications for the brain and the mind
●
1.
2.
Inheritance of vast knowledge, grammar (not just linguistic!)
Multiple parallel “behaviours & personalities” (computing)
Evolutionary theory
1. Revolution beyond the
"central dogma of biology"
2. Related NN trends with architectures and processes
3. Computational Neuro-Genetic Modelling
4. Implications for the brain and the mind
“The gold standard for NNs, far off in the distance, IS the human brain...”
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Junk DNA as code
A revolution in the "central dogma of biology"?
●
–
John Mattick (UQueensland), others over several years
Eukaryotic DNA coding >>> genes for proteins
●
~1.5% of human DNA codes for proteins, but most DNA
transcribed to RNA!
● poor relation between organism's complexity & # of proteincoding genes, more consistent relation with non-coding
DNA?
● DNA expression
•
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Junk DNA as code (cont'd)
–
Genes:
–
Junk -> RNA and micro-RNA, regulatory role
conventionally thought as literally "assembly language
programming" for proteins – perhaps the simplest and lowest level
of programming on the DNA?
●
●
–
Chromatin – mask/reveals DNA code
Architecture – need precise plans, “drawings” - highly
specific
Cambrian period bio-complexity explosion (~1 Gy ago)
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
John Mattick: “Cambrian complexity explosion”
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Possible Examples?
●
•
•
•
Timing circuits in cells/ organism (special emphasis on time,
sequencing, coordination of parallel and sequential
processes)
– multitemporal, multifractals
Cell-line specific genes/ structures/ functions – cell signaling
mRNA -> [datamining, eg constellations]
High cross-over rate genes (individual variabilities such as
appearances)
Linguistics – Chomsky, Pinker (1994), note that the distinction
between genes and "non-gene" DNA is not emphasized by many
authors
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Neuron to DNA signal & control?
●
Feedforward - basis of current descriptions of npcRNA
●
Feedback from neuron to DNA
–
–
–
A given for regulatory control, but is this limited to
“normal” physiological sensing/ control pathways?
Is there a means for the neurons to direct SPECIFIC
DNA expression (protein or non-protein coding
DNA), in a way that isn’t simply physiology, but is
directly for specific information processing/ data
tasks?
Could lead to Recurrent Neural Networks (RNNs)
that use DNA coding/ “programs” (programming
metaphor)- with very powerful advantages!
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Advantages of Junk DNA as Code
●
Mattick:
–
–
–
●
away from the combinatorial soup of proteins for regulation,
towards a highly specific "program" to direct spliceosomes
(low side effects, fast, efficient)
specification of architecture/ growth allows vastly greater
complexity of organisms (just what we are looking for with the
brain!)
alternative splicing & "overloading" of genes – assemble
protein-coding-RNA in one of several ways, code has different
functionality in different cell types
Other possible advantages - metaphor of computing
– Vastly parallel (as fits NNs)
– Not just “static” code – dynamic interactions between coding
– ??Co-resident junk & protein code (data/ methods -> superobjects), “calls” to ensembles of NNs
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Challenges – extrapolating to the brain
●
Genes – "relatively easy" to see, identify coding
it's still a work in progress, start/ stop sequences, intron removal
etc, how to decide when two conformations are possible?
●
Cell physiology -> "event" amoung many others happening
in parallel, still can "see" amongst all of the mRNA & cell
signalling happening at the time
●
●
Ontogeny (growth) -
visibility like cell physiology, but how
easy is it to quantify subtle structural changes?
But what about more abstract processes, and
thought? How can one identify these?
how to link code & its effect?
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
What toolsets do geneticists have & need?
●
Already a great base & fallback position – current
genomics/ bioinformatics/ neuroinformatics
●
Need - greater exactness of:
–
–
–
●
●
[Reading, precisely changing] DNA code, RNA & epigenetics
Measuring structural changes in neurons & NN ensembles
Also, what are “feedback” mechanisms from NN changes to
DNA?
Signaling – how good is this for the purposes in
mind?
Computational Intelligence tools?
(third section)
1. Revolution beyond the “central dogma of biology"
2. Related trends with NN architectures
and processes
3. Computational Neuro-Genetic Modelling
4. Implications for the brain and the mind
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
NN Architecture & Multi-phase modules:
"Crystalline to Gaseous"
●
Fixed weight neural networks (IJCNN05 Guang-bin
Huang extreme learning machines)
●
Echo state networks (IJCNN05 Special Session Rao &
Principe)
●
Some aspects of:
–
Neural Gas Models (other name of similar NNs?)
–
Chaotic NNs
(IJCNN05 SS - Kozma)
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Functional overloading
Functional overloading – similar to neuro-modulators and
gene networks
●
●
Same NN module is simultaneously part of many
different “models” and model trainings
Variable object inputs/ problem types
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Networks, hierarchies of NNs
● Networks, hierarchies of NN ensembles (Mexican?,
ART, SOMs)
●
Adaptive Critics and ObjectNets (Werbos,
Venayagamoorthy) and agent-like systems
● What is happening to Neurosolutions software?
Are there an emergent systems for NN ensembles? (Principe)
● Neurology - Purkinje cells, climbing fibres etc etc
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Dynamic transitions
● Dynamic transitions - during learning and action
● Searching problem/ solution statespace
–
–
●
evolutionary theory, particle swarms
chaos - ?directed?, stochastic
Variable object inputs/ problem types - as with
functional overloading
●
Dynamically reconfiguring structures of NN modules
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Learning &Training implications
Previous issues tie in well to Junk DNA/CNGM
●
●
●
●
●
manner of selecting pre-specified, powerful starting points
for brain/ systems of ANNs (eg Huang and ?de Jaeger?)
capability of building robust, effective systems of ANNs, and
hybrid systems involving ANNs
don't need a small set of tools, not restricted to a single
learning theory/ method
looking for orders of magnitude faster training plus much
more accurate/ more robust solutions where that is possible,
with powerful general learning techniques for unusual/
difficult problems
for some problems, perhaps we cannot expect fast solutions –
evolution of capabilities over much time or many generations
may be required
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Learning &Training
● Meta-learning
–
–
Kenji Doja (2002)
learning parameter initialization/ adjustment
(cont'd)
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
"Small-world” universal function
approximation
●
Small world domain – what is the smallest set of NNs, of
various functional capabilities (general to specific), that is
sufficient to solve most of the problems in a domain of interest?
●
How does one restructure/ combine/ train these NN
modules to obtain ultra-fast, accurate learning/ control? (eg for
control – ObjectNet adaptive critics)
●
●
●
What happens for a much broader or universal
domain? (combine both general & special NN)
Building from level of abstraction to the next
Converse issue – false confidence in good fits (eg Global
Circulation Models for climate, Valdes @ IJCNN05)
1. Revolution beyond the “central dogma of biology"
2. Related NN trends with architectures and processes
3. Computational Neuro-Genetic
Modelling
4. Implications for the brain and the mind
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Expanded approach to NNs
●
Nik Kasabov, Benuskova, Wysoski (Aukland
UTech) - architectures for gene networks, extending the
concept to advance NN architectures
●
What are the desired capabilities/ opportunities?
–
–
–
Co-resident [code, genes] - beyond OOPS?, new
RNNs?
Data [delivery mechanism] - code segments identify
data (DNA or RNA code keys physically bring data
and destination together!), data can drive code &
architecture…
Dynamic structures – switch/evolve in real time!
Approximate Dynamic Programming (ADP)
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Expanded NNs (cont'd)
●
Hypothesis of bidirectional action:
–
–
●
Junk DNA -> drives neuron states
Neuron states -> initiate junk DNA sequences
Different object inputs to, and functional behaviour
of, a single NN module or architecture of NN modules
●
Rapid reconfiguration of ensembles of NN modules
along “high likelihood” arrangements, longer term more
exhaustive evolution
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
What differentiates
CNGMs from current NNs?
●
Will they simply be faster/ more accurate?
–
●
Like fixed weight NN - much faster system
identification/ learning
Or are there new capabilities that will arise?
–
Greater “aptitude” for symbolism?
●
–
–
●
Semantics and Logic as emergent properties for very
complex systems (abstraction -> soft logic, giving
explainability)
Greater functional/ mapping specificity
Ease of combining networks of NN modules?
What else? (my feeling: HUGE conceptual advances)
1. Revolution beyond the “central dogma of biology"
2. Related NN trends with architectures and processes
3. Computational Neuro-Genetic Modelling
4. Implications for the brain and the
mind
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Implications - for the brain and the mind
●
Inheritance of vast knowledge (content) - form
very specific to general and to highly abstract
–
Data, procedures, “operating systems”, behaviours …
–
Selection of the right blends of different levels of
abstraction for inheritance
–
Eg language - could “know” all words, but it is the
dynamic bindings that give language its power
–
Far beyond “Nature versus Nurture” (which was
somewhat a ?dysfunctional? discussion anyways - eg ontogeny
effects on identical twins, compression effects in relating
synapses to inherited DNA). The problem with dichotomies…
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Implications (cont’d)
●
Inheritance of a vast grammar
–
●
Multiple parallel behaviours & personalities
–
●
As per Chomsky, but not just linguistics - much more
fundamental than that, at the connectionist levels as well as at
the symbolic/ linguistic levels
But will this somehow affect goal identification/ prioritization?
Problem decomposition/ modularisation,
reconstruction
–
Multiple models that are dissimilar, yet collaborate effectively
and simultaneously on a variety of models of the problem(s)
being addressed
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Evolutionary Theory
●
Lamarckian heredity - pass on capabilities (strong arms,
knowledge) developed over course of lifetime
–
●
Easily done for individuals, organisations, society,
computers -> learning, charters & constitution, laws etc etc
Would a Lamarckian evolutionary approach “rediscover” basic principles and laws that have long
been established in other areas?
–
learning (pass on through teaching, apprenticeship,
experience, management fashions)
–
Economics, organization structure and processes?? etc
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
Mindcode
●
"Given that computer code is used to program
computers, then mindcode..."
–
●
The perspective here isn't to "program" a child/adult brain by
some external means, but rather to seek an understanding of
junk DNA coding (and perhaps other sources of coding such as
epigenetics) that may define the basis of our brains from
conception. What might such code tell us about ourselves and
our history that is different from current psychology, sociology,
anthropology, management theory, economics?
This is pure speculation and fantasy, but I think that it
is a useful fantasy to drive lines of investigation, to maintain an
awareness of the types of results that we should be looking for (its
easier to find something when you are looking for it).
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
References
1. Junk DNA, epigenetics, neuromodulation
Ast, Gil “The alternative genome” Scientific American, vol 292 iss 4, Apr05, pp58-65
· Doja, Kenji; Dayan, Peter; Hasselmo, Michael (guest editors) "2002 Special Issue: Computational
Models of Neuromodulation" Neural Networks, Vol 15 Nos 4-6, June-July 2002: Kenji Doya
"Metalearning and neuromodulation" pp495-506
· Eddy, Sean R. “Non-coding RNA genes and the modern RNA world” Nature Reviews Genetics,
vol 2, pp 919-929, December 2001(from Gibbs)
Gibbs, W. Wayte "The unseen genome: Gems among the junk" Scientific American, vol 289 no 5,
Nov03 pp46-53
Ingoglia, N. organizer (American Society of Neurochemistry (ASN)) “RNA Interference (small
RNAs): Applications to neural systems” Pre-meeting workshop, ASN Annual Meeting, 14-18Aug04,
New York city
· Krichevsky, Anna (Harvard Medical School) - research focussed on neurons: Kim, J;
Krichevsky A; Grad, Y: Hayes, G.D; Kosik, K.s; Church, G.M; Ruvkun, G “Identification of many
microRNAs that copurify with polyribosomes in mamalian neurons” PNAS vol 101 no 1, 06Jan04,
pp360-365
Mattick, John S. “Challenging the dogma: the hidden layer of non-protein-coding RNAs in
complex organisms” BioEssays, vol 25 pp930-939, Oct03
· Mattick, John S. (UQueensland) "The hidden genetic program of complex organisms" Scientific
American, Oct04 pp60-67. See http://imbuq.edu.au/groups/mattick
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
References
2. Related trends with NN architectures and processes
Guang-Bin Huang, Qin-Yu Zhiu, Chee-Kheong Siew, Nanyang TechU, Singapore
"Extreme learning machine: A new learning scheme of feedforward neural networks"
IJCNN04 Budapest
Ganesh Kumar Venayagamoorthy (UMissouri-Rolla) "Dynamic optimization of a
multimachine power system with a FACTS device using identification and control
ObjectNets" (IAS2004 Yikes – I just have the paper and forget which conference!) 07803-8487-3/04/$20.00 © 2004 IEEE
Roberto.A. Santiago, NW Computational Intelligence Lab, Portland StateU, Oregon
"Context discerning multifunction networks: reformulating fixed weight neural
networks" IJCNN04 Budapest
?"Co-Evolutionary Learning of Liquid Architectures" Igal Raichelgauz, Karina
Odinaev Yehoshua Y. Zeevi, Israel unpublished yet?
a)
b)
c)
d)
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
References
3. Computational Neuro-Genetic Modelling
a) N. Kasabov, L. Benuskova, S.G. Wysoski (Knowledge Engng & Discovery Research Institute
(Kedri), Aukland UofTech, New Zealand) "Computational neurogenetic modelling: Gene
networks within neural networks" IJCNN04 Budapest
b) Rui Xu and Donald C. Wunsch v- see IJCNN05 references
International Joint Conference on Neural Networks 2005, Montreal
NNs and Bio-/Neuro- Informatics
IJCNN05 References
IJCNN05 = International Joint Conference on Neural Networks 2005, Montreal, International
Neural Netwsork Society and IEEE Computational Intelligence Society. Because of the ease of
accessing conference papers, many are listed below. This is certainly NOT exhaustive!!
a) Relevant sessions include:
– S2 "Computational Neuro-Genetic Modelling" chair Nik Kasabov
– S8 "Constructive/Hierarchical Self-Organizing Maps" chairs Ernesto Cuadros-Vargas and
Roseli Francelin Romero
– Sa "Computational Dynamical Modeling with Echo State Networks" chairs Yadunandana Rao
and Jose Principe
– Sf "Evolvable and Emergent Neural Systems" chairs Seong Kong and Jacek Zurada
– P1-Gf "Neural network architectures and structures" chairs: IJCNN05 program chairs
b) Related papers include:
1. 1728 "Gene Regulatory Networks Inference with Recurrent Neural Network Models" Rui Xu
and Donald C. Wunsch II, ACIL, University of Missouri-Rolla, United States
2. 1016 "Functional Grouping of Genes Using Spectral Clustering and Gene Ontology" Nora
Speer, Holger Froehlich, Christian Spieth and Andreas Zell, Centre for Bioinformatics
Tuebingen (ZBIT), Germany
3. 1603 "Modeling Cortico-Subcortical Interactions During Planning, Learning, and Voluntary
Control of Actions" Daniel Bullock, Boston University, United States
4. 1257 "Protein Sequence Classification Using Extreme Learning Machine" Dianhui Wang, La
Trobe University, Australia; and Guang-Bin Huang, Nanyang TechUniversity, Singapore