I'd taken a break from all the AI business for a few weeks. I'm surprised at how much I got done working with a clear head. I've run some initial tests through my concept for the Biological Based but not Weights Based Neural Network. The results look unbelievably amazing. The program starts out with absolutely no info and then learns everything quickly and efficiently. Here's what I've added to the log (read Concept AI part 1 and 2 first).
******
Miller's Law: A person can only keep 7 plus or minus 2 items in mind at one time.
http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Introduction%20to%20neural%20networks
Firing rules to figure out weather a synapse fires based upon the input.
http://www.spiegel.de/international/spiegel/0,1518,466789,00.html
The nerve cells open or close it's ion channels before it fires.
*Going back to some more basic research about the fundamentals of the Brain itself.
http://www.enchantedlearning.com/subjects/anatomy/brain/Neuron.shtml
The brain has about 100 billion neurons. Then there are glial cells, which provides support (what kind?). The sizes of the Neurons differ.
Neruron: Cell Body, Dentrites (signal receivers), projections (axon, which conduct the nerve signal). Axon Terminals transmit electro-chemical signals over the gap known as the synapse.
Dentrites bring info to the cell body. The axon takes information away from the cell body. Bundles of them are called nerves or as nerve tracts or pathways. Dendrites branch from the cell body and receive message.
One neuron has about 1000-10,000 synapses, meaning it communicates with that many other neurons.
Glial cells: Nerve cells that don't carry nerve impulses. They basically manufucture stuff for the neurons.
Basically:
Dendrites: Input Connections (one or more)
Axon: Output Connection (only one). So it's like a linked list where the last elements connects to the first element. One axon can speak to multiple dentrites on the reveiver.
Axon Terminals: (more than one)
Synapse: Space between input & output connection
Hebbs Rule: Cells that fire together, wire together.
A synapse's strength depends on the number of Ion-Channels it has.
9-26-08
6:47
It works! I went through the previously non-working AI program and reworked the whole thing. It's version 0.03 now. I'm amazed that it was able to work so efficiently when I put in the charges concept into it. There are still many lil bugs to work out. I've just tested it with some lettter comparisons. The current data set is just 2 sets of 3 letters. It goes like this:
a b
b c
/end
d e
e f
/end
Then I can go in and reset the charges and add another connection linking the two node-network segments toghether into one.
/rnodecharges
c d
Before those statements, a connection to node a would activate node b and c, but after the statements, it activates b, c, d, e and f (with smaller charges as you go futher down the line.)
The main bug now is that the connection between c d is looking too strong. IDK. Maybey it's supposed to be like that. D is emitting a charge similar to that of a.
10:59. After examining it a bit more, I've found and fixed some more bugs. I'm using a new data set now. It goes like this:
Greek is spoken in Greece
/end
Greece is in England
/end
The Greeks created the Olympics
/end
Athens is the capital of Greece
/end
Sparta is the enemy of Greece
/end
Where refers to the places like Greece Athens and Sparta
/end
After clearing the charges, I ask it:
where is Greek spoken
/end
And I got the result:
/end
/charges
greek 1.0
is 1.0
spoken 1.0
in 2.4935020205414458
greece 3.4648806640625
england 1.4565926569239502
the 0.9368278391792345
greeks 0.43610654105691715
created 0.6253771713864278
olympics 0.6719345918743638
athens 1.8224265625
capital 1.4442937950131418
of 1.4180175011313327
sparta 1.6934484375000003
enemy 1.3801804402274132
about 1.2319124453341073
spartans 1.3034834714624521
where 1.0
refers 0.55
to 0.72875
places 0.75366328125
like 0.7785765625000001
and 0.85331640625
This CLEARLY gives you Greece as the answer to the question. It's amazing how quickly it can learn. All of this was achieved with a minimal amount of processing and with NO INFORMATION (except the /end and the other /commands) hard-coded in. Such clear results with just a single repetition!
I think I'm onto something here. I'm now trying to incorporate larger and larger datasets to see how it'll handle it. Oh, by the way: The dataset is completly made up. I have no clue what the capital of Greece is. This is just an example.
9/27/08
11-34 AM
I'm now working with a data set of 40 Nodes creating 221 Connections (Which becomes 185 when sleep is applied with a 0.001 charge). A basic version of the Sleep method has been created to clean up the weak nodes to increase efficiency. I've also put in a Feedback system (+ or - depending on the answer) that strengthens or weakens the node based upon the feedback. It's still looking amazingly good. Will keep updating!
1:47 PM
Grammar is the real trouble here. grammar words like Of, and, or, an, a, the ect. is connected to a whole bunch of different nodes, so it creates some unnecessary charges.
An Idea for improvement.
Greece refers to a place
Where refers to place and Greece, China, England, America, India
Place refers to Greece
So rework the system so
Where refers to place
place refers to India, Greece, china, england, America ect.
This reduces the number of connections. Adding this to the sleep system will allow you to delete the unnecessary (unacceptably weak) connections and the looping connections.
I'm still amazed at it's efficiency though. It's able to comprehend simple inputs like:
My name is Cobalt
Your name is Ron New
What refers to Name
What is My name
?
Cobalt
What is your name
?
Ron New
Ron New is New Ron spelled backwards and with a w instead of a u.
I've now moved it up a notch to read files. Each sentence must be put in a different line, or else it gets REALLY slow.
The current data set is:
184 Nodes with 1678 connections!
The system resources taken up (At the start of the program):
Java.exe PID: 1472 CPU: 00% CPU Time: 00:00:03 Mem Usage: 348 K
This fluctuates between as low as 384 KB to as high as 10,000 KB. System.gc is called a few times in the program so that the memory is used wisely.
The network is working, but it still needs work. I'm going back to more research to apply new concepts into the network to improve it.
http://www.learnartificialneuralnetworks.com/
Action potentials are the electric signals that neurons use to convey information to the brain. All these signals are identical. Therefore, the brain determines what type of information is being received based on the path that the signal took. The brain analyzes the patterns of signals being sent and from that information it can interpret the type of information being received.
It turns out one aspect of my program, the weakening of signals, is not consistent with the biological system. "There are uninsulated parts of the axon. These areas are called Nodes of Ranvier. At these nodes, the signal traveling down the axon is regenerated. This ensures that the signal traveling down the axon travels fast and remains constant (i.e. very short propagation delay and no weakening of the signal)."
Instead of letting one node send pretty much equal amount of strength to it's connected nodes, how about a system where the signal strength is determined by percentage. This would mean that all charges must originate from the initial charges. The output would then be smaller numbers. This might be good or this might be bad. Let's find out!
Nop, that doesn't work. I don't know why, but it doesn't work. I think I was just lucky hitting the target on that first try after the break. Some small, ever so minute changes can sometimes throw the system off.
*********************
Mission Accomplished?? Not exactly. Answering one question (will this work) leads to a thousand new questions. I'm creating a hand-made knowledge base for it to work out off (some basic grammar concepts and general knowledge). The program can handle any information in any language that you teach it! The charges concept has been accomplished (sorta), so now I have to work on some other concepts to make the system independent. It has to take the answer and then figure out how it's going to be applied or presented as an answer. There are a hundreds of applications for the charge-based neural network concept. I assure you, as soon as I'm done with a stable version of the program, it'll be released as an Open Source Program :) I'm planning on releasing it as part of my other project known as OSCEAN (Open Source Content Environment and Abstract Network) that will put the AI to work in a few logical application. But until then, the actual code will have to be kept under wraps. Just for the thrill of suspense when it's finally released!
But now, please reply with some other types of tests that you want me to run on the system and I'll post you the results. Wish me & Ron (NewRon get it?) luck!
Subscribe to:
Post Comments (Atom)
11 comments:
Hey :)
I read about your project via your post on the AI Forums, which I found while looking into projects on AI.
Being a student of linguistics, I have long dreamed about creating, or helping to create, an AI with a working linguistic competence.
I am writing to you here because you said that you know little about grammar - and I know a lot about grammar, but nothing about neural networks or programming in general. My main interest in linguistics is the creation, and testing of, theories of grammar - some of which could possibly assist in providing the framework for a Neural Network based AI analysing and using Natural Language?
If you're interested in working on this with me, or if you have any comment or question, you're welcome to get back to me at ccb at sol dot dk.
/Carsten
Hey carsten!
Yes, I'd be very interested in working with you on the AI project. I'm releasing the program as open source, open for public development. None of the grammar is actually programmed into the AI, it learns it over time. If I keep using bad grammar with it, it'll learn to accept that as how things are. This let's the program be abstract, able to learn any languages.
I was planning on implementing downloadable language packs that let's the AI automatically read, train, and learn from the sample dataset that you give it. If you want to try out some inputs with it, just post it over here and I'll post the results.
I was planning on releasing the project along with another Open Source program as a bundle, but I've been caught up with a lot of stuff lately. The project will be released before the end of the year, even if it's not completed by then. Keep in touch!
Hey Cobalt :)
I assume a neural network learns based on statistic analysis of distribution? As I said I have little knowledge of programming in praxis, and I won't be able to work on the program on my own, I am afraid.
What I envision is some form of implementation of a rather new theory of grammar called Functional Discourse Grammar, by Kees Hengeveld. I am not an expert on it by any means, but I have enough knowledge of the basic concepts to offer some insight into how not only syntax, but also sermantics and pragmatics can be looked at through one unified theory. You should also check out the book if you can find it in a library near you :) It's extremely enlightening, and offers more valuable insights on the actual functions of language than Traditional Grammar and Chomskyan grammar does.
I have also been thinking about how we should train the AI. Some of the research done by Elissa Newport on language acquisition, specifically relating to Gold's proof on the unlearnability of language from data without some sort of restriction on search space, might be worth testing. Basically, we she has suggested that the memory of a NN should be severely limited to start with, and expand at a rate which mimics the expansion of actual cognitive ability as it happens with a child learning language. I believe it has been tried on a limited scale by Morten H Christiansen of Cornell. His papers should be available online, somewhere, but I will see if I can dig some of them out of my library. I'll mail them to you when I find them.
We should also try and teach the AI using actual corpora, as for instance the Bernstein-Ratner corpus, which is also available online. It is a relatively small corpus of actual linguistic input which some children have been exposed to. The benefit of this would be that the AI would learn in a realistic manner. It will complicate the programming somewhat, I assume, but the resulting AI when we get it right would be stronger, I believe.
I am not sure if this is what you have imagined, but I thought it couldn't hurt to try out some of my ideas on you :)
"Some of the research done by Elissa Newport on language acquisition, specifically relating to Gold's proof on the unlearnability of language from data without some sort of restriction on search space"
Yes, I'm planning on adding that in very soon. Right now, it can handle 1000-3000 connections (coming to about 500+ words) in memory with no lag at all. When the program starts, you give it a word. It searches it's hard-drive memory to find that word. Then it loops through 7 turns to find words related to that word and so on. This should add up to less than 500 words. This is the active context memory. As the program is thinking about other things, some more words are added to the memory and some others that are no longer in context are deleted from the memory. So it's always thinking about what it's currently doing/talking about.
Staying as true to a human system is priority one. I like the ideas that you are presenting. I'll look into it and try to come up with some sort of implementation plan. The real goal here is to gain mastery of Language in General, not just English. The theories and ideas incorporated into the program should be abstract enough that it can be applied to multiple languages. I'm fluent in 2 languages and I know just a bit of 2 others. Learning them, I've noticed patterns of association with words and concepts that are already known. So this would mean that beneath the surface, some other very abstract conceptual language is working. By the time the thoughts get to the highest level of our consciousness, we hear it in our native tongue. In the end, the true test is it's ability to adapt to different languages as well as adapt the language to it's needs.
The Neural Network works really well to adapt to new systems. No real statistics are involved either, but I'll incorporate that into the Heuristics Network that will later be combined into this. I'm also planning on adding pattern analysis. Combined, all of this should be able to "understand" concept, communicate using language and solve problems using math (using math that is taught to it, not the internal calculator)...
I'm open to all ideas, but I'm trying to keep the code as uncluttered as I can so that other coders can easily join in once it's released ( I'm working on the website. The forum is up! It's at http://oscean.21.forumer.com/ ) Since this is not the final version of the program, I'll be putting up the source code right away(I'll be doing that in a few weeks when the code is debugged and attached to the main OSCEAN package). You'll have to download java in order to try it out yourself. If not, you can just post up sample input that you want to try and I'll give you the results.
"The real goal here is to gain mastery of Language in General, not just English. The theories and ideas incorporated into the program should be abstract enough that it can be applied to multiple languages."
The beauty of my branch of theoretical grammar is that it does not focus on one language, or on only Indo-European languages, but tries to be typologically adequate. This means that if a language somewhere in the world makes a distinction in the grammar based, it needs to be incorporated in the theory.
This means that the theory of grammar actually functions as a sort of inter-language, making it possible to use the same set of basic abstract principles for all languages. The inter-language is also very formally expressed, making it easy to translate into computational principles. (Or so I assume. Simon C. Dik made an early implementation of the theory in PROLOG in the late 80s).
Of course, this assumes that some sort of lexicon and some sort of 'rule book' for each language is written (or learned from context and conversation), which is what I was envisioning the NN doing. I have no idea if this is possible, but that is precisely the reason I am turning to you :)
For reference, I am now talking about the theory known as 'Functional Grammar' by Simon C. Dik, not the spin-off/extension Functional Discourse Grammar, about which I know very little beyond the basic principles, and which is less readily accesible to linguists and laymen alike.
The basic underlying structure of a sentence in any language is roughly this;
Grammatical operators Predicate frame lexical operators
π4 π3 π2 π1 f (x)(x)n σ1 σ2 σ3 σ4
E X Tense Aspect Predicate Manner Loc X E
with Speed
Arguments
E = speech acts/illocutionary operators such as Declarative, Interrogative, and Imperative
X = proposition of possible fact
The predicate (f) with arguments (x)n plus π and σ operators on level one and two is called a located state of affairs (e)
For instance:
"The duckling was killed by the farmer" would be represented on the underlying level as;
DECL E1: X1: Past e1: kill(verb) (d1x1: farmer(noun)(x1))Agent) (d1xj: duckling (xj))GoalSubject
I realise this may look odd to you, but it does make a lot of sense. The only thing that differs from language to language in this kind of representation is the lexical items 'kill', 'farmer' and 'duckling', and the expression rules which determine the form of the final expression.
A less formal description of the underlying sentence would be;
Declarative E1, such that X1 is a past tense e1, such that the verb kill is the predicate, such that a definite singular entity x1, such that the noun farmer applies to x1, such that x1 fills the semantic role of Agent, such that definite singular entity xj, such that the noun duckling applies to xj, such that the semantic role of Goal and the syntactic role of Subject applies to duckling.
Does this make any sense to you? :/ This is of course just an illustration of how the underlying representation looks, and just a short overview at that, and it took me a while to take in, so don't worry if it looks incomprehensible. I was merely hoping you can say whether or not the.. approach is something which could be incorporated in your program.
Hmm.. the hand-typed diagram doesn't look as intended.. I've uploaded it as a pdf here;
http://www.evilcartyen.net/FG%20ULS.pdf
"Of course, this assumes that some sort of lexicon and some sort of 'rule book' for each language is written (or learned from context and conversation), which is what I was envisioning the NN doing. I have no idea if this is possible, but that is precisely the reason I am turning to you :)"
The Neural Network is currently able to learn completely from context.
"The duckling was killed by the farmer" (Why? I don't know!)
I understand part of what you are trying to say. In the beginning, I tried to program syntax into the system. But that didn't work when I introduced a new language. So I scrapped that and wen't totally abstract. The problem I'm really getting with that is this:
When the AI hears
"The duckling was killed by the farmer"
It already has some knowledge on those words
Duckling - Animal, Noun, Duck, Small (and so on..)
Farmer - Animal, Human, Noun, Smelly (ect...)
Killed - Verb, Noun killed Noun
So using this info, it could come to two main understandings (ignoring the other minor ones..)
A. The duckling was killed by the farmer
B. The farmer was killed by the duckling
It doesn't have the knowledge of how to separate the Subject and the Goal Subject and the Verb from the sentence. Everything, to it, is just a word. Over time though, a pattern arises and the system just works itself out. But that is VERY inefficient.
If I were to program in syntax, then the result might be:
A) The ducking was killed by the farmer
BUT if I use a different syntax
"The farmer killed the duckling"
or, if it were to be written in a different language translated directly to English:
"Duck child by killed farmer"(Tense understood by ending, like in Spanish)
Then it'd start getting confused.
You can download Java and give the source code a try. If you need help with it, I can help out. That'll give you a better understanding of how Neural Network works.
May the Duckling Rest In Peace!
Hmm, just a short note to what you said, as I will try and give the sourcecode a try and not bother you with technicalities ;)
Syntax is not a part of the underlying representation I wrote there - that is an order free formula based on semantics, so to speak, although the syntax rules of English to some degree serve the purpose of illustrating the differences between the poor duckling and the farmer, the actual syntactic representation is not part of it at this stage.
In any case, I'll try and play around with the program, and see what I can arrive at.
Hey carsten. Hope you are reading this! We'll be starting some real active development next week at oscean.forumer.com (Sub section: New Ron). I'll work on having a packaged, ready to run without all the Java Compiling, version of the program ready too (hopefully). I've been trying to contact a whole bunch of people in many different fields to come together and make some real progress in the field of Artificial Intelligence. Sign up! Hope to see you there!
Already registred ;) I'm EvilCartyen.
I saw the link to the forums on the A-I forums.
See you there! :)
I think this was a very good discussion we had going here, so I'll make a copy of it and paste it in the forum to be continued.
Post a Comment