Wow!

I'm dying to see what you can do with this thing...
Posted on 2002-01-14 05:24:38 by Qweerdy
Thomas,

wonderful job. This time the :alright: is very worth.

Programming could be easy but Mathematis & algoritms are bad beasties.

Maybe one day i will understand something about that...

Congratulation again.

B7
Posted on 2002-01-14 07:05:14 by Bit7
Let me add my admiration for some ground breaking work. I have never studdied nural nets, maybe this is my change to add then to my mind's toolbox32.

I agree, if you draw them large enough, it recognised the letters (or the portion of letters it knows) extremely well. But being a jokster, I then gave it letters NOT on the list.

It was quite sure an E was a C. Also took L for a C. 3 it took for D.

I point this out mostly to ask: Is that part of the training of a net? You give it good data and define the output as good, then give bad data and define the output as wrong?

How does one keep it from allowing false positives?
Posted on 2002-01-14 07:19:05 by Ernie
Once a nural net has learned what it is suposed to do will its processing of data be efficant? and could it be programmed to output tradional style programming code that would be more optimized once the learning was complete? Have I misunderstood the whole point here?
Posted on 2002-01-14 07:48:43 by Quantum
Thanks for all the replies!

bitrake:
Maybe, this could be merged into the newobject macro? Okay, maybe not - this really only works when dealing with single objects


We could offer it as additional macros in the next object include version...

doby:
what is the data u use to train the network? just a raw bitmap or u have some preprocessing?


At every 15 pixels in the total line you draw, a point is taken and the angle of the line that is formed by that pixel and the previous, is measured. All those angles are stored in an array.
When you finished drawing, the angle-array is split into 8 average angles, and these values are the input of the neural network.
The training is done by just drawing each character, then those 4 different sets of inputs are trained 100 times. I repeated this about 8 times and stored the weights in a file.
Feeding it the raw data would be very bad, as it's too much input (320x240 inputs...) and in that case the network would only learn specific pixel combinations which are useless because there are billions of possibile pixel combinations to draw an 'A'..
It's better to use limited inputs, but ones that carry much information.
For example, you shouldn't fire raw WAVE data at the inputs, but instead the FFT data of it.
how many inputs to the network?


8, see above.

and the last one what is network u r using? backpropagate?


Yes it is a back-propagation network, that was the hardest part of the calculations.

sluggy:
How difficult is it to take your concept one step further, and build a OCR type engine, for recognising text on a scanned image, or for ripping text out of a screen-scrape image?


Well the network would be very good for OCR purposes, but the problem is the inputs. In my example you draw the line and it measures real-time. However bitmap data are just pixels, and you'll have to find the relation of those pixels by tracing lines for example... But if you found a good algorithm to do that, you can feed the output as input to the NN.

Ernie:

I point this out mostly to ask: Is that part of the training of a net? You give it good data and define the output as good, then give bad data and define the output as wrong?
How does one keep it from allowing false positives?


It's behaviour entirely depends on the training.. You give the inputs, and the desired outputs. The network then produces an output, based on the input, and compares it's output with the desired output. The difference (error) is back-propagated to the network to adjust the weights of the connections.
I trained the example only with positive examples. I.E. I gave the input for an C, and set the desired output to "not A, not B, C, not D".
This way it assumes the input is either an A, B, C or D. The example will always give an answer, even if the outputs are all zero. I just check which one has the highest output value.. But you could change that and set a treshold on 50% chance or something like that. You can also train negative example by drawing an L and setting all outputs to zero. I just choose not to because it's just an example..

Quantum:

Once a nural net has learned what it is suposed to do will its processing of data be efficant?


Yes calculating the output isn't that hard.. The inputs of one neuron are multiplied by the weights on each input, summed up and put into the output function (1/(1+e^-x)). This is done for each neuron and can be done quite fast..

and could it be programmed to output tradional style programming code that would be more optimized once the learning was complete? Have I misunderstood the whole point here?

That would be really hard I guess but it won't give much improvement either I think. Right now there are a few nested loops that work out all the proper values in the right order. If you hardcode each seperate calculation into output code it would be very large code, so I think it's not efficient.

Thomas
Posted on 2002-01-14 09:52:18 by Thomas
Ernie,

I only know the *theory* behind this, Thomas has got light-years beyond this to actually making it work! (Still in awe). But i can tell you your pretty well correct with your assumptions. However, to my knowledge you dont train "wrong" situations, unless a wrong situation still needs to be "specifically" understood and indentifies a wrongness (if that makes sense).

Usually there is too many "wrong" situations to think of (in this case at least 22 more characters, + 26 if you add cases). To do this 9 times would not be alot of fun training. So what you do is provide some tolerance for corect identifications, and beyond that generalized anything else as "wrong".

Quantum,

Yes. But this is a VERY large leap from where Thomas is currently at! My school is doing alot of research into N.Nets for circuit design and simulators specific to Radio Freqency. They have trained software to provide accurate solutions to MAXWELL's equations ( very nasty mathematics, on basic geometries ), for very irregular shapes in antenna design. And is some cases shown more accuracy that traditional calculus!!

The beauty of this is: once trained, it will never forget. Likewise, you can "clone" another in a second, as all its "intellegence" is encapsulated into a matrix of decmal numbers (I believe Thomas said it was 108 interconnects ~ which is the "learned" info). So once you got a "Matrix" learned, you can hardcode it into software (and not alown any more learning), or have it an INI and continuously learn from the user, all taking a second to build into a new application.

I dont want to take the wind outa Thomas' sails here (and since he *made* it work ~ where i never got to), i will respectfully stop myself here :)

NaN
Posted on 2002-01-14 10:04:49 by NaN
by NaN:Usually there is too many "wrong" situations to think of (in this case at least 22 more characters, + 26 if you add cases). To do this 9 times would not be alot of fun training. So what you do is provide some tolerance for corect identifications, and beyond that generalized anything else as "wrong".


I just want to add that when you look at the actual output values of the network (which I did for debugging of course), the scores are lower when you draw other letters.. It's just that the example only takes the one with the highest score, regardless whether it's 0% or 100%..

The example could be much better, as there are some things that shoudn't be allowed..
For example, when you draw a simple circle, the angle changes from 0 to 359, but when it has reached 360 it falls down to 0 again. This produces a great (the maximum) difference between two angles which are almost the same (1 degree and 359 are almost the same, but the values differ very much).
It would be better to calculate the sinus and cosinus of the angle, and use those two as input. Because of the shape of them, there are no gaps.
Another thing is drawing the characters too fast. For example, if you would allow an "I": you can draw the I very fast, producing only a few points (few mouse messages) and straight line parts. It could split (interpolate) those line parts to get more evenly spaced points.

Thomas
Posted on 2002-01-14 10:47:45 by Thomas
Here's a version that shows the output values.

Thomas
Posted on 2002-01-14 11:18:25 by Thomas
Very nice program Thomas.
Do you think it would be hard to write A.I. to a game where the enemies learn and get better?
Posted on 2002-01-14 11:48:24 by ante
Hmmm... Thomas, here's something else strange: if I draw a mirrored A, it has these outputs:

A: 0 (!!!)
B: 5
C: 92 (!!!)
D: 5

Don't really get that.

Also, are you sure you trained the "D" to be drawn downwards? I get results between 20% - 30% better if I draw it upwards...
Posted on 2002-01-14 11:55:55 by Qweerdy

Hmmm... Thomas, here's something else strange: if I draw a mirrored A, it has these outputs:
A: 0 (!!!)
B: 5
C: 92 (!!!)
D: 5


That's not strange, as I trained it with A's drawn from left to right. Remember that it's not the actual image that serves as input for the network, but the angles of the drawn lines. As drawing it mirrored produces quite different angles, it doesn't recognize it.. You could train it for both (maybe even create two outputs for A, one normal, the other mirrored), but most handhelds require the right direction, to distinguish two similar characters like V and U by drawing them differently (left to right or right to left).

Also, are you sure you trained the "D" to be drawn downwards? I get results between 20% - 30% better if I draw it upwards...


I'm sure I trained it downwards, I don't get better results when drawing it upwards, it says it's a C or a B then.

Thomas
Posted on 2002-01-14 12:04:26 by Thomas

Very nice program Thomas.
Do you think it would be hard to write A.I. to a game where the enemies learn and get better?


I'm not very experienced with game programming but you could give it a try. Maybe start with something simple like tic-tac-toe...

Thomas
Posted on 2002-01-14 12:05:54 by Thomas
FASCINATING!!!
Posted on 2002-01-14 14:50:45 by NOP-erator
Heheh...

Just cluckling to myself, after all its is the right time of year.

What i mean is, last year at this time everybody was busy designing IDE's. Some were saying 2001 was the "year" of the IDE.

Now its fairly certian Im going to see alot of "year of the IA" type work. You can just tell by the tone of this thread..

NaN
Posted on 2002-01-14 15:35:18 by NaN
I just had an idea of combining both.. Opera has a feature called 'mouse gestures', which allows you to make specific movements while you press the right mouse button to trigger an event. For example, when you move to the left while holding the right mouse button, you go back in the history list, moving right goes forward, etc..
It would be cool to have such features in an IDE. Imagine you're using a variable somewhere:


mov eax, offset introString

Then you could right-click 'introString' and move down with the mouse to popup a small screen that lets you edit that variable.
When you get used to the movements this could be very effective and time saving... These mouse gestures are a piece of cake for neural networks.

What do you think?

Thomas
Posted on 2002-01-14 15:54:30 by Thomas

Then you could right-click 'introString' and move down with the mouse to popup a small screen that lets you edit that variable.

But would this require shifting from mouse to keyboard? That would
be very bad. For programming, I tend to stay with they keyboard...
that' s where most of the action is... and switching is bad.
Posted on 2002-01-14 16:04:20 by f0dder
This is what I mean:
Posted on 2002-01-14 16:04:35 by Thomas


But would this require shifting from mouse to keyboard? That would
be very bad. For programming, I tend to stay with they keyboard...
that' s where most of the action is... and switching is bad.


Hmm that's true, but when you're looking for a variable defined somewhere you would probably use the mouse as well to look for it.. Well maybe not, you could use shortkeys too. I'll leave it to the IDE designers :grin:

Thomas
Posted on 2002-01-14 16:08:46 by Thomas
This is what I mean:
Damn, good idea, i would feel totally spoiled if i had that in an IDE :). Anyway, now that you have voiced the idea, expect it to show up in a MS IDE sometime soon :rolleyes:
Posted on 2002-01-14 17:42:06 by sluggy
Thomas,

For example, when you draw a simple circle, the angle changes
from 0 to 359, but when it has reached 360 it falls down to 0
again. This produces a great (the maximum) difference between
two angles which are almost the same (1 degree and 359 are
almost the same, but the values differ very much).

Could you change your coordinate system so that
you only have to deal with angles between 0 & 90?
I'm assuming you are using a system like the one
on the right, but could you use one similar to the
one on the right?


farrier
Posted on 2002-01-15 00:55:33 by farrier