I've been directed to this mb by bitnaut, who has been using my 16-bit interpreter for a couple of years now. I intend producing a 32-bit version of Ketman in due course, but I'm not the fastest programmer around, and you might beat me to it. So I'll give you a couple of pointers. If you start on this project, you'll find there is no end to it. First, locking it to MASM isn't a good idea. Your project is more than just an additional tool, it's quite fundamental in what it does. You must go the whole hog and write your own assembler. Better to write your own editor as well, and have a trio of interlocking tools. Only then can you guarantee complete consistency between them. Also, to gain an extra advantage over conventional debuggers, you need to be able to back-step through your program, effectively reversing all operations. Maybe fifty percent of an interpreter's value would lie in being able to go backwards.

But be prepared for huge psychological resistance from programmers brought up in an interpreter-free environment. My list of registered users consists almost entirely of newbies and/or fresh starters. Those who already use MASM, TASM, A86, etc won't touch your project. That's not because it isn't any good, but because they think it oughtn't to be any good. I don't know why, but that's the way it is. But newbies will love it, and if later they have to use MASM, etc, (as bitnaut is doing at the moment) they will always be complaining about all the things it can't do which your interpreter can. So what you end up with is a small but fanatical users club, too small to make you think yourself a success, but fanatical enough to keep pressuring you for new versions with more features. They'll never let you move on to something that might make you rich.

But having the only asm interpreter in existence is becoming a bore. I'd like some competition. So good luck.
Posted on 2001-10-08 14:48:29 by Ketman
Perhaps many in here haven't used Visual Basic. It runs interpreted or it compiles and you run an exe.

The VB debugger is close to what I'm describing. You run it not to run it interpreted in the end result. You run it to change your lines of code while it's still executing. You run it to see the values of your variables during run-time. You run it to execute a line of code, see the error, change that line of code, go back a line or to, step trace again, find an error, change the line of code, go back a line or to and repeat until corrected. That's one more reason why VB is easier to debug. It's not nearly as powerful a debugger as VC++ debugger or SoftICE or IDA, but it's a different kind of debugger. One that lets you modify the source while it executing and execute that modified source without restarting or recompiling. That's the aim of this.

This is not to make an interpreter. This is to add a new type of debugger. Simple. I, myself, will only use it for debuggin purposes but I'll use it a lot. Ketman, is your source available?

Thanks,
_Shawn
Posted on 2001-10-08 15:13:50 by _Shawn
Ketman,
Thank you for the advice. "Been there & done that" IS useful knowledge aka experience. If you have anything more you want to spill dial rafe@scinq.org. Bet you've got even better dish on this.

The ground-up asm idea is what I was thinking too. Internally, it will be in a form that the interpreter needs but we need to be able to draw upon the huge program base out there so input of masm (later fasm, spasm....) is key. Some will never adopt the interpreter & others may if the transition is slow enuf & they don't loose what is out there already.

Thanks again
rafe


PS: The current project at work is to parse Genbank files using perl & put the data into postgresql. Funny how this project all of a sudden became useful experience for the job :D
Posted on 2001-10-08 15:26:02 by rafe
Shawn, your understanding of the meaning of the word "interpreter" matches mine entirely. To run a finished program with it would be insanely slow. No, of course, it's a debugger. Assembling your program is the final indispensable stage. No point having asm programs if they don't run as machine code. The purpose of my interpreter is exactly what you say yours would be, which is to brainstorm a program while it's still under construction.

I haven't used VB but the old QBASIC worked along similar lines. It had a primitive interpreter, allowing single-stepping, but the display left a lot to be desired. In fact, I think it was while using it in 1992 that I sketched out the requirements for my asm interpreter.

No, I'm afraid the source for Ketman isn't available. For one thing, I'd have to translate it from Ketman-ese to MASM-ish. Long job. Also the Ketman Schoolmaster is a finished piece of software, a complete package that took five years of work before V1.0 made its appearance. I wouldn't give the source away.

Just seen your message, Rafe. So there are two of you working on the idea? Great stuff. If you want to see how mine works you can download it from http://geocities.com/SiliconValley/Office/6208/
It's V2.4, shortly to be replaced by V3.0, with luck.
Posted on 2001-10-08 16:02:30 by Ketman
Ketman, Yeah, we're working on it together... He's doing the parsing and lexing and scanning, I'm doing the interpreting. It's a fun project...


Thanks,
_Shawn
Posted on 2001-10-08 16:12:21 by _Shawn
I can see that this would be of good use, if it could be
done in windows. I wrote one I callled zrun it was for
the z80 it could single step code while showing all
reg's, and even data blocks, and you was able to
change the code, and or add code just like you would
in your source code, infact the code looked just like
your source code with all the comments and lables
the way I did it was, when ever you added or changed
the code, zrun would add that code at that point
and if it was adding more code to your program it would
insert it into the compiled code in memory and then
relocate all instructions that may have been effected
I also kept track of data areas, and there locations as
well as all the labels, this is not hard to do in DOS, but
now my question is this, in wondows how would you
do this, one, how would you, or how can you trap a
instruction so your code will not run away, so you can
do a call into a API and not really step into it? these
things have stoped me from writting one for windows
as I can't really set brake points on each instruction the
way I could in DOS. and is it posible to open a memory
area and load a program into it and single step the code
and windows will give it it's own handles as if it was it's
own prossess, even though my program was the one
that gave it the memory area and not windows? I have
alot of questions on this subject, as it seems I don't have
enough info on the thru workings of windows to solve
any of these problems.


Zcoder....
Posted on 2001-10-09 23:27:27 by Zcoder
Rafe,

perhaps you have some info you'd like to share on this?



ZCoder,

The way I see it, we aren't taking the compiled form of the program and step tracing through it isn't what we have in mind. Actually, the reason I always say "simulate" is because we will simulate the instruction and before it gets the next instruction if there's a break point, it'll pause. Of course, some people in here will have problems. But I can't see how it would work any other way because of API's and resources and stuff. We really have to contain it in our own little "box". However, we'll opimize it. Once we have that part of it working, we'll start to look into optimization. There's a few approaches, like the way you mentioned, but with processes and threads and stuff, it'll be harder to do it "out of the box". Then you have performance problems. But then again, this isn't to run the program only to help interactively debug. In all, I think for Windows, it's much harder but we'll do it and I think the source will be open to all to see. So perhaps once it gets rolling people can pitch in and help.


Thanks,
Shawn
Posted on 2001-10-09 23:52:06 by _Shawn
Years ago when I was writing in SDK C in win 3.1, I used a german written dialect of basic as well that had the unusual characteristic of being able to run API code and resources directly from the editor which was a basic interpreter. It was not without the odd bug here and there and to get code reliable, you still compiled it near the end of the development cycle and tested it to see if it was working correctly.

The old QB and dos qbasic was done with a big runtime library, QBRUN45.exe but they were a lot simpler than API code and what you guys have in mind. I would suggest you see if some of these things can be done first as it will save a lot of development time not chasing things that are impossible.

Stepping back with asm instructions may be possible but with API code it sounds very difficult to do. It sounds like handling each API as a block, pushes and call syntax that could be modified before the call was made.

There would also need to be some robust exception handling to catch API errors that would normally crash the application. I personally dont mind the compile cycle as you are testing the real thing.

What you are aiming at sounds like a single step debugger that works at the source code level. This can already be done but back stepping may be a lot harder to do as the inbuilt debugging support in windows is poor along side DOS and real mode.

Regards,

hutch@pbq.com.au
Posted on 2001-10-10 06:42:54 by hutch--
Zcoder & Hutch,
Actually, you've hit the nail on the head as far as backing up thru an api call... Shawn & I have been kicking around a few ideas but we haven't settled just yet.

We're going to take baby steps. Right now I'm more concerned about getting all of the masm allowable forms of base+index*size+offset addressing to work correctly & that damned dot is overused... grrr. Ketman's point about dumping masm is quite tempting right about now ;)

Hutch,
This project will never be all things to all people... it may do nothing more than adorn my mother's refrigerator but we obviously hope otherwise. This thing isn't going to drive you to work every morning. We're going to have to target a small audience at first & then try & widen appeal on sucessive revs. It may never get to the point where you'd personally use it ever/often but you could always kick the tires a bit & put in your wish list.

which leads me to getting back to avocation stuff...

Shawn,
I'm out of the 16-hr workday mode & back to the project. On the first time thru (after PoC drafts) I'd like to target the newbies. Immediate feedback makes for *much* quicker learning, that much I learned from Dr.-Mrs.-Wifee-poo. What do you think?
Posted on 2001-10-10 10:00:41 by rafe
hmmm, backing through code... this will be impossible with a lot of
API calls, since the changes that an API call can do are overwhelming...
file modification, internal windows structures, etc.

For backstepping through your own code, especially algos (which
is the only place I can see where it's useful), should be possible.
However, there's a few things to keep in mind... data access/modification.
What will you do about that? For each instruction, keep a list of
memory that will be modified, and the previous state? Ouch. And
what about code running in loops? :).

The project sounds interesting, but it will be a tough one.
Posted on 2001-10-10 11:13:19 by f0dder
We make some compromises. Again, the Visual Basic debugger. You can backstep or even drag the pointer to the line of code you want to execute, both forward and backward. But if you go backwards, nothing gets reversed. If you deleted a file and go backwards to decide you wanted to change something, the file is still deleted.

If you change the contents of a parameter or variable, from 5 to 15, then no matter where you go in your source, provided the variable is still in scope, it is still 15. If you call an API, whatever effects that API had, are still in effect even if you go back a few lines of code. That much is understandable.

But if you are doing calculations or manipulating objects or something like that, that is not truly reliant on API persistance (there are API's who effect lasts for a while throughout code, and some whos effect is immediate and that's it), then you can keep stepping through code for an eternity until you get it right. I'd say 60% of the defects I've had to debug in any language are simple bugs like that, that are more or less incorrect constants, missing a crucial line of code, or a line of code out of order, or a calculation incorrect that I have to correct, much moreso than wrong API's and things like that.

That's just me. Like I said, everything must have a compromise. I don't think that we'll ever be able to reverse the effects of an API put in motion, unless we want to track every API and give it knowledge about what every API does. That won't make this useless.

By focusing on what we can do makes it easier to accomplish, rather than achieving what we can't do, or don't know how to do, won't get us anywhere but dead ends (unless a miricle takes place).

For all the, the value will still be there for those who choose to at least see what it can do.

Thanks,
_Shawn
Posted on 2001-10-10 11:38:31 by _Shawn
_Shawn,
i've been reading what you say about the VB interpreter, and i totally agree with you, it's functionality is excellent (i'm a VB developer by trade). 90% of its functionality is also available in the VC editor, and yes, you have the choice of either stepping through your source, or stepping through a mixture of the source and the compiled result (for this it gives your asm line, then what that line compiles to, then your next asm line, etc). I do all my asm writing in the VC editor, the only thing it doesn't provide which i really miss is syntax highlighting and navigation between procedures.

The only real negative point about the VC editor is that it costs, and a lot of hobby/part-time programmers don't want to have to pay several hundred dollars for it, and because of its size it is not easy to obtain a "demo" (wink, warez, wink) copy of it via the net.

So, knowing the complexity of the VC interpreter, i can say you've got a hell of a job ahead of you :)

sluggy
Posted on 2001-10-10 20:53:12 by sluggy
The debuggers/interpreters I've used don't attempt to reverse any action when the instruction pointer is moved - I assume this is for the sake of consistancy - if you can't reverse every action, don't reverse any action. Well, at least you can usually guess the state things are in -- then set proper breaks, or manually adjust registers/varaibles. Works for me. :)
Posted on 2001-10-10 23:36:30 by bitRAKE