But I think the idea of format-dedicated separate linkers is good. I'm thinking about writing a PE linker (MS COFF objects -> PE executable, of course) that would automatically search for unresolved references in the DLLs in the user-specified directories (eg. WINDOWS\SYSTEM), so you wouldn't need any libraries of that kind. When I have some time, I'll try to implement it (shouldn't be really hard).
Posted on 2003-01-25 08:12:33 by Tomasz Grysztar
Ok, I made a quick hack to support EXTERNDEF in NASM ;)
NASM didn't track the use of labels, so I had to implement it, too.

BTW, I checked out FASM, and gotta say, you've done good work, Thomasz!

I was also coding my own assembler a long time ago, but the project ended because the assembler I was developing it on, had so many bugs, and then, when I got a good one, I realized there were so many good existing assemblers, there wasn't much point in coding another one.

It's great, that some people still code assemblers in asm :grin:

-Stealth
Posted on 2003-01-25 10:01:48 by Stealth

Randall, I think separate linkers would be wasteful, as there's very much duplicate code. Rather, having input/output format DLLs, or (what I prefer), build-time inclusion of the formats you want support for.


Looking at the source code for ALINK, I think I'd probably have to agree with you.
The PE stuff is the largest part of the code. The OMF/EXE and COM output stuff
was quite short. Not having looked much at ELF yet, but understanding that
it's based on COFF, I suspect the same is true for ELF as well.

I tried to get ALINK to work with FASM (and MASM) output, but it has a few
defects that prevented linking. Alas, the license on the code expressedly
forbids making modifications to the source and distributing them, and I've
yet to hear back from the author about the code, so I suspect I'll have to
write my own linker...
Randy Hyde
Posted on 2003-01-25 12:04:37 by rhyde
the alink author is very nice, but i think he's been rather inactive for a longer period of time.

i think the way to write a linker (if you want to do something worthwhile, and not just "yet another linker") is to (sort of) adopt some of the GNU ld ideas. convert input formats into some "canonical" internal format (ie, preserve as much info as you can). be sane, though; ld wants to support "everything", while 'we' probably only need a handful of formats. then comes the reorder/resolve state, which only has to be written once, and is independent of input/output formats. finally comes the output stage, which is output format dependant. I don't have any clear finalized idea of how much code goes into each stage, but I think it should be possible to do something decent and reasonable.
Posted on 2003-01-26 16:47:44 by f0dder

the alink author is very nice, but i think he's been rather inactive for a longer period of time.

i think the way to write a linker (if you want to do something worthwhile, and not just "yet another linker") is to (sort of) adopt some of the GNU ld ideas. convert input formats into some "canonical" internal format (ie, preserve as much info as you can). be sane, though; ld wants to support "everything", while 'we' probably only need a handful of formats. then comes the reorder/resolve state, which only has to be written once, and is independent of input/output formats. finally comes the output stage, which is output format dependant. I don't have any clear finalized idea of how much code goes into each stage, but I think it should be possible to do something decent and reasonable.


As I believe I mentioned elsewhere, I haven't looked at ELF yet, but my
understanding is that it's based on ELF. Looking at the ALINK source code
certainly suggests that (from an input perspective) most OBJ file formats are
quite similar. For output, supporting COM and (non-PE) EXEs was just a small
amount of code.

Of course, for my immediately purposes, I only need the ability to handle
PE/COFF and, at some point, ELF. So I'll probably start with those.
I may regret the decision later (i.e., fret that I should have done the design
up front), but it turns out that a linker is a sufficiently small program that
writing it over is not a big task (unlike, say, my HLA compiler which is in
excess of 150,000 lines of code at this point).
Randy Hyde
Posted on 2003-01-26 23:37:11 by rhyde
While we all think about making a new linker, maybe we could make it common project?
Posted on 2003-01-27 04:07:53 by Tomasz Grysztar
There already exists a COFF-PE linker that automatically searches for unresolved symbols in dlls. (GoLink, http://www.godevtool.com)
Posted on 2003-01-28 13:02:52 by Sergo
But I mean the linker with full support for COFF libraries (as well as ELF maybe) that would search in DLLs only for symbols unresolved from objects and libs. I actually have some design in mind, maybe I'll try to implement it in spare time. Such linker would have to make table of sections and symbols of all given objects (also those from libraries), then mark section containing entry point symbol as "needed" and repeat marking this way sections containing symbols that are referenced by relocations of already "needed" sections. Then it would link all "needed" section, resolving all the relocations, and when some symbols still remained unresolved, the appropriate final formatter would treat them in its own way, for example PE linker would search for such symbols in DLLs and make the imports then.
Posted on 2003-01-28 15:29:05 by Tomasz Grysztar

But I mean the linker with full support for COFF libraries (as well as ELF maybe) that would search in DLLs only for symbols unresolved from objects and libs. I actually have some design in mind, maybe I'll try to implement it in spare time. Such linker would have to make table of sections and symbols of all given objects (also those from libraries), then mark section containing entry point symbol as "needed" and repeat marking this way sections containing symbols that are referenced by relocations of already "needed" sections. Then it would link all "needed" section, resolving all the relocations, and when some symbols still remained unresolved, the appropriate final formatter would treat them in its own way, for example PE linker would search for such symbols in DLLs and make the imports then.


Besides "smart" linking capabilities (i.e., only incorporating the actual object modules that contain code referenced by the main program's call tree), the linker should be able to produce library (.lib) output. It should also be capable of producing EXEs with appropriate debug information (for programs like OllyDbg), and support most of the important
MS-link options, e.g.,
-align:nnnn
-base:address
-dll
-debug
-entry:name
-heap:size:commit
-map
-merge:from=to
-out:filename
-section:<options>
-stack:size:commit
-subsystem:{windows|console}

(I think I've got all the "necessary" options; someone correct me if I've missed something important.)
It would also be nice if it handled .DEF files.

I'm working on such a linker right now.
Once I get it to the point that it handles HLA/FASM output reasonably
well, I'll put it in the public domain and people can play with it and
extend it all they want.
Randy Hyde
Posted on 2003-01-28 23:03:48 by rhyde

(...) and support most of the important MS-link options, (...)


Well, these are just a little additions to the final linking stage - not really much work IMO


I'm working on such a linker right now.
Once I get it to the point that it handles HLA/FASM output reasonably
well, I'll put it in the public domain and people can play with it and
extend it all they want.


All right, since you're already working on it, I'll concentrate on other projects. We should avoid doing the same work twice. ;)

BTW, do you have any good reference how should debug info for Microsoft COFF look like? I was thinking about adding some listing-like debug info support for FASM (that means storing offset in output file for every line in source - essential for source-level debugging), but it seems that debug info usually holds much less information... Well, if I won't be able to store such info in object files, I will have to add just a listing feature only (IMHO symbol-level debugging facility isn't really helpful, at least when compared to source-level debugging).
Posted on 2003-01-29 06:08:15 by Tomasz Grysztar
flexible linker scripts!
being able to control section merging, inserting global symbols between sections,
controlling alignment, ordering, . . . -- like the gnu linker can do.
Posted on 2003-01-29 06:17:24 by f0dder
Privalov: IMHO symbol-level debugging facility isn't really helpful, at least when compared to source-level debugging


IMHO: You are absolutely right ! :alright:

More: I don't like link stage at all. Yes I know it's a very flexible and powerfull, but I don't like it. I want a compiler that produce executable files, not some kind of object files. FASM is great namely with this feature.
Posted on 2003-01-29 22:09:58 by JohnFound



IMHO: You are absolutely right ! :alright:

More: I don't like link stage at all. Yes I know it's a very flexible and powerfull, but I don't like it. I want a compiler that produce executable files, not some kind of object files. FASM is great namely with this feature.


Still,without the link stage,you can't use external libraries such as masm32.lib. :)

Regards,

Vortex
Posted on 2003-01-30 03:29:35 by Vortex
Hi Vortex.
If I want to use masm32.lib, I will use masm.:) I don't need second masm. The advantage of fasm is that I can take direct binary file, not objects, not linking. But I think fasm needs some different way of making reusable code. Something like Pascal units, on source level, not on object level.
Posted on 2003-02-01 00:44:06 by JohnFound

Hi Vortex.
If I want to use masm32.lib, I will use masm.:) I don't need second masm. The advantage of fasm is that I can take direct binary file, not objects, not linking. But I think fasm needs some different way of making reusable code. Something like Pascal units, on source level, not on object level.


That's great, until you write a compiler that generates FASM code.
Now the compiler has to sift through umpteen thousand lines of source
code to inject the appropriate library routines into the source code for
compilation by FASM. Not a healthy idea!

As for use MASM, once again that's a great idea for an individual, but if
you're writing a compiler that generates FASM code to avoid the licensing
headaches associated with emitting MASM output, having FASM work
with linkers (which it does) is a good idea. But you do need a linker
to pull this off.

I completely fail to see a user's aversion to a linker.
No matter how fast FASM (or whatever) is, linking is always going to be
faster than assembling. Even if assembly is virtually instantaneous,
linking is not going to hurt assembly performance. And the linker is
sufficiently small that the extra code size (bloat) is trivial compared to
the benefits obtained. Whether the linker is an external program or
built into FASM (and keep in mind, there *is* a rudimentary linker built
into FASM today, or you wouldn't be able to refer to objects in DLLs)
is strictly an implementation issue. I suspect that if FASM incorporated
all of MS' link.exe program internally, people would still be arguing that
they wouldn't want a "linker" associated with the program. More probably,
they don't want to have to manually run a separate program to get an
executable. Of course, you *don't* have to do this. Typing something
like "FASM -f:pecoff -o:x.exe x.asm, y.obj, z.lib" should be sufficient to
do everything, regardless of whether the linker is an external program
or built directly into FASM (e.g., HLA certainly works this way, so does
MASM ).

Of course, for those who absolutely insist on writing single-source
compilation units that produce an executable directly, FASM could
pretty much work the way it always does. To the end user, there
would be very little difference.

What's preferable? Running FASM multiple times to produce a series
of object files and then explicitly running an external linker (as is the
case today) or running FASM to do all the work? Then again, it's easy
enough to write a shell program that automates all of this stuff
(e.g., ml.exe and hla.exe are both examples of such types of code).

The bottom line, however, is that a free linker is necessary in this
equation.
Randy Hyde
Posted on 2003-02-01 02:14:16 by rhyde
I have read this thread with some interest and I still do not understand the aversion to a seperate linker that allows the creation of seperate modules that can be put together as a library for reuse.

MASM is an old product from the era when everything did it this way but it makes it a very useful capacity in that prebuilt blocks of code can be written and stored by the programmer. I started on the MASM32 library a long time ago and with the help of many different contributors, it does handle some thing reasonably but I do not claim for one moment that this capacity has ever been fully realised in MASM32.

Putting this capacity into FASM would be a quantum leap forward and it would make the construction of much larger applications possible within a far smaller time scale. Factors like this are the ones that make a programming tool succeed as few have the life or time to produce everything at the bare mnemonics level every time.

I suggest that if a linker is seen as the way to go, probably seperating them on the basis of platform would be the way to go. I see no problem with keeping DOS com/EXE and PE EXE in the same linker as they can all be used on the same box but file formats that are specific to linux would probably be better served by a seperate linker that can build those on linux platforms.

There is no reasonable reason why tiny apps cannot be handled in much the same way as they are done now but add the capacity to create and use library modules to extend the capacity of the assembler from a one shot to a many shot tool.

Regards,

hutch@movsd.com
Posted on 2003-02-01 04:03:19 by hutch--
What about makefiles? Very handy and powerful, I was using them with TASM and I'm still using them with FASM under Linux (where I use system's "ld" linker).

I think it's important to let you know, that I have no aversion to linking. Well, I don't use it actually, but it's just because all my current assembly projects are relatively small (including FASM itself). But if I was used to multiple objects/libraries approach before I have written FASM, and it wasn't implemented in my assembler just because I didn't have any good docs about format of object files. I was writing the very first version of FASM mainly for OS programming stuff, and I needed only output in plan binary format, and only this output was available in FASM 0.90 - it was also usable for writing .COM files and boot sectors, you could even create an MZ EXE by maually filling the headers. So I thougt it's a nice tool for writing such a small DOS programs (or small assembly OS kernels) and I've released it on FreeDOS project pages. Then I've added support for MZ EXE output to make it bit (but only a bit) easier to create such programs (it was FASM 1.0) and then I've got the old documentation of PE format (but without a word about COFF) in text file from the Internet (my access to web was still very occasional these days), and thought it'd be fun to implement such format generator in FASM. Since 1.04 it was able to produce Win32 programs, altough you still had to manually build most of the structures. And, suprisingly, I've got some feedback - people liked this approach, altough it's suitable only for small programs, altough you still can make and use some libraries by using DLL mechanism - and it's good to use it when many of your programs share large amount of the same code. This way FASM got some popularity. And so I've even started project of FASMW, which has to be small tool you can take on a floppy with yourself and use for creating small Win32 programs without any additional tools. But still the command line FASM is the main part of my project, and I had been planning support for object output and the linker for a very long time, many people requested them (just because they liked many of FASM features, but was unable to use it in their non-trivial projects because of lack of object output). I've added COFF, then I've added ELF (altough still without some features), and people are using external linkers. Maybe I should write some dedicated linker for FASM to make this package complete, but people will probably stick to their favourite linkers, because I wouldn't have enough time to write such complete linker that would satisfy everyone. But maybe I'll make some small linker that I could put on my website for those, who don't have any and want to make/use libraries. Or maybe I could point to your linker, when it's finished?
I see it this way: good programming environment consists of many separate tools, like compiler, linker, text editor, make command, resource editors/compilers etc. It's hard for one person to make all these tools and keep every one in high quality. And so it'd be best if there was one person for every such component, making it really good, and then one could grab them all together (choosing the favourite text editor BTW, people have very different preferences about this) and compose the IDE of really high quality.
I'm making the assembler already, and I'll concentrate mainly on it. I hope some else could make a good free linker that could be used with FASM, there are already many good editors that could be used with it, etc.
And my main aim right now is to make more of really good documentation for FASM, about its internal structure for those advanced users who want to modify the compiler for their needs, and some Win32 tutorials (like Iczelion's, but FASM-dedicated) for beginners. Maybe also DOS/Linux tutorials. I've been planning it since July, it's high time to make something about it... :)
Posted on 2003-02-01 04:39:05 by Tomasz Grysztar