On source conventions and documentation [Release!]

A General Discussion forum for TI calculators

Moderator: MaxCoderz Staff

User avatar
Timendus
Calc King
Posts: 1729
Joined: Sun 23 Jan, 2005 12:37 am
Location: Netherlands
Contact:

On source conventions and documentation [Release!]

Post by Timendus »

(I'm not entirely sure this topic belongs in General Discussion; if one of the mods sees a better place to put it, feel free to move it)

The first fun little spin-off of the revived Vera project may come to you in an unexpected form: assembly coding standards and proper code documentation.

As you may all be aware, keeping source organized is not something the TI community is especially keen on. In fact, many people just produce "write-once-and-never-try-to-read-again" code that only they can understand (and only for the duration of the project) and afterwards the code will get lost on some harddrive anyway.

There are several more experienced programmers in the scene though, who have discovered that jotting down a few pre- and postconditions (and perhaps even something about the internal logic) in the comments of a routine really IS worth those few minutes of your time, as it will save you countless headaches as the project grows larger and harder to maintain. Diederik Kingma (Duck) was one of the first people I saw doing this in his greyscale programming package, though of course I couldn't have been looking at everyone's source. These experienced programmers all have their own ways of noting this metadata though, and usually it is still only meant to be read by themselves as it is mostly quite minimalistic.

Duck's GPP showed clearly in which circumstance having well documented code is absolutely necessary: in the case where you're releasing a library and you hope others will find it beneficial and want to interface with it. I was one of the first people to integrate his library into a game we were back then working on, and thanks to his easy to read comments and good documentation we didn't just manage to implement greyscale in our game (which was about 30k and quite complex) in just a few days, we also managed to track down a few interrupt related bugs in the Ti-83 specific code that we ran into. Through Ticalc.org I came in contact with Duck, I reported the bugs we had found, he introduced me on MaxCoderz, and the rest is history.

Later, having worked with javadoc a bit in college, and realizing the ease of generating documentation from your comments, I tried to standardize the headers above routines and in file descriptions among assembly programmers by providing a web-based asmdoc tool that would accept a few tags and turn your source into HTML output. It wasn't very popular though, partly because people didn't care or see any purpose in it, partly because of the limited use of a web-based application for this, and probably partly because of the poor implementation (only one file at a time, no proper export...).

Vincent (KV) did see the point in generating documentation though, and when he tried to set up a database of often used assembly snippets (the codebank), he contacted me to integrate it with my asmdoc tool, or something similar. As so often with ambitious projects, in the end it turned out we didn't have the time to finish it. It did however spark my API, which combined the code snipped database idea with an easy to use macro system (so you wouldn't have to go back to the website for every new snippet) as well as the automatically generated asmdoc idea.

Ben also got interested in automatic documentation from comments when he was working on Latenite and Brass, and together we tried to come up with a simple and easy to use "protocol" for documenting assembly code. However, we tried to make it kinda backwards compatible with existing source, and unintrusive to the user, which turned out to be impossible, and the attempt was postponed.

Now, almost five years after reading through Duck's code, it seems we're finally nearing both a good and simple asmdoc-protocol and the tools to generate documentation from it, thanks to the Vera project.

The typical trends of throw-away-code are exactly the opposite of what the Vera project is aiming for: fairly modular, easy to read and understand code that many people have to be able to work on, and that can be a learning experience for newcomers as well as a tool for research and play for the old farts. Vera has to be transparent. People have to be able to write applications against our API. This is why we have started to draw up some coding standards for this project.

As we were working on that, Ben wrote to the mailing list:
We could use this as an opportunity to (finally!) decide on a comment documentation standard. A parser that can rip out the comments and produce some nice HTML pages (and a CHM) would be relatively easy to write and would aid developers (who really wants to trawl through tons of source code?)
So we started designing that standard, finally finishing what we started years ago, and we've now come to the point where we have a coding standard for routine headers that is both easily human readable and parsable by software.

You can find the Vera coding standard on our wiki here:
http://timendus.student.utwente.nl/~ver ... _standards

(Please remember that it is still in development though, and that it could be subject to change at any time. However, no major changes are expected in the foreseeable future.)

In addition to that, both ElfPrince and I have been implementing two seperate comment parsers aimed at these standards, which are nearing usefulness. A bit of healthy competition has made sure that after just a week we have two almost done parsers that we will probably release to the public some time soon. This is our test assembly source (you'd best view the coding standards on the wiki first)...:

Code: Select all

;; Vera - the calc lover's OS,
;; copyright (C) 2007 The Vera Development Team.
;; 
;; This is a test assembly file to test the reference implementation
;; asmdoc tool.
;
; This program is free software: you can redistribute it and/or modify
; it under the terms of the GNU General Public License as published by
; the Free Software Foundation, either version 3 of the License, or
; (at your option) any later version.
;
; This program is distributed in the hope that it will be useful,
; but WITHOUT ANY WARRANTY; without even the implied warranty of
; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
; GNU General Public License for more details.
; 
; You should have received a copy of the GNU General Public License
; along with this program.  If not, see <http://www.gnu.org/licenses/>.

; ===================
; We'll put in a fake console routine here

;; === console.printline ===
;;
;; Print a null terminated string to the display
;; using some crappy display port writes
;;
;; Don't use it too often, it'll fry your screen :-)
;;
;; Pre:
;;   hl = pointer to string
;;
;; Post:
;;   String displayed on screen
;;   hl,bc,a destroyed
;;
;; SeeAlso:
;;   console.putchar, console.set_cursor_x, console.set_cursor_y
;;
;; Warning:
;;   Do not call this function if you have manually
;;   set console_cursor_x or console_cursor_y outside
;;   the console boundaries.
;;
;; Example:
;;    	ld hl,str_welcome
;;    	call console.printline
;;    	ret
;;   str_welcome:
;;    	.db "Welcome to Vera",0

console.printline:
	ret

;; === console.newline ===
;;
;; Print a newline to the console
;;
;; Post:
;;   Updated cursor position, scrolled screen if necessary
;;   hl,a destroyed
;;
;; SeeAlso:
;;   console.printline, console.set_cursor_x, console.set_cursor_y
;;
;; Example:
;;    	call console.newline

console.newline:
	ret

; === console.putchar ===
;
; Internally used routine, just one semicolon
;
; Pre:
;   a = character
;
; Post:
;   Character put on display
;   hl,a destroyed
;
; SeeAlso:
;   console.printline
;
; Example:
;   	ld a,'F'
;    	call console.putchar

console.putchar:
	ret

; ===================
; End of file
; ===================
...and my own implementation currently generates this from it (rendered by Firefox, click to enlarge):

Image

Or see for yourself here:
http://timendus.student.utwente.nl/~vera/asmdoc/
(Opera incompatibility is a known bug)

I'm quite sure ElfPrince will quickly post a screeny of his own output :) (I can't get your script to run as well as I would like, and I don't want to give myself an unfair advantage)

It almost seems like programming in assembly for calculators is finally growing up just a little bit :)

Edit: I've released the first public version of my tool, see this post.
Last edited by Timendus on Fri 19 Oct, 2007 11:22 am, edited 1 time in total.
http://clap.timendus.com/ - The Calculator Link Alternative Protocol
http://api.timendus.com/ - Make your life easier, leave the coding to the API
http://vera.timendus.com/ - The calc lover's OS
User avatar
benryves
Maxcoderz Staff
Posts: 3087
Joined: Thu 16 Dec, 2004 10:06 pm
Location: Croydon, England
Contact:

Post by benryves »

One possible query; will we be using #define XYZ value for numeric constants? I'd prefer to use XYZ = value, chiefly as this avoids the following problem:

Code: Select all

#define XYZ 3+3
    ld a,XYZ*2 ; would be 9 rather than 12.
Text replacement macros are not (IMO) suitable for numeric constants.

I'll have to nick the "Warning" colour. ;) Incidentally, the graphics I am using are from the famfamfam silk icon set.
User avatar
Timendus
Calc King
Posts: 1729
Joined: Sun 23 Jan, 2005 12:37 am
Location: Netherlands
Contact:

Post by Timendus »

Good point (though perhaps better brought up on the mailing list or the talk page ;)) - I've been in doubt about this, but you're right, we should use the XYZ = value notation for numeric constants. Shall I put that on the wiki, or do you want the honors? :)

And nick away, it'd only be fair :mrgreen:
http://clap.timendus.com/ - The Calculator Link Alternative Protocol
http://api.timendus.com/ - Make your life easier, leave the coding to the API
http://vera.timendus.com/ - The calc lover's OS
User avatar
benryves
Maxcoderz Staff
Posts: 3087
Joined: Thu 16 Dec, 2004 10:06 pm
Location: Croydon, England
Contact:

Post by benryves »

I've hacked together a "veradoc" plugin for Brass that inspects all of the comments and emits an XML file. I followed your example and used an XSL stylesheet to format it (screenshot) though it obviously could look better!

I forgot to take a copy of your sample XML output - it would probably be easier for all if the Brass plugin exported documentation in the same format as your tool. :)
User avatar
Timendus
Calc King
Posts: 1729
Joined: Sun 23 Jan, 2005 12:37 am
Location: Netherlands
Contact:

Post by Timendus »

benryves wrote:I've hacked together a "veradoc" plugin for Brass that inspects all of the comments and emits an XML file. I followed your example and used an XSL stylesheet to format it
Looking like fun! So we're getting a Brass parser too after all :)
I forgot to take a copy of your sample XML output - it would probably be easier for all if the Brass plugin exported documentation in the same format as your tool. :)
I agree, otherwise we'd be doing a lot of work twice. See here:
http://timendus.student.utwente.nl/~vera/asmdoc/
("View source")

Edit: Or check on the SVN in docs/asmdoc/
http://clap.timendus.com/ - The Calculator Link Alternative Protocol
http://api.timendus.com/ - Make your life easier, leave the coding to the API
http://vera.timendus.com/ - The calc lover's OS
User avatar
benryves
Maxcoderz Staff
Posts: 3087
Joined: Thu 16 Dec, 2004 10:06 pm
Location: Croydon, England
Contact:

Post by benryves »

Sample output.

I've made some modifications to your design with the Brass 3 plugin.

First of all, Brass listing plugins are designed to output files, not directories. The veradoc plugin emits a .zip archive containing all of the files, then.

Secondly; the initial page shown in the rightmost panel when index.html is opened is the first assembled file rather than being set to main.asm.

Thirdly; naming conventions. Each XML document is exported as dir.name.ext.xml. So, there's "page0.asm.xml", "test.txt.xml", "test.asm.xml", "devices.console.asm" and so on and so forth. Just in case we end up with a "devices.asm" and "devices.inc", for example. :)

It's a bit of a quick hack, but the source is here if anyone is interested.
User avatar
Timendus
Calc King
Posts: 1729
Joined: Sun 23 Jan, 2005 12:37 am
Location: Netherlands
Contact:

Post by Timendus »

benryves wrote:Sample output.
It's looking really good. No newlines anywhere in the source though, you may want to add a few for the sake of readablility. Does it also generate tree.xml?
Secondly; the initial page shown in the rightmost panel when index.html is opened is the first assembled file rather than being set to main.asm.
I've just been changing that to display a welcome.html first for my release. Not that it matters much; it's one filename in one of the lay-out files, so it's not hardcoded anywhere or anything.
Thirdly; naming conventions. Each XML document is exported as dir.name.ext.xml. So, there's "page0.asm.xml", "test.txt.xml", "test.asm.xml", "devices.console.asm" and so on and so forth. Just in case we end up with a "devices.asm" and "devices.inc", for example. :)
I've been thinking the same thing. But my application recreates the directories in the destination directory, so putting the directory name in the file name would be a bit overkill. Adding the original extention would be a good idea though; it wouldn't be all too weird to have a devices.asm implementing a few routines and a devices.inc defining some headers for it (thinks devices.c and devices.h). I'll add it to the todo list.

I've been thinking about also exporting a routines.xml containing the names of all routines and the name of the xml file in which it can be found. We could use that to add an alphabetic overview of all routines in a frame in the bottom left part or somewhere.
http://clap.timendus.com/ - The Calculator Link Alternative Protocol
http://api.timendus.com/ - Make your life easier, leave the coding to the API
http://vera.timendus.com/ - The calc lover's OS
User avatar
benryves
Maxcoderz Staff
Posts: 3087
Joined: Thu 16 Dec, 2004 10:06 pm
Location: Croydon, England
Contact:

Post by benryves »

Timendus wrote:
benryves wrote:Sample output.
It's looking really good. No newlines anywhere in the source though, you may want to add a few for the sake of readablility. Does it also generate tree.xml?
tree.xml is automatically generated, much like the rest of the XML; by default the XmlWriter class doesn't bother formatting the output XML. I can tell it to, though.
I've just been changing that to display a welcome.html first for my release. Not that it matters much; it's one filename in one of the lay-out files, so it's not hardcoded anywhere or anything.
Fair enough. :)
But my application recreates the directories in the destination directory, so putting the directory name in the file name would be a bit overkill.
I had it doing that, I'll revert back to match your system.
Adding the original extention would be a good idea though; it wouldn't be all too weird to have a devices.asm implementing a few routines and a devices.inc defining some headers for it (thinks devices.c and devices.h). I'll add it to the todo list.
That was pretty much my intention; I only really spotted it as I have a "test.txt" and a "test.asm" in the same directory as it was.
I've been thinking about also exporting a routines.xml containing the names of all routines and the name of the xml file in which it can be found. We could use that to add an alphabetic overview of all routines in a frame in the bottom left part or somewhere.
Good idea. :)

I'm obviously missing the "authors" field; I could also make this plugin raise compiler warnings if mandatory fields are missing (like the end-of-file bit).

Are the tag names case-sensitive ("pre:" / "Pre:")?

Should we enforce certain fields as being mandatory? The C# compiler moans about any public class, method, argument, property or anything else that isn't documented.
User avatar
Timendus
Calc King
Posts: 1729
Joined: Sun 23 Jan, 2005 12:37 am
Location: Netherlands
Contact:

Post by Timendus »

benryves wrote:I'm obviously missing the "authors" field; I could also make this plugin raise compiler warnings if mandatory fields are missing (like the end-of-file bit).
Should we enforce certain fields as being mandatory? The C# compiler moans about any public class, method, argument, property or anything else that isn't documented.
I'd say we raise compiler warnings if a file doesn't have an end of file thingy or if a routine definition misses at least a description. We can't really make it moan about pre/postconditions because in many cases you'll have only one of the two. Example perhaps..?

(I'm not sure I'm going to implement any of this any time soon though, I'd like to finish this version and release it, and it'd be of more use if the actual compiler complains about them anyway than just some docgen tool that you use only once every few weeks.)
Are the tag names case-sensitive ("pre:" / "Pre:")?
They are case-sensitive, and the proper spelling is Pre / Post / SeeAlso / etc.
http://clap.timendus.com/ - The Calculator Link Alternative Protocol
http://api.timendus.com/ - Make your life easier, leave the coding to the API
http://vera.timendus.com/ - The calc lover's OS
User avatar
benryves
Maxcoderz Staff
Posts: 3087
Joined: Thu 16 Dec, 2004 10:06 pm
Location: Croydon, England
Contact:

Post by benryves »

Timendus wrote:I'd say we raise compiler warnings if a file doesn't have an end of file thingy or if a routine definition misses at least a description. We can't really make it moan about pre/postconditions because in many cases you'll have only one of the two. Example perhaps..?
Examples should only be required for nontrivial routines, and guessing whether a routine is nontrivial or not is not very easy (eg, a routine "maths.abs" that just returns abs(hl) wouldn't merit an example IMO).
(I'm not sure I'm going to implement any of this any time soon though, I'd like to finish this version and release it, and it'd be of more use if the actual compiler complains about them anyway than just some docgen tool that you use only once every few weeks.)
Well, this helps me design Brass; currently you just get a flat array of Compiler.SourceStatements to work with, and can't currently (for example) tell if a statement is the last in a file or not without running through the following statements and checking their filenames.
Are the tag names case-sensitive ("pre:" / "Pre:")?
They are case-sensitive, ...
Just checking. :)

Now, we're using this to document routines; surely we can also use it to document variables and constants? We'd need to specify things like datatype and range (for variables), or whether it is a variable or a constant. Examples will be useful, warnings can be added... the design is fairly scalable.

Also, as we're using XML; would you mind if I added some extra attributes (or nested elements?) to routines, such as page number or address? That way the documentation could be easily loaded into an emulator for the debugger.
User avatar
Timendus
Calc King
Posts: 1729
Joined: Sun 23 Jan, 2005 12:37 am
Location: Netherlands
Contact:

Post by Timendus »

benryves wrote:Examples should only be required for nontrivial routines, and guessing whether a routine is nontrivial or not is not very easy (eg, a routine "maths.abs" that just returns abs(hl) wouldn't merit an example IMO).
So what should we do? Not require anything..? Require *something* besides the name? :)
Now, we're using this to document routines; surely we can also use it to document variables and constants? We'd need to specify things like datatype and range (for variables), or whether it is a variable or a constant. Examples will be useful, warnings can be added... the design is fairly scalable.
Yup, once again you're voicing my thoughts :) We'd need to think this through properly though, and I suggest we wait a bit so we have some example source to work with.

My first thought would be to do something like this:

Code: Select all

;; === Routine ===
;; ~~~ Variable ~~~
;; ### Constant ###
And of course tags after that...
Also, as we're using XML; would you mind if I added some extra attributes (or nested elements?) to routines, such as page number or address? That way the documentation could be easily loaded into an emulator for the debugger.
I wouldn't mind, as long as it requires no action from my side you will not hear me complain ;)
http://clap.timendus.com/ - The Calculator Link Alternative Protocol
http://api.timendus.com/ - Make your life easier, leave the coding to the API
http://vera.timendus.com/ - The calc lover's OS
User avatar
benryves
Maxcoderz Staff
Posts: 3087
Joined: Thu 16 Dec, 2004 10:06 pm
Location: Croydon, England
Contact:

Post by benryves »

Timendus wrote:So what should we do? Not require anything..? Require *something* besides the name? :)
I'd require a description, but that's it.
My first thought would be to do something like this:

Code: Select all

;; === Routine ===
;; ~~~ Variable ~~~
;; ### Constant ###
And of course tags after that...
That could work, yep. :) Corresponding XML tags; <routine />, <variable />, <constant /> ?
I wouldn't mind, as long as it requires no action from my side you will not hear me complain ;)
Just didn't want you to accuse me of polluting the format. :)
User avatar
benryves
Maxcoderz Staff
Posts: 3087
Joined: Thu 16 Dec, 2004 10:06 pm
Location: Croydon, England
Contact:

Post by benryves »

I've updated the plugin a bit; it now raises the following errors:

Warnings
  • Routine name isn't valid (uppercase characters, non-ASCII, starts with number, contains underscores at start or end of name).
  • Documented name doesn't match compiled name (console.xyz in documentation, Console.XYZ in source).
  • File or routine doesn't have a description.
Errors
  • Routine name doesn't have a matching label.
  • File doesn't have an end of file marker.
  • File has source (other than whitespace or ; =====) after the end of file marker.
The XML is also easier to read (I asked the XmlWriter to indent it).

Each routine has the following additional element:

Code: Select all

    <label>
      <value>16385</value>
      <page>31</page>
      <size>1</size>
    </label>
Documentation is here.
User avatar
Timendus
Calc King
Posts: 1729
Joined: Sun 23 Jan, 2005 12:37 am
Location: Netherlands
Contact:

Post by Timendus »

I understand I can safely throw my implementation out of the window as soon as Brass 3 comes out ;) When do you plan/hope to release it?
Documented name doesn't match compiled name (console.xyz in documentation, Console.XYZ in source).
This should be an error (as it produces incorrect documentation for case-sensitive compilers)
File doesn't have an end of file marker.
This should be a warning though; I don't think it shouldn't compile just because there's no end of file marker - people outside our project will not like that kind of behaviour from their compiler :)
File has source (other than whitespace or ; =====) after the end of file marker.
This is a good error though; if you go through the trouble of marking the end of your file, you'll probably not want anything after it.

Concerning the routine, variable, constant thing; are we sure that those are the three things we want? Is that a logical distinction to make? In a way we now have "file" and "routine", it would make sense to add predefined memory locations to it (like saferams), I suppose that's what you have in mind with "variables"? And "constants" are things like val = 123 that don't point to locations in memory but define some value (link port writes, things like flags and stuff)?

It would be fairly annoying to have to define every constant like this:

Code: Select all

;; ### LINKPORT ###
;; 
;; Defines the port that you need to write to for changing the
;; current of the link port lines
;;
;; Value:
;;    0

LINKPORT = 0
Especially if we have a lot of them in something like vera.inc. And it would be completely pointless, seeing that most of the information is already in the statement LINKPORT = 0. Maybe we could allow that, but also allow to abbreviate that a bit to something like:

Code: Select all

;; -###-
;; Defines the port that you need to write to for changing the
;; current of the link port lines
LINKPORT = 0
And have the docgen extract the name and the value automagically?

(Of course, this would be hell for my parser, but seeing that Brass parses everything anyway..?)
http://clap.timendus.com/ - The Calculator Link Alternative Protocol
http://api.timendus.com/ - Make your life easier, leave the coding to the API
http://vera.timendus.com/ - The calc lover's OS
User avatar
Timendus
Calc King
Posts: 1729
Joined: Sun 23 Jan, 2005 12:37 am
Location: Netherlands
Contact:

Post by Timendus »

Asmdoc tool v1.4 release! Get 'em while they're hot! :)
Both Windows and Linux binaries.

http://timendus.student.utwente.nl/~ver ... c_v1.4.zip
Unzip somewhere and read the README file (if it's garbled in notepad, try using Wordpad or a proper text editor).

Feel free to post bug reports or feature requests in this thread.
http://clap.timendus.com/ - The Calculator Link Alternative Protocol
http://api.timendus.com/ - Make your life easier, leave the coding to the API
http://vera.timendus.com/ - The calc lover's OS
Post Reply