Convert LDMud docs to a Doxygen-friendly format


update log:

  • 10-10-12: Two tweaks to the process_classes function; 1, a file with the same name as the folder containing it is now parsed with rewrite_doc_lines instead of rewrite_function_doc_lines, and 2: the "index" variable was declared outside of the foreach loop when it should have been declared inside of it. These should be resolved in both snippets and full source.
  • 10-11-12: Adjusted the head_comment variable to make the initial comment a non-doxygen comment to keep it from being included in generated documentation.

This section is both long and optional, as some may elect to keep their driver documentation in its native format and simply write their manual commands/system to locate and read this documentation in plain text. We elected to take advantage of Doxygen's ability to generate/track references to other code and incorporate this documentation directly. The code described in this post is also still imperfect and quickly changing as we identify issues with it.

The structure of this post is as follows:

  • Possible approaches
  • Implementation
    • Converting Pages
    • Converting Classes
    • Translating references
  • Notes/todo
  • Full source code

Possible approaches

Once we've decided that we do want to convert our LDMud driver docs into a Doxy-friendly format, there are two obvious paths we can take: manually convert the documents into Doxygen-friendly idioms in an accurate manner, or automate the conversion of the documents in a less-than-perfect but far faster approach. Within this second method, there are two main methods of implementing this--attempt to identify the document structure and reliably parse it out and cobble it back together in a Doxygen-friendly format, or attempt to identify as little of the document structure as we can, and intervene as little as possible in restructuring it.

For now we've decided to go with the latter-latter approach--generating doxy-friendly documents with as little intervention as possible. While we like the idea of rewriting the documents in a Doxygen-friendly format and donating them to the community, one of the concerns is that future changes to the original documentation will cause a divergence or drift in the documentation copies unless either both are maintained, or we convince the LDMud crew to adopt the doxy-format.

Implementation of a lazy conversion approach

In our lazy conversion approach we're going to split out our existing driver documentation into two types, which we'll call "pages" and "classes" as this is the format they're going to end up in for Doxygen--in the terms of the driver we'll be converting documentation for functions into "fake" classes, and we'll be converting documentation for concepts into pages. Before we get into specifics, we'll set up some defines:

1#include <strings.h>
2#include <regexp.h>
3#define GUIDE "/obj/doxy_guide/"
4#define START_ROOT "/doc/driver/"
5#define END_ROOT GUIDE+"driver/"

Converting Pages

We'll start with pages since the logic here is simplest. We'll define an array of string folder names, in our case this looks like: string *page_folders = ({"concepts", "driver", "hook", "lpc", "obsolete"});. We'll use this array to define the list of folders we'll parse into pages with our process_pages function. We'll take a closer look at the scaffolding/logistics here, first.

Logistics

The conversion functions themselves aren't horribly complex, but we do have to read, convert and write several hundred doc files in the process--so here's the function we'll use to process our pages/concepts into doxy/markdown pages.

 1void process_pages(string driver_index) //#1
 2{
 3    /*
 4    1. start with a list of valid directories and prepare to write an .md file for each
 5    2. if there is a file with the directory name in it, it becomes the content of this file
 6    3. take each additional sub file and write an .md for it, convert headers/references, then insert a subpage link in index file
 7    */
 8    foreach(string folder : page_folders)
 9    {
10        driver_index += sprintf("- @subpage driver_%s\n", folder); //#2
11        mapping files = map(mkmapping(get_dir(START_ROOT+folder+"/")), (:explode(read_file(START_ROOT+folder+"/"+$1), "\n"):)); //#3
12        string index = sprintf("%s {#driver_%s}\n=======\n", folder, folder); //#4
13        if(files[folder]) //#5
14        {
15            index += rewrite_doc_lines(files[folder])+"\n";
16        }
17        foreach(string file : get_dir(START_ROOT+folder+"/")-({folder})) //#6
18        {
19            index += sprintf("- @subpage driver_%s_%s\n", folder, file);
20            write_file(END_ROOT+folder+"/"+file+".md", sprintf("%s {#driver_%s_%s}\n=======\n", file, folder, file)+rewrite_doc_lines(files[file]), 1); //#7
21        }
22        write_file(END_ROOT+"driver_"+folder+".md", index, 1); //#8
23    }
24}

The main steps in our process will be iterating over this list, writing an index file, and running each page through a doc_converting function. Because the number of files we'll be working with is fairly long and we generate docs for both our pages/classes at the same time, we have some risk of overshooting our eval limit of just shy of 5m. If you'd like to intervene more in the process you can perform more assignments than I do in the code above and process your pages/classes in separate calls. Below are explanations of the code notes:

  1. We're going to pass this string by reference when we call, because both our class and page processing functions need to contribute to the index before we actually "write" it.
  2. We're going to define each of our individual concept files and section files as "subpages" of each other, allowing us to re-create directory hierarchy in Doxygen; in this line we'll write subpage links into driver.md for each of the chapter indexes. They end up looking like driver -> chapter -> topic
  3. This one's a bit complicated, but it helps save eval time on some assignments. It makes a mapping like ([file_name: file_contents]) by getting the contents of our folder, turning it into a mapping, and then using the map() efun to read in each file and convert it into an array of strings exploded on line-breaks.
  4. Start our chapter-specific index as well.
  5. If one of the files in this folder has the same name as the folder, it's following a documentation convention for explaining what the purpose of that folder is--so if there's one present, we're going to convert it and write the result at the top of the chapter's index.
  6. Make sure to take the folder name out of our file list. We're just grabbing the list again because this is quicker than running m_indices (and re-sorting it--they won't be ordered) on our file mapping.
  7. Write a file (in a new location) with an appropriate doxy/markdown title line, followed by the rewritten doc file. We're using mode 1 because we want to overwrite the existing files so we can use this to propegate updates if necessary.
  8. We wrap up by writing our chapter index files, the driver_index will be written in (in our case) in a function which calls both process_pages(driver_index_string) and process_classes(driver_index_string)

With the logistics scaffolding in place, it's time to get our hands dirty with the actual conversion function, rewrite_doc_lines.

Conversion

The real conversion work is handled by the string rewrite_doc_lines(string *lines) function, which we pass an array of lines for a single file at a time (See note #7 in the previous section), and which returns a modified string (not an array!) copy of the text we'll be writing to the file. I'll give you a copy of the code first, and then describe the workflow via notes.

Code

 1string rewrite_doc_lines(string *lines)
 2{
 3    string *found_order = ({}); //#1a
 4    mapping descs = ([]); //#1b
 5    string open_tag; //#1c
 6    string *desc = ({}); //#1d
 7    int trim_len = 0; //#1e
 8    foreach(string line : lines)
 9    {
10        if(!stringp(line)){continue;} // sanity check
11        else if(!sizeof(line)||line=="\n"){desc += ({""});} //#2a
12        else if(line[0] != ' ' && line == upper_case(line)) //#2b
13        {
14            if(open_tag)
15            {
16                descs[open_tag] = desc;
17                desc = ({});
18            }
19            open_tag = line;
20            found_order += ({open_tag});
21        }
22        else //#2c
23        {
24            if(trim_len)
25            {
26                desc += ({line[trim_len..<1]});
27            }
28            else
29            {
30                trim_len = sizeof(line) - sizeof(trim(line, 1));
31                desc += ({line[trim_len..<1]});
32            }
33        }
34    }
35    if(open_tag) //#3
36    {
37        descs[open_tag] = desc;
38    }
39    else
40    {
41        descs["DESCRIPTION"] = desc;
42    }
43    if(member(descs, "SEE ALSO"))
44    {
45        descs["SEE ALSO"] = map(descs["SEE ALSO"], #'convert_manual_style_reference);
46    } //#4
47
48        //#5
49    descs = map(descs, (:sprintf(section_formats[$1]||"## "+$1+" ##\n%s\n", implode($2, section_formats[$1, 1]||"\n")):)); //#5a
50    return implode(map(found_order-({"CONCEPT", "NAME"}), (:descs[$1]:)), "\n"); //#5b
51}

Process

  1. First weā€™re going to set up a few variables:

    1. string *found_order is for recording the order we found sections in the original document (weā€™re going to store them in a mapping and want to know what the order was).
    2. mapping descs will be used to store all of our associated sections in the format ([section name : array_of_section_lines])
    3. string open_tag stores just the name of the last section name we encountered.
    4. string *desc will be used to hold an array of description lines. Every time we encounter a new section, weā€™ll save the old one to our descs mapping and reset this variable to a blank array.
    5. int trim_len will hold an integer number for the number of characters we want to trim off the left side of the document. At first we simply trimmed 8 chars off the left margin of any line not beginning with text as this is the usual format for most of the driver docs, but there are a few documents (/driver/lpc/mappings, for example) which donā€™t follow this rubric. Our next stab was to just use the efun trim(line, TRIM_LEFT) to tidy the strings, but this caused problems with a few documents which rely on an ascii visual reproduction of information (i.e., /driver/lpc/modifiers)
  2. Now weā€™ll iterate over each line in the file and look for three main conditions (plus a safety/sanity check):

    1. if there is no text or only a line break, weā€™ll convert this to a simple blank string, "" in our new description array.
    2. if there is ALL UPPER CASE text on the current line starting in position 0 (this is how our section titles look in LDMudā€™s manpage format) weā€™re going to take this text and call it our tag name, save the previous desc to our mapping by tag name if there is one, and add the new tag to our found_order
    3. otherwise, weā€™ll assume that this line is actually part of the documentation. If weā€™ve already established a trim_len weā€™ll chop that number of characters off and store it in the desc array, and if not weā€™ll compare our current line length with the output of trim(line, TRIM_LEFT) to establish what the trim_len should be, chop that number off and then add it to the desc array.
  3. Once weā€™re out of the foreach loop, weā€™ll check if we have a final open tag and add it to the descs mapping if so. If we donā€™t have an open tag (we should always have one if this followed the format described in /driver/README), weā€™re going to save any text in our desc variable under the tag DESCRIPTION so we still get whatever content was in the file converted.

  4. If one of our keys in descs is "SEE ALSO", weā€™re going to pass each line assigned to that key separately through convert_manual_style_reference which Iā€™ll discuss in greater detail in the final section of the post.

  5. Once weā€™ve converted our references, weā€™re ready to prepare the descs mapping for printing; these two lines are fairly dense so Iā€™ll unpack them a bit.

    1. First weā€™re going to call sprintf() for each element in the descs array with arguments that need further describing:

      1. We have a mapping, section_formats, which contains keys matching the sections we expect to see, followed by one value which is a sprintf format string, and a sectiond value which will be used as the implode separator for each line in the array for that desc. Letā€™s take a look at what our current values are:

        1mapping section_formats = ([
        2    "SYNOPSIS": "@synopsis{\n%s\n}\n";"\n",
        3    "DESCRIPTION": "@verbatim\n%s\n@endverbatim\n";"\n",
        4    "SEE ALSO": "@see %s\n";" ",
        5    "EXAMPLES": "@par Usage:\n@code\n%s\n@endcode\n";"\n",
        6    "EXAMPLE": "@par Usage:\n@code\n%s\n@endcode\n";"\n",
        7    "HISTORY": "@history{\n%s\n}\n";"\n",
        8    "OVERLOADED": "@warning This efun is obscured %s and may not behave as documented.\n";"\n",
        9]);
        

      As you can see, the sprintf formats are used to insert the text of our description into Doxygen commands. You can safely ignore the ā€œOVERLOADEDā€ key for now, itā€™ll be discussed in greater detail when we move on to discussing class/function documentation. Itā€™s worth noting that the DESCRIPTION key simply wraps the text in verbatim/endverbatim commandsā€“this is far less complicated than attempting to parse every idiom in the documentation. If you want to make your converter less lazy than ours, youā€™ll probably want to write an additional conversion function which you pass all lines of your description through in order to attempt to identify LDMud idioms and parse them into Doxygenā€™s idioms. Note that aside from ā€œSEE ALSOā€ and ā€œOVERLOADEDā€, most of the formats rely directly or via aliases on @code/endcode to achieve a similar effect of preserving line breaks without taking the time to identify structure.

      If our section doesnā€™t exist in the map, it falls back on defaults of "## "+section_name+" ##\n%s\n" for the sprintf format and "\n" for the implode separator.

    2. Afterward, each key in descs will contain a single documentation string, but because the mapping isnā€™t order-specific, weā€™re going to use the map efun on the found_order array to construct, implode and then return the strings in order for writing. The main caveat here is that weā€™re knocking out the NAME/CONCEPT sections if they exist, as we can generate this information on our own.

Converting classes / Functions

Logistics

The class logistics function is fairly similar to the pages one, though we do make a special catch for ā€œefunā€ as weā€™d like to make a few caveats to the normal parsing flow for efuns. See the notes for more details on these.

 1void process_classes(string driver_index)
 2{
 3    foreach(string folder : class_folders)
 4    {
 5        string index = "";
 6        string outfile = END_ROOT+"driver_"+folder+".c";
 7        mapping files = map(mkmapping(get_dir("/doc/driver/"+folder+"/")-({"[]"})), (:explode(read_file("/doc/driver/"+folder+"/"+$1), "\n"):)); //#
 8        if(files[folder])
 9        {
10            index += rewrite_doc_lines(files[folder])+"\n";
11        }
12        if(folder == "efun")
13        {
14            driver_index += sprintf("- @subpage %s\n", folder);
15            outfile = END_ROOT+"efun.c"; //#
16            string *overloaded = functionlist("/obj/simul_efun.c") & map(m_indices(files), #'basename); //#
17            map(overloaded, (:files[$1] = fix_overloaded_efuns($1, files[$1]):)); //#
18        }
19        else
20        {
21            driver_index += sprintf("- @ref driver_%s\n", folder);
22        }
23        write_file(outfile, sprintf(head_comment, folder, index), 1); //#
24        foreach(string func, string *contents : filter(files-({folder}), (:!member((["DEPRECATED", "OBSOLETE"]), $2[0]):))) //#
25        {
26            //write the doc
27            write_file(outfile, write_doc(contents));
28            //then write a synopsis
29            write_file(outfile, sprintf("%s(){}\n\n", func)); //#
30        }
31    }
32}

The code notes arenā€™t as comprehensive this time, and focus on differences between this and process_pages:

  1. This is more or less the same as in process_pages, but here we take out the file ā€œ[]ā€(included in our efun docs...) for array_indexing.

  2. Weā€™re going to save our efun file as efun.c instead of driver_efun.c; while weā€™d like to preserve the ā€œmasterā€ and ā€œappliedā€ namespaces within doxygen, we want to make use of the ā€œefunā€ namespace.

  3. Weā€™re getting the list of functions defined by our simul_efun object, as these definitions will obscure an efun of the same name and intersecting it with our list of efuns to see which efuns are obscured.

  4. Weā€™re taking the list generated above and feeding them through a brief function, fix_overloaded_efuns(string function_name, string *lines) which adds two lines at the head of the array, one reading ā€œOVERLOADEDā€ and the other containing a statement saying that the function is overloaded by simul_efun::function_name.

  5. Here weā€™re writing a head_comment to the file. This is run through sprintf to insert the name of the class and is meant to provide some documentation since this object could confuse a wizard who finds it in the directory structure. This comment currently looks like:

    1string head_comment = "/* This file exists to help Doxygen parse %s documentation "
    2    "It has no bearing on how these functions actually work. It is generated, "
    3    "so there's no need to edit it.*/\n/**%s\n*/\n#include <files.h>\n";
    

    The #include statement is a bit of a kludge to ensure the LPC filter starts the class in the right place.

  6. We currently knock deprecated/obsolete efuns out of this documentation, though we haven't made a final decision yet on what to do with them.

  7. After we've written the comment, we're going to write a dummy function member for each function. We're going to keep this simple and not attempt to get the type or arguments correct (for now, at least--our desire is to get this up and running quickly).

Keep in mind that these steps are still in flux and while they reflect our priorities at the moment, there's probably room to improve these. Let us know what you come up with.

Conversion

In this instance, string rewrite_function_doc_lines(string *lines) is almost identical to rewrite_doc_lines with the primary exception of the return statement, which makes use of a slightly more controlled display order for the function information which maintains more or less what we'd like to see in most of our mudwide function documentation. You can look at the order array string in the final code detail for the order, but for our purposes here we'll just show the single line and how it follows this order first, and then appends all other sections in the order they were encountered in the documentation:

1return implode(map(order+(found_order-order), (:descs[$1]:)), "\n");

Translating references

Both rewrite_function_doc_lines and rewrite_doc_lines run the contents of the "SEE ALSO" section of each document through our reference translator, string convert_manual_style_reference(string reference) in order to convert the reference(s) to a Doxygen_friendly format. The converter is pretty simple--it just uses a switch statement and follows the chapter designations discussed in /driver/README. We explode the reference string by ", " just to make sure we process for multiple references if they're present, and perform a quick regex check to make sure the references conform to the format we anticipate (and to grab the chapter/topic indicated. I've included the code here with little discussion.

Code

 1string convert_manual_style_reference(string reference)
 2{
 3    string *out = ({});
 4    mixed *matches;
 5    foreach(string part : explode(reference, ", "))
 6    {
 7        matches = regmatch(part, "(\\w+)\\(([A-Z]+)\\)", RE_MATCH_SUBS);
 8        if(!matches)
 9        {
10            out += ({part});
11            continue;
12        }
13        switch(matches[2])
14        {
15            case "A":
16                if(matches[1] == "applied")
17                {
18                    out+= ({"driver_applied"});
19                }
20                else
21                {
22                    out+= ({"driver_applied."+matches[1]});
23                }
24                break;
25            case "C":
26                if(matches[1] == "concepts")
27                {
28                    out+= ({"@ref driver_concepts"});
29                }
30                else
31                {
32                    out+= ({"@ref driver_concepts_"+matches[1]});
33                }
34                break;
35            case "D":
36                if(matches[1] == "driver")
37                {
38                    out+= ({"@ref driver_driver"});
39                }
40                else
41                {
42                    out+= ({"@ref driver_driver_"+matches[1]});
43                }
44                break;
45            case "H":
46                if(matches[1] == "hook")
47                {
48                    out+= ({"driver_hook"});
49                }
50                else
51                {
52                    out+= ({"driver_hook_"+matches[1]});
53                }
54                break;
55            case "LPC":
56                if(matches[1] == "lpc")
57                {
58                    out+= ({"driver_lpc"});
59                }
60                else
61                {
62                    out+= ({"@ref driver_lpc_"+matches[1]});
63                }
64                break;
65            case "O":
66                if(matches[1] == "obsolete")
67                {
68                    out+= ({"driver_obsolete"});
69                }
70                else
71                {
72                    out+= ({"@ref driver_obsolete_"+matches[1]});
73                }
74                break;
75            case "E":
76                if(matches[1] == "efun")
77                {
78                    out+= ({"efun"});
79                }
80                else
81                {
82                    out+= ({"efun."+matches[1]});
83                }
84                break;
85            case "SE":
86                if(matches[1] == "simul_efun")
87                {
88                    out+= ({"simul_efun"});
89                }
90                else
91                {
92                    out+= ({"simul_efun."+matches[1]});
93                }
94                break;
95        }
96    }
97    return implode(out, ", ");
98}

Notes/todo

There are a few important caveats and places I think there's room to improve this conversion process by a good bit.

Caveats

  1. This object doesn't include logic to create the destination folders it needs. They only need to be made once, so we just made them manually. They're only needed for the "pages" subtype. In our case our docs are headed to /obj/doxy_guide/driver/ for now, so we had to make subfolders for concepts, driver, hook, lpc and obsolete.
  2. This may not work well with any custom documentation sections you've added locally.
  3. The use of the "verbatim" command introduced a "* " at the start of each line (inside doxygen). If you notice this before we've posted the section on parsing doxygen's output, you may want to trim them out of your documentation.
  4. This doesn't clean up any old files in the directory, it just overwrites them. If you generate a file and then make changes to ensure it isn't generated in future updates, you'll still need to manually remove it.
  5. We exclude docs which start with a single-line "OBSOLETE" or "DEPRECATED" from our function documentation to avoid encouraging their use, but we've yet to take any steps to see them documented in some more appropriate fashion.
  6. The potential of overwriting updates suggests that updates should be made to the original documentation and not to the generated copy.

Todo:

  1. Our handling of the master-object documentation could be better.
  2. docfiles with a period in the name don't work properly, and we haven't yet written in a catch to convert these to underscores (ex: driver/concepts/intermud.basic)
  3. We'd like to develop some amount of crosstalk between our manual command and this converter which allows for the automatic insertion of back-references to driver topics--links to all topics outside of the driver docs which also reference the driver docs. This would largely be for enabling efuns to link-back to simul_efuns which include them in their list of see-also references.
  4. It's a bit pie-in-the-sky, but ultimately we think it'd be nice to integrate our documentation with actual doxygen documentation of the driver source, and link our efuns to the driver-level efun source.
  5. Write in some ability to detect updates to relevant files and re-generate documentation only when they've changed (useless until our implementation is stable, however.)

Full source

  1#include <strings.h>
  2#include <regexp.h>
  3#define GUIDE "/obj/doxy_guide/"
  4#define START_ROOT "/doc/driver/"
  5#define END_ROOT GUIDE+"driver/"
  6
  7/*
  8We want to:
  9x. note all of our sefuns
 10x. go through /doc/driver/efun/* and write the documentation and a
 11    fake synopsis into OUTPUT_FILE, noting if the efun is obscured
 12    by an SEFUN of the same name.
 13*/
 14
 15/*start Prototypes*/
 16void process_all();
 17void process_classes();
 18void process_pages();
 19string rewrite_function_doc_lines(string *lines);
 20string rewrite_doc_lines(string *lines);
 21string write_doc(string *file_contents);
 22string *fix_overloaded_efuns(string func, string *file_contents);
 23string convert_manual_style_reference(string reference);
 24/*end prototypes*/
 25
 26
 27string head_comment = "/* This file exists to help Doxygen parse %s documentation "
 28    "It has no bearing on how these functions actually work. It is generated, "
 29    "so there's no need to edit it.*/\n/**%s\n*/\n#include <files.h>\n";
 30string *class_folders = ({"applied", "efun", "master"});
 31string *page_folders = ({"concepts", "driver", "lpc", "obsolete", "hook"});
 32
 33string *order = ({"DESCRIPTION","OVERLOADED","SYNOPSIS","EXAMPLES","EXAMPLE","HISTORY","SEE ALSO"});
 34mapping section_formats = ([
 35    "SYNOPSIS": "@synopsis{\n%s\n}\n";"\n",
 36    "DESCRIPTION": "@verbatim\n%s\n@endverbatim\n";"\n",
 37    "SEE ALSO": "@see %s\n";" ",
 38    "EXAMPLES": "@par Usage:\n@code\n%s\n@endcode\n";"\n",
 39    "EXAMPLE": "@par Usage:\n@code\n%s\n@endcode\n";"\n",
 40    "HISTORY": "@history{\n%s\n}\n";"\n",
 41    "OVERLOADED": "@warning This efun is obscured %s and may not behave as documented.\n";"\n",
 42    ]);
 43
 44void process_classes(string driver_index)
 45{
 46    foreach(string folder : class_folders)
 47    {
 48        string index = "";
 49        string outfile = END_ROOT+"driver_"+folder+".c";
 50        mapping files = map(mkmapping(get_dir("/doc/driver/"+folder+"/")-({"[]"})), (:explode(read_file("/doc/driver/"+folder+"/"+$1), "\n"):));
 51        if(files[folder])
 52        {
 53            index += rewrite_doc_lines(files[folder])+"\n";
 54        }
 55        if(folder == "efun")
 56        {
 57            driver_index += sprintf("- @subpage %s\n", folder);
 58            outfile = END_ROOT+"efun.c";
 59            string *overloaded = functionlist("/obj/simul_efun.c") & map(m_indices(files), #'basename);
 60            map(overloaded, (:files[$1] = fix_overloaded_efuns($1, files[$1]):));
 61        }
 62        else
 63        {
 64            driver_index += sprintf("- @ref driver_%s\n", folder);
 65        }
 66        write_file(outfile, sprintf(head_comment, folder, index), 1);
 67        foreach(string func, string *contents : filter(files-([folder]), (:!member((["DEPRECATED", "OBSOLETE"]), $2[0]):)))
 68        {
 69            //write the doc
 70            write_file(outfile, write_doc(contents));
 71            //then write a synopsis
 72            write_file(outfile, sprintf("%s(){}\n\n", func));
 73        }
 74    }
 75}
 76
 77
 78string rewrite_function_doc_lines(string *lines)
 79{
 80    string *found_order = ({});
 81    mapping descs = ([]);
 82    string open_tag;
 83    string *desc = ({});
 84    int trim_len = 0;
 85    foreach(string line : lines)
 86    {
 87        if(!stringp(line)){continue;}
 88        else if(!sizeof(line)||line=="\n"){desc += ({""});}
 89        else if(line[0] != ' ' && line == upper_case(line))
 90        {
 91            if(open_tag)
 92            {
 93                descs[open_tag] = desc;
 94                desc = ({});
 95            }
 96            open_tag = line;
 97            found_order += ({open_tag});
 98        }
 99        else
100        {
101            if(trim_len)
102            {
103                desc += ({line[trim_len..<1]});
104            }
105            else
106            {
107                trim_len = sizeof(line) - sizeof(trim(line, 1));
108                desc += ({line[trim_len..<1]});
109            }
110        }
111    }
112    if(open_tag)
113    {
114        descs[open_tag] = desc;
115    }
116    else
117    {
118        descs["DESCRIPTION"] = desc;
119    }
120    if(member(descs, "SEE ALSO")){descs["SEE ALSO"] = map(descs["SEE ALSO"], #'convert_manual_style_reference);}
121    descs = map(descs, (:sprintf(section_formats[$1]||"## "+$1+" ##\n%s\n", implode($2, section_formats[$1, 1]||"\n")):));
122    return implode(map(order+(found_order-order), (:descs[$1]:)), "\n");
123}
124
125string rewrite_doc_lines(string *lines)
126{
127    string *found_order = ({});
128    mapping descs = ([]);
129    string open_tag;
130    string *desc = ({});
131    int trim_len = 0;
132    foreach(string line : lines)
133    {
134        if(!stringp(line)){continue;}
135        else if(!sizeof(line)||line=="\n"){desc += ({""});}
136        else if(line[0] != ' ' && line == upper_case(line))
137        {
138            if(open_tag)
139            {
140                descs[open_tag] = desc;
141                desc = ({});
142            }
143            open_tag = line;
144            found_order += ({open_tag});
145        }
146        else
147        {
148            if(trim_len)
149            {
150                desc += ({line[trim_len..<1]});
151            }
152            else
153            {
154                trim_len = sizeof(line) - sizeof(trim(line, 1));
155                desc += ({line[trim_len..<1]});
156            }
157        }
158    }
159    if(open_tag)
160    {
161        descs[open_tag] = desc;
162    }
163    else
164    {
165        descs["DESCRIPTION"] = desc;
166    }
167    if(member(descs, "SEE ALSO")){descs["SEE ALSO"] = map(descs["SEE ALSO"], #'convert_manual_style_reference);}
168    descs = map(descs, (:sprintf(section_formats[$1]||"## "+$1+" ##\n%s\n", implode($2, section_formats[$1, 1]||"\n")):));
169    return implode(map(found_order-({"CONCEPT", "NAME"}), (:descs[$1]:)), "\n");
170}
171
172string write_doc(string *file_contents)
173{
174    return sprintf("/// %s\n\n", implode(explode(rewrite_function_doc_lines(file_contents), "\n"), "\n/// "));
175}
176
177string *fix_overloaded_efuns(string func, string *file_contents)
178{
179    return ({"OVERLOADED",sprintf("        by simul_efun::%s", func), ""})+file_contents;
180}
181
182void process_pages(string driver_index)
183{
184    /*
185    1. start with a list of valid directories and prepare to write an .md file for each
186    2. if there is a file with the directory name in it, it becomes the content of this file
187    3. take each additional sub file and write an .md for it, convert headers/references
188        a. insert subpage link in index file
189    */
190    foreach(string folder : page_folders)
191    {
192        driver_index += sprintf("- @subpage driver_%s\n", folder);
193        mapping files = map(mkmapping(get_dir(START_ROOT+folder+"/")), (:explode(read_file(START_ROOT+folder+"/"+$1), "\n"):));
194        string index = sprintf("%s {#driver_%s}\n=======\n", folder, folder);
195        if(files[folder])
196        {
197            index += rewrite_doc_lines(files[folder])+"\n";
198        }
199        foreach(string file : get_dir(START_ROOT+folder+"/")-({folder}))
200        {
201            index += sprintf("- @subpage driver_%s_%s\n", folder, file);
202            write_file(END_ROOT+folder+"/"+file+".md", sprintf("%s {#driver_%s_%s}\n=======\n", file, folder, file)+rewrite_doc_lines(files[file]), 1);
203        }
204        write_file(END_ROOT+"driver_"+folder+".md", index, 1);
205    }
206}
207
208void process_all()
209{
210    string driver_index = "Driver {#driver}\n======\n\n";
211    process_pages(&driver_index);
212    process_classes(&driver_index);
213    write_file("/obj/doxy_guide/driver.md", driver_index, 1);
214}
215
216/*
217A for applied
218C for concepts
219D for driver
220E for efun
221H for hook
222LPC for LPC
223M for master
224O for obsolete
225OTHER for other
226SE for sefun
227S for std
228format foo(X)
229translate to one of two forms: x.foo or driver_X_foo
230*/
231string convert_manual_style_reference(string reference)
232{
233    string *out = ({});
234    mixed *matches;
235    foreach(string part : explode(reference, ", "))
236    {
237        matches = regmatch(part, "(\\w+)\\(([A-Z]+)\\)", RE_MATCH_SUBS);
238        if(!matches)
239        {
240            out += ({part});
241            continue;
242        }
243        switch(matches[2])
244        {
245            case "A":
246                if(matches[1] == "applied")
247                {
248                    out+= ({"driver_applied"});
249                }
250                else
251                {
252                    out+= ({"driver_applied."+matches[1]});
253                }
254                break;
255            case "C":
256                if(matches[1] == "concepts")
257                {
258                    out+= ({"@ref driver_concepts"});
259                }
260                else
261                {
262                    out+= ({"@ref driver_concepts_"+matches[1]});
263                }
264                break;
265            case "D":
266                if(matches[1] == "driver")
267                {
268                    out+= ({"@ref driver_driver"});
269                }
270                else
271                {
272                    out+= ({"@ref driver_driver_"+matches[1]});
273                }
274                break;
275            case "H":
276                if(matches[1] == "hook")
277                {
278                    out+= ({"driver_hook"});
279                }
280                else
281                {
282                    out+= ({"driver_hook_"+matches[1]});
283                }
284                break;
285            case "LPC":
286                if(matches[1] == "lpc")
287                {
288                    out+= ({"driver_lpc"});
289                }
290                else
291                {
292                    out+= ({"@ref driver_lpc_"+matches[1]});
293                }
294                break;
295            case "O":
296                if(matches[1] == "obsolete")
297                {
298                    out+= ({"driver_obsolete"});
299                }
300                else
301                {
302                    out+= ({"@ref driver_obsolete_"+matches[1]});
303                }
304                break;
305            case "E":
306                if(matches[1] == "efun")
307                {
308                    out+= ({"efun"});
309                }
310                else
311                {
312                    out+= ({"efun."+matches[1]});
313                }
314                break;
315            case "SE":
316                if(matches[1] == "simul_efun")
317                {
318                    out+= ({"simul_efun"});
319                }
320                else
321                {
322                    out+= ({"simul_efun."+matches[1]});
323                }
324                break;
325        }
326    }
327    return implode(out, ", ");
328}

Let me know if you have questions, comments or suggestions.

Discussing this elsewhere?
Enter 'link' for a markdown link
or 'tweet <message>' for a pre-populated Tweet :)
Want to subscribe? Enter 'rss' for a feed URL.
>