The Weaver

To weave a portion of the code into a tree of rendering instructions.

§1. The Master Weaver. Here's what has happened so far, on a weave. The web was read completely into memory and fully parsed. A request was then made either to swarm a mass of individual weaves, or to make just a single weave, with the target in each case being identified by its range. A further decoding layer then translated each range into rather more basic details of what to weave and where to put the result: and so we arrive at the front door of the routine Weaver::weave below.

The basic idea is to build a new tree of rendering instructions, and then to hand it over for rendering. A fair critique of this approach would be that, because LP source is now stored in a semantically detailed tree already, there's no real need to manufacture another slightly different one. This came about for historical reasons, i.e., because LP source used to be stored less comprehensibly. But it turns out to be convenient, and it's well-understood code which works well enough in practice.

void Weaver::weave(weave_order *wv) {
    WeavingFormats::render(wv, Weaver::weave_tree(wv));
}

§2. The tree is independent of format except that, to save passing arguments endlessly around, it contains a marker in the root of its head node which has a pointer to the weave order wv, and this does include a note of the format. But we make no use of that.

heterogeneous_tree *Weaver::weave_tree(weave_order *wv) {
    wv->paragraphs_woven = 0;
    heterogeneous_tree *tree = WeaveTree::new_tree(wv, Swarm::footnotes_appear(wv));
    TEMPORARY_TEXT(banner)
    WRITE_TO(banner, "Weave of '%S' generated by %s", wv->booklet_title, PROGRAM_NAME);
    tree_node *head = WeaveTree::head(tree, banner);
    DISCARD_TEXT(banner)
    tree_node *body = WeaveTree::body(tree);
    tree_node *tail = WeaveTree::tail(tree, I"End of weave");
    Trees::make_child(head, tree->root);
    Trees::make_child(body, tree->root);
    Trees::make_child(tail, tree->root);
    Weave body2.1;
    Trees::prune_tree(tree, &Weaver::prune_empty_material_nodes, NULL);
    return tree;
}

int Weaver::prune_empty_material_nodes(tree_node *N, void *state) {
    if ((N->type->required_CLASS == weave_material_node_CLASS) && (N->child == NULL))
        return TRUE;
    return FALSE;
}

§2.1. The body of the tree contains everything matching the theme (if set) contained in sections within the range of the weave.

Note that the tree only contains a chapter node if at least one of the sections for that chapter is in range, and that only those sections in range will have section nodes beneath it. So, for example, if the range picks out just a single section S, which is in chapter C, and there is no theme, then the top of the tree will look like this:

root (document node)
	head node
	body node
		chapter node for C
			chapter header node
			section node for S
				section header node
				...
				section footer node
			chapter footer node
	tail node

If, however, the theme X picks out only a little material from section S4 of chapter C2 and section S17 of chapter C6, we would have:

root (document node)
	head node
	body node
		chapter node for C2
			section node for S4
				...
		chapter node for C6
			section node for S17
				...
	tail node

Weave body2.1 =

    ls_web *W = wv->weave_web;
    weaver_state state_at; weaver_state *state = &state_at;
    Start the weaver with a clean slate2.1.2;
    ls_chapter *C, *contributing_chapter = NULL;
    ls_section *S;
    LOOP_OVER_LINKED_LIST(C, ls_chapter, W->chapters)
        if (C->imported == FALSE) {
            LOOP_OVER_LINKED_LIST(S, ls_section, C->sections)
                if (WebRanges::is_within(WebRanges::of(S), wv->weave_range)) {
                    Weave any necessary chapter node2.1.3;
                    Weave a section node2.1.5;
                    LanguageMethods::begin_weave(S, wv);
                    Weave this section2.1.7;
                    Weave any necessary section footer2.1.6;
                }
        }
    Weave any necessary chapter footer2.1.4;

This code is used in §2.

§2.1.1. The "weaver state" keeps track of some positional markers as we build the tree, but contains nothing else. Each position should be set to the current instance being created, and should be nulled when that is complete.

classdef weaver_state {
    struct tree_node *chapter_node;
    struct tree_node *section_node;
    struct tree_node *para_node;
    struct tree_node *carousel_node;
    struct tree_node *material_node;
} weaver_state;

The structure weaver_state is private to this section.

§2.1.2. Start the weaver with a clean slate2.1.2 =

    state->chapter_node = NULL;
    state->section_node = NULL;
    state->para_node = NULL;
    state->carousel_node = NULL;
    state->material_node = NULL;

This code is used in §2.1.

§2.1.3. The children of the body node are the chapter nodes. Unless a theme is set, which filters content down to just paragraphs with a particular sort of content, then the chapter node has a chapter-header node as its first child, and a chapter-footer node as its last. All other children are section nodes.

Weave any necessary chapter node2.1.3 =

    if (C != contributing_chapter) {
        Weave any necessary chapter footer2.1.4;
        tree_node *CH = WeaveTree::chapter(tree, C);
        Trees::make_child(CH, body);
        state->chapter_node = CH;
        state->section_node = NULL;
        state->para_node = NULL;
        state->carousel_node = NULL;
        state->material_node = NULL;
        contributing_chapter = C;
        if (Str::len(wv->theme_match) == 0) {
            tree_node *H = WeaveTree::chapter_header(tree, C);
            Trees::make_child(H, state->chapter_node);
        }
    }

This code is used in §2.1.

§2.1.4. Weave any necessary chapter footer2.1.4 =

    if ((Str::len(wv->theme_match) == 0) && (contributing_chapter)) {
        tree_node *F = WeaveTree::chapter_footer(tree, contributing_chapter);
        Trees::make_child(F, state->chapter_node);
    }

This code is used in §2.1 and §2.1.3.

§2.1.5. Weave a section node2.1.5 =

    tree_node *SH = WeaveTree::section(tree, S);
    Trees::make_child(SH, state->chapter_node);
    state->section_node = SH;
    state->para_node = NULL;
    state->carousel_node = NULL;
    state->material_node = NULL;
    if (Str::len(wv->theme_match) == 0) {
        tree_node *H = WeaveTree::section_header(tree, S);
        Trees::make_child(H, state->section_node);
    }

This code is used in §2.1.

§2.1.6. Weave any necessary section footer2.1.6 =

    if (Str::len(wv->theme_match) == 0) {
        tree_node *F = WeaveTree::section_footer(tree, S);
        Trees::make_child(F, state->section_node);
    }

This code is used in §2.1.

§2.1.7. Weaving a section. Weave this section2.1.7 =

    if (Str::len(wv->theme_match) == 0) {
        if (LiterateSource::unit_has_purpose(S->literate_source)) Weave the purpose2.1.7.1;
        Weave the table of contents2.1.7.2;
    }
    int thematic_subheading_placed = FALSE;
    for (ls_paragraph *par = S->literate_source->first_par; par; par = par->next_par)
        if (ParagraphTags::is_tagged_with(par, wv->theme_match)) {
            if (Str::len(wv->theme_match) > 0) Subheadings for thematic extracts2.1.7.3;
            Weave this paragraph2.1.7.5;
        }

This code is used in §2.1.

§2.1.7.1. Weave the purpose2.1.7.1 =

    text_stream *purp = Str::duplicate(LiterateSource::unit_purpose(S->literate_source));
    Trees::make_child(WeaveTree::purpose(tree, purp), state->section_node);

This code is used in §2.1.7.

§2.1.7.2. Weave the table of contents2.1.7.2 =

    int noteworthy = 0;
    for (ls_paragraph *par = S->literate_source->first_par; par; par = par->next_par)
        if (Str::len(LiterateSource::par_title(par)) > 0)
            if (ParagraphTags::is_tagged_with(par, wv->theme_match))
                noteworthy++;
    if (noteworthy > 0) {
        tree_node *TOC = WeaveTree::table_of_contents(tree, WebRanges::of(S));
        Trees::make_child(TOC, state->section_node);
        for (ls_paragraph *par = S->literate_source->first_par; par; par = par->next_par)
            if (Str::len(LiterateSource::par_title(par)) > 0)
                if (ParagraphTags::is_tagged_with(par, wv->theme_match)) {
                    TEMPORARY_TEXT(loc)
                    WRITE_TO(loc, "%S%S", LiterateSource::par_ornament(par), par->paragraph_number);
                    Trees::make_child(
                        WeaveTree::contents_line(tree, loc,
                            par->titling.operand1, par), TOC);
                    DISCARD_TEXT(loc)
                }
    }

This code is used in §2.1.7.

§2.1.7.3. Subheadings for thematic extracts2.1.7.3 =

    if (thematic_subheading_placed == FALSE) {
        if (W->is_page == FALSE) {
            TEMPORARY_TEXT(extr)
            WRITE_TO(extr, "From %S: %S", C->ch_title, S->sect_title);
            tree_node *D = WeaveTree::subheading(tree, extr);
            Trees::make_child(D, state->section_node);
            DISCARD_TEXT(extr)
        } else {
            TEMPORARY_TEXT(extr)
            WRITE_TO(extr, "Paragraphs tagged %S", wv->theme_match);
            tree_node *D = WeaveTree::subheading(tree, extr);
            Trees::make_child(D, state->section_node);
            DISCARD_TEXT(extr)
        }
        thematic_subheading_placed = TRUE;
    }
    text_stream *cap = ParagraphTags::retrieve_caption(par, wv->theme_match);
    if (Str::len(cap) > 0) {
        tree_node *D = WeaveTree::subsubheading(tree, cap);
        Trees::make_child(D, state->section_node);
    }

This code is used in §2.1.7.

§2.1.7.4. Weaving a paragraph. At this point we need to make a small diversion. Paragraph content is largely placed under "material nodes", which are convenient hold-alls for material which shares the same general type. state->material_node is set to the current material node (if one is being built at present), and the following function guarantees that this will be the case. Note that we use an existing node if it has the right type, or start a new node with the right type if not.

We will sometimes generate a material node which has no children, and thus no content, but this doesn't matter because the tree is pruned to remove empty material nodes before it is used.

enumerate COMMENTARY_MATERIAL 1
enumerate HOLON_DECLARATION_MATERIAL /* when a holon is being defined */ 
enumerate DEFINITION_MATERIAL /* when a definition is being made */ 
enumerate CODE_MATERIAL /* code-like material, such as in an extract */ 
enumerate ENDNOTES_MATERIAL /* endnotes at the foot of a paragraph */ 
enumerate FOOTNOTES_MATERIAL /* footnote texts for a paragraph */

tree_node *Weaver::ensure_material(heterogeneous_tree *tree,
    weaver_state *state, int new_material, int plainly,
    ls_chunk *chunk, ls_section *S) {
    int current_material = -1;
    if (state->material_node) {
        weave_material_node *C =
            RETRIEVE_POINTER_weave_material_node(state->material_node->content);
        current_material = C->material_type;
    }
    if (current_material != new_material) {
        programming_language *pl = NULL;
        text_stream *note = NULL;
        if ((new_material == CODE_MATERIAL) || (new_material == DEFINITION_MATERIAL)) {
            if (chunk) pl = chunk->extract_language;
            if (pl == NULL) pl = WebStructure::section_language(S);
            if (new_material != CODE_MATERIAL) pl = NULL;
            if ((S) && (S->owning_chapter) && (S->owning_chapter->owning_web) && (chunk)) {
                ls_line *lst = chunk->first_line;
                ls_web *W = S->owning_chapter->owning_web;
                if (ParagraphTags::is_tagged_with(LiterateSource::par_of_line(lst), I"Preform")) {
                    programming_language *prepl = Languages::find(W, I"Preform");
                    if (prepl) pl = prepl;
                }
            }
            if ((chunk) && (Str::len(chunk->extract_to) > 0)) {
                note = Str::new();
                WRITE_TO(note, "This is part of the extract file %S.", chunk->extract_to);
            }
        }
        tree_node *D = WeaveTree::material(tree, new_material, plainly, pl, note);
        if (state->carousel_node) Trees::make_child(D, state->carousel_node);
        else if (state->para_node) Trees::make_child(D, state->para_node);
        else internal_error("nowhere to place material node");
        state->material_node = D;
    }
    return state->material_node;
}

§2.1.7.5. The count of paragraphs woven is kept in order to be able to warn the user if the weave turned out to be empty: for instance, if it was filtered on a theme such as "all paragraphs with pictures", and in fact there weren't any.

Weave this paragraph2.1.7.5 =

    wv->paragraphs_woven++;
    state->material_node = NULL; /* this should be unnecessary, but just in case */
    int slide_no = -1, slide_of = -1;
    Paragraph header2.1.7.5.1;
    for (ls_chunk *chunk = par->first_chunk; chunk; chunk = chunk->next_chunk)
        Weave this chunk2.1.7.5.3;
    Paragraph endnotes2.1.7.5.2;
    state->material_node = NULL;

This code is used in §2.1.7.

§2.1.7.5.1. Paragraph header2.1.7.5.1 =

    LanguageMethods::reset_syntax_colouring(WebStructure::section_language(S));
    state->para_node = WeaveTree::paragraph_heading(tree, par, FALSE);
    Trees::make_child(state->para_node, state->section_node);

This code is used in §2.1.7.5.

§2.1.7.5.2. Many paragraphs do not have endnotes: if so, their endnotes material nodes will be empty.

Paragraph endnotes2.1.7.5.2 =

    tree_node *ends = Weaver::ensure_material(tree, state, ENDNOTES_MATERIAL, FALSE, NULL, S);
    if (Str::eq_insensitive(LiterateSource::par_title(par), I"Index")) {
        Trees::make_child(WeaveTree::index_begins(tree, par), ends);
        ls_index *index = wv->weave_web->index;
        if (index->lemmas_sorted == NULL) WebIndexing::sort(index, I"0");
        if (index->lemmas_sorted)
            for (int i=0; i<(int) (index->no_lemmas_sorted); i++) {
                ls_index_lemma *lemma = index->lemmas_sorted[i];
                Trees::make_child(WeaveTree::index_lemma(tree, par, lemma), ends);
            }
        Trees::make_child(WeaveTree::index_ends(tree, par), ends);
    }
    Weaver::weave_endnotes(tree, wv, ends, par);

This code is used in §2.1.7.5.

§2.1.7.5.3. Weaving a chunk. The perhaps unexpected logic here is because whether or not a chunk has a holon is not fully determined by its chunk type. Some EXTRACT_LSCT chunks do (these represent nameless holons), while others do not (these represent extracts of code which do not form part of a web's program).

Weave this chunk2.1.7.5.3 =

    if (chunk->holon) Deal with a holon2.1.7.5.3.1
    else switch (chunk->chunk_type) {
        case EXTRACT_LSCT: Deal with an extract2.1.7.5.3.2; break;
        case DEFINITION_LSCT: Deal with a definition2.1.7.5.3.3; break;

        case COMMENTARY_LSCT: Deal with a Markdown commentary chunk2.1.7.5.3.4; break;

        case INSERTION_LSCT: Deal with an insertion2.1.7.5.3.5; break;

        case HOLON_DECLARATION_LSCT:
        case HOLON_ADDENDUM_LSCT:
        case HOLON_FILE_LSCT:
        case HOLON_FILE_ADDENDUM_LSCT:
            break; /* these all have a holon and should be dealt with above */

        case DEFINITIONS_HERE_LSCT: break;
        case CLASSES_HERE_LSCT: break;

        case OTHER_LSCT: Deal with some other sort of chunk2.1.7.5.3.7; break;

        default: internal_error("unknown chunk type in weave");
    }

This code is used in §2.1.7.5.

§2.1.7.5.3.1. A holon is a declaration node (if it has a name) followed by a code-material node having potentially numerous children.

Deal with a holon2.1.7.5.3.1 =

    if (Str::len(chunk->holon->holon_name) > 0) {
        tree_node *hm = Weaver::ensure_material(tree, state, HOLON_DECLARATION_MATERIAL,
            FALSE, chunk, S);
        tree_node *decl = WeaveTree::holon_declaration(tree, chunk->holon,
            WebNotation::commentary_variation(wv->weave_web));
        Trees::make_child(decl, hm);
    }

    tree_node *cm = Weaver::ensure_material(tree, state, CODE_MATERIAL, FALSE, chunk, S);
    Weaver::weave_code_excerpt(tree, cm, chunk->code_excerpt, NULL, W, C, S, chunk,
        NULL, state, wv);

This code is used in §2.1.7.5.3.

§2.1.7.5.3.2. The chunk can also contain code-like matter in two other ways. Firstly, it might be an extract chunk containing code which is not part of the program, and therefore has no holon:

Deal with an extract2.1.7.5.3.2 =

    tree_node *cm = Weaver::ensure_material(tree, state, CODE_MATERIAL, chunk->plainer, chunk, S);
    programming_language *pl = chunk->extract_language;
    if (pl == NULL) pl = WebStructure::section_language(S);
    Weaver::weave_chunk_code(tree, cm, chunk, W, C, S, state, wv, pl);

This code is used in §2.1.7.5.3.

§2.1.7.5.3.3. Secondly, the chunk might be declaring a constant or macro. Note that this falls under a DEFINITION_MATERIAL material node, not a CODE_MATERIAL node.

Deal with a definition2.1.7.5.3.3 =

    int suppress = FALSE;
    text_stream *prefatory = NULL;
    if (chunk->metadata.minor == DEFINE_COMMAND_MINLC) prefatory = I"define";
    else if (chunk->metadata.minor == ENUMERATE_COMMAND_MINLC) prefatory = I"enumerate";
    else if (chunk->metadata.minor == DEFAULT_COMMAND_MINLC) prefatory = I"default";
    else if (chunk->metadata.minor == FORMAT_COMMAND_MINLC) prefatory = I"format";
    else if (chunk->metadata.minor == SILENTLY_FORMAT_COMMAND_MINLC) suppress = TRUE;
    if (suppress == FALSE) {
        tree_node *dm = Weaver::ensure_material(tree, state, DEFINITION_MATERIAL,
            chunk->plainer, chunk, S);
        tree_node *CL = WeaveTree::code_line(tree, chunk->first_line);
        Trees::make_child(CL, dm);
        if (Str::len(prefatory) > 0)
            Trees::make_child(
                WeaveTree::weave_defn_node(tree, prefatory, chunk->symbol_defined), CL);
        Weaver::weave_code_excerpt(tree, dm, chunk->code_excerpt, CL, W, C, S,
            chunk, chunk->first_line, state, wv);
    }

This code is used in §2.1.7.5.3.

§2.1.7.5.3.4. Markdown commentary can be very structurally complex, but in the weave tree, a single node represents each stretch of Markdown.

Deal with a Markdown commentary chunk2.1.7.5.3.4 =

    if (chunk->as_markdown == NULL) internal_error("no Markdown commentary");
    tree_node *cm = Weaver::ensure_material(tree, state, COMMENTARY_MATERIAL, FALSE, chunk, S);
    tree_node *C = WeaveTree::Markdown_commentary(tree, chunk->as_markdown, chunk->first_line,
        WebNotation::commentary_variation(wv->weave_web));
    Trees::make_child(C, cm);

This code is used in §2.1.7.5.3.

§2.1.7.5.3.5. Deal with an insertion2.1.7.5.3.5 =

    ls_line *lst = chunk->first_line;
    if (lst) { /* which should always happen, but for the sake of error recovery */
        text_stream *o1 = lst->classification.operand1, *o2 = lst->classification.operand2;
        switch (lst->classification.minor) {
            case AUDIO_MINLC: Weave an audio clip2.1.7.5.3.5.1; break;
            case CAROUSEL_SLIDE_MINLC: Weave a carousel2.1.7.5.3.5.2; break;
            case CAROUSEL_END_MINLC: Weave a carousel end2.1.7.5.3.5.3; break;
            case DOWNLOAD_MINLC: Weave a download2.1.7.5.3.5.4; break;
            case EMBEDDED_AV_MINLC: Weave an embed2.1.7.5.3.5.5; break;
            case FIGURE_MINLC: Weave a figure2.1.7.5.3.5.6; break;
            case HTML_MINLC: Weave a raw HTML extract2.1.7.5.3.5.7; break;
            case VIDEO_MINLC: Weave a video clip2.1.7.5.3.5.8; break;
            default: internal_error("unimplemented insertion type in weave");
        }
    }

This code is used in §2.1.7.5.3.

§2.1.7.5.3.5.1. Weave an audio clip2.1.7.5.3.5.1 =

    tree_node *cm = Weaver::ensure_material(tree, state, COMMENTARY_MATERIAL, FALSE, chunk, S);
    int w, h;
    text_stream *figname = Weaver::parse_dimensions(o1, &w, &h);
    Trees::make_child(WeaveTree::audio(tree, figname, w), cm);

This code is used in §2.1.7.5.3.5.

§2.1.7.5.3.5.2. Weave a carousel2.1.7.5.3.5.2 =

    if (slide_of < 0) {
        slide_no = 0; slide_of = 0;
        ls_chunk *ch = chunk;
        while (ch) {
            if (ch->chunk_type == INSERTION_LSCT) {
                if (ch->first_line->classification.minor == CAROUSEL_SLIDE_MINLC) slide_of++;
                if (ch->first_line->classification.minor == CAROUSEL_END_MINLC) break;
            }
            ch = ch->next_chunk;
        }
    }
    slide_no++;
    tree_node *C = WeaveTree::carousel_slide(tree,
        chunk->metadata.operand1, chunk->carousel_caption_position, slide_no, slide_of);
    Trees::make_child(C, state->para_node);
    state->material_node = NULL;
    state->carousel_node = C;

This code is used in §2.1.7.5.3.5.

§2.1.7.5.3.5.3. Weave a carousel end2.1.7.5.3.5.3 =

    state->material_node = NULL;
    state->carousel_node = NULL;
    slide_no = -1;

This code is used in §2.1.7.5.3.5.

§2.1.7.5.3.5.4. Weave a download2.1.7.5.3.5.4 =

    tree_node *cm = Weaver::ensure_material(tree, state, COMMENTARY_MATERIAL, FALSE, chunk, S);
    Trees::make_child(WeaveTree::download(tree, o1, o2), cm);

This code is used in §2.1.7.5.3.5.

§2.1.7.5.3.5.5. Weave an embed2.1.7.5.3.5.5 =

    tree_node *cm = Weaver::ensure_material(tree, state, COMMENTARY_MATERIAL, FALSE, chunk, S);
    int w, h;
    text_stream *ID = Weaver::parse_dimensions(o2, &w, &h);
    Trees::make_child(WeaveTree::embed(tree, o1, ID, w, h), cm);

This code is used in §2.1.7.5.3.5.

§2.1.7.5.3.5.6. Weave a figure2.1.7.5.3.5.6 =

    tree_node *cm = Weaver::ensure_material(tree, state, COMMENTARY_MATERIAL, FALSE, chunk, S);
    int w, h;
    text_stream *figname = Weaver::parse_dimensions(o1, &w, &h);
    Trees::make_child(WeaveTree::figure(tree, figname, o2, w, h), cm);

This code is used in §2.1.7.5.3.5.

§2.1.7.5.3.5.7. Weave a raw HTML extract2.1.7.5.3.5.7 =

    tree_node *cm = Weaver::ensure_material(tree, state, COMMENTARY_MATERIAL, FALSE, chunk, S);
    Trees::make_child(WeaveTree::raw_HTML(tree, o1), cm);

This code is used in §2.1.7.5.3.5.

§2.1.7.5.3.5.8. Weave a video clip2.1.7.5.3.5.8 =

    tree_node *cm = Weaver::ensure_material(tree, state, COMMENTARY_MATERIAL, FALSE, chunk, S);
    int w, h;
    text_stream *figname = Weaver::parse_dimensions(o1, &w, &h);
    Trees::make_child(WeaveTree::video(tree, figname, w, h), cm);

This code is used in §2.1.7.5.3.5.

§2.1.7.5.3.6. It's possible, optionally, to specify width and height for some visual matter. This is the syntax used.

text_stream *Weaver::parse_dimensions(text_stream *item, int *w, int *h) {
    *w = -1; *h = -1;
    text_stream *use = item;
    match_results mr = Regexp::create_mr();
    if (Regexp::match(&mr, item, U"(%c+) at (%d+) by (%d+)")) {
        *w = Str::atoi(mr.exp[1], 0);
        *h = Str::atoi(mr.exp[2], 0);
        use = Str::duplicate(mr.exp[0]);
    } else if (Regexp::match(&mr, item, U"(%c+) at height (%d+)")) {
        *h = Str::atoi(mr.exp[1], 0);
        use = Str::duplicate(mr.exp[0]);
    } else if (Regexp::match(&mr, item, U"(%c+) at width (%d+)")) {
        *w = Str::atoi(mr.exp[1], 0);
        use = Str::duplicate(mr.exp[0]);
    } else if (Regexp::match(&mr, item, U"(%c+) at (%d+)cm by (%d+)cm")) {
        *w = INWEB_POINTS_PER_CM*Str::atoi(mr.exp[1], 0);
        *h = INWEB_POINTS_PER_CM*Str::atoi(mr.exp[2], 0);
        use = Str::duplicate(mr.exp[0]);
    } else if (Regexp::match(&mr, item, U"(%c+) at height (%d+)cm")) {
        *h = INWEB_POINTS_PER_CM*Str::atoi(mr.exp[1], 0);
        use = Str::duplicate(mr.exp[0]);
    } else if (Regexp::match(&mr, item, U"(%c+) at width (%d+)cm")) {
        *w = INWEB_POINTS_PER_CM*Str::atoi(mr.exp[1], 0);
        use = Str::duplicate(mr.exp[0]);
    }
    Regexp::dispose_of(&mr);
    return use;
}

§2.1.7.5.3.7. For now we will ignore these.

Deal with some other sort of chunk2.1.7.5.3.7 =

    PRINT("***\n");
    LiterateSource::write_chunk(STDOUT, chunk);
    for (ls_line *lst = chunk->first_line; lst; lst = lst->next_line) {
        if (LanguageMethods::skip_in_weaving(WebStructure::section_language(S), wv, lst) == FALSE) {
            ;
        }
    }

This code is used in §2.1.7.5.3.

§3. Weaving blocks of code. A block of code is a run of one or more whole lines of computer program or similar techy material, which are to appear in a block, probably indented and offset in some way.

As we have seen above, code blocks occur in holons, and in definitions, in which case their material is represented by a ls_code_excerpt object, and they are woven into the tree with Weaver::weave_code_excerpt; or else in extracts which aren't holons, and are then handled by Weaver::weave_chunk_code.

Note that, regardless of all this, code blocks can occur in still another way by being part of Markdown commentary. Those are not directly represented in the weave tree, and tend to be rendered differently.

§4. Chunks are easier, so we'll do those first.

For convenience, white space lines at the bottom of a chunk are removed, and white space lines within or at the top of a chunk are replaced by entirely blank lines. (This can't change the meaning of the program, since chunked code blocks are used only for material which is not part of the program.)

So for example a code chunk like so:

this is line 1
    this is line 2, which is indented

would become:

code material node
	code line node
		source code node "this is line 1"
	code line node
		source code node "    this is line 2, which is indented"

Note that the indentation in the second line would be stored as 4 spaces, even if the original material used a tab: this is because of the use of Str::rectify_indentation.

void Weaver::weave_chunk_code(heterogeneous_tree *tree, tree_node *cm, ls_chunk *chunk,
    ls_web *W, ls_chapter *C, ls_section *S, weaver_state *state, weave_order *wv,
    programming_language *pl) {
    ls_line *line_break_pending_from = NULL;
    LanguageMethods::reset_syntax_colouring(pl);
    for (ls_line *lst = chunk->first_line; lst; lst = lst->next_line) {
        if (LanguageMethods::skip_in_weaving(pl, wv, lst) == FALSE) {
            TEMPORARY_TEXT(matter)
            Str::copy(matter, LiterateSource::line_weaving_matter(lst));
            if (line_break_pending_from) {
                tree_node *CL = WeaveTree::code_line(tree, line_break_pending_from);
                Trees::make_child(CL, cm);
                Weaver::source_code(tree, CL, W, C, S, wv, line_break_pending_from, I"", pl, FALSE);
                line_break_pending_from = NULL;
            }
            if (Regexp::string_is_white_space(matter)) {
                line_break_pending_from = lst;
            } else {
                tree_node *CL = WeaveTree::code_line(tree, lst);
                Trees::make_child(CL, cm);
                Str::rectify_indentation(matter, 4);
                Weaver::source_code(tree, CL, W, C, S, wv, lst, matter, pl, FALSE);
            }
            DISCARD_TEXT(matter)
        }
    }
}

§5. Code excerpts involve more work. They are part of the actual program in the web, and because of that we want to insert more markup into the code. Consider the following pretty awful code excerpt:

int x = 1; @<Possibly increment x@>; /* if we
feel like it, otherwise not */ printf("%d", x);

This becomes:

code line node
	source code node "int x = 1; "
	holon usage node "Possibly increment x"
	comment node " if we" with start "/*"
code line node
	comment node "feel like it, otherwise not " with end "*/"
	source code node " printf("%d", x);"

Happily the fragmentation has already been done, so we just need to reflect that in the nodes created:

void Weaver::weave_code_excerpt(heterogeneous_tree *tree, tree_node *ap,
    ls_code_excerpt *ex, tree_node *CL, ls_web *W, ls_chapter *C, ls_section *S,
    ls_chunk *chunk, ls_line *current_lst, weaver_state *state, weave_order *wv) {
    programming_language *pl = WebStructure::section_language(S);
    LanguageMethods::reset_syntax_colouring(pl);
    holon_splice *hs;
    LOOP_OVER_CODE_EXCERPT(hs, ex) {
        ls_line *lst = hs->line;
        if (LanguageMethods::skip_in_weaving(pl, wv, lst) == FALSE) {
            if (current_lst != lst) {
                current_lst = lst;
                CL = WeaveTree::code_line(tree, lst);
                Trees::make_child(CL, ap);
            }
            Weave a splice5.1;
        }
    }
}

§5.1. Weave a splice5.1 =

    switch (hs->type) {
        case CODE_LSHST: Deal with a splice of raw content5.1.4; break;
        case EXPANSION_LSHST: Deal with a named holon usage5.1.1; break;
        case COMMAND_LSHST: Deal with a tangler command5.1.2; break;
        case VERBATIM_LSHST: Deal with a verbatim5.1.3; break;
        case COMMENT_LSHST: Deal with a comment5.1.5; break;
        default: internal_error("unimplemented splice type");
    }

This code is used in §5.

§5.1.1. Deal with a named holon usage5.1.1 =

    Trees::make_child(WeaveTree::holon_usage(tree, hs->expansion,
        WebNotation::commentary_variation(wv->weave_web)), CL);

This code is used in §5.1.

§5.1.2. Deal with a tangler command5.1.2 =

    Trees::make_child(WeaveTree::tangler_command(tree, hs->texts[0]), CL);

This code is used in §5.1.

§5.1.3. Deal with a verbatim5.1.3 =

    Trees::make_child(WeaveTree::verbatim(tree, hs->texts[0]), CL);

This code is used in §5.1.

§5.1.4. Deal with a splice of raw content5.1.4 =

    TEMPORARY_TEXT(matter)
    Str::copy(matter, hs->texts[0]);
    Str::rectify_indentation(matter, 4);
    Weaver::source_code(tree, CL, W, C, S, wv, lst, matter, pl, TRUE);
    DISCARD_TEXT(matter)

This code is used in §5.1.

§5.1.5. Deal with a comment5.1.5 =

    if (hs->comment_as_markdown) {
        Trees::make_child(WeaveTree::comment_in_holon(tree, hs->texts[0], hs->texts[1], hs->texts[2],
            hs->comment_as_markdown, WebNotation::commentary_variation(wv->weave_web)), CL);
    } else {
        TEMPORARY_TEXT(matter)
        WRITE_TO(matter, "%S%S%S", hs->texts[1], hs->texts[0], hs->texts[2]);
        Str::rectify_indentation(matter, 4);
        Weaver::source_code(tree, CL, W, C, S, wv, lst, matter, pl, FALSE);
        DISCARD_TEXT(matter)
    }

This code is used in §5.1.

§6. In either of the above methods, source code nodes are made by the following.

void Weaver::source_code(heterogeneous_tree *tree, tree_node *ap, ls_web *W,
    ls_chapter *C, ls_section *S, weave_order *wv, ls_line *lst, text_stream *matter,
    programming_language *pl, int spot_function_uses) {
    TEMPORARY_TEXT(OUT)
    ls_line_analysis *L = (ls_line_analysis *) lst->analysis_ref;
    if (LanguageMethods::weave_code_line(OUT, pl, wv, W, C, S, L, matter, NULL)) {
        tree_node *V = WeaveTree::verbatim(tree, OUT);
        Trees::make_child(V, ap);
    } else {
        See to this ourselves6.1;
    }
    DISCARD_TEXT(OUT)
}

§6.1. See to this ourselves6.1 =

    TEMPORARY_TEXT(colouring)
    LanguageMethods::syntax_colour(pl, wv, lst, matter, colouring, Pathnames::path_to_inweb());
    Str::truncate(colouring, Str::len(matter));
    int from = 0;
    if (spot_function_uses)
        for (int i=0; i < Str::len(matter); i++) {
            if (Str::get_at(colouring, i) == FUNCTION_COLOUR) {
                TEMPORARY_TEXT(fname)
                int j = i;
                while (Str::get_at(colouring, j) == FUNCTION_COLOUR)
                    PUT_TO(fname, Str::get_at(matter, j++));
                if (CodeAnalysis::is_reserved_word_for_section(S, fname, FUNCTION_COLOUR))
                    Spot the function6.1.1;
                DISCARD_TEXT(fname)
            }

        }
    Weaver::source_code_piece(tree, ap, matter, colouring, from, Str::len(matter));
    DISCARD_TEXT(colouring)

This code is used in §6.

§6.1.1. So: if we have syntax-coloured a function name, and it turns out to be the name of a function we know, then we insert annotations accordingly: if it occurs in its own declaration then we replace it with a function definition node, and if not we place it with a function usage node. For example:

int factorial(n) {
	if (n == 1) return 1;
	return factorial(n-1);
}

becomes

code line node
	source code node "int "
	function defn node "factorial"
	source code node "(n) {"
code line node
	source code node "    if (n == 1) return 1;"
code line node
	source code node "    return "
	function usage node "factorial"
	source code node "(n-1);"
code line node
	source code node "}"

Note that this all happens only if we were called with spot_function_uses set, so it does not happen for mere extracts which are not part of the main program in the web.

Spot the function6.1.1 =

    language_function *fn = CodeAnalysis::get_function(S, fname, FUNCTION_COLOUR);
    if (fn) {
        ls_paragraph *par = Functions::declaration_lsparagraph(fn);
        if (lst == fn->function_header_at) {
            if (fn->usage_described == FALSE) {
                Weaver::source_code_piece(tree, ap, matter, colouring, from, i);
                int local_usages = 0, local_direction = 0, section_usages = 0, external_usages = 0;
                hash_table_entry_usage *hteu = NULL;
                Functions::count_usage(par, fn, &local_usages, &local_direction,
                    &section_usages, &external_usages, &hteu);
                tree_node *FD = WeaveTree::function_defn(tree, fn,
                    local_usages, local_direction, section_usages, external_usages, hteu);
                Trees::make_child(FD, ap);
                Weaver::show_function_usage(tree, wv, FD, par, fn, FALSE);
                i += Str::len(fname) - 1;
                from = i+1;
            }
        } else {
            Weaver::source_code_piece(tree, ap, matter, colouring, from, i);
            TEMPORARY_TEXT(url)
            Colonies::paragraph_URL(url, par, NULL, wv->weave_to, wv->weave_colony);
            WRITE_TO(url, "LN%d", fn->function_header_at->sequence_number_in_paragraph);
            tree_node *U = WeaveTree::function_usage(tree, url, fn);
            Trees::make_child(U, ap);
            i += Str::len(fname) - 1;
            from = i+1;
        }
    }

This code is used in §6.1.

§7. We don't want to leak tab characters out into woven code, where they are at the mercy of web browsers, which render tabs slightly oddly (and not to the width this author happens to like). So tabs are automatically converted to spaces sufficient to reach the next tab-stop position, calculated as:

define SPACES_PER_TAB_IN_WOVEN_CODE 4

void Weaver::source_code_piece(heterogeneous_tree *tree, tree_node *ap,
    text_stream *matter, text_stream *colouring, int from, int to) {
    if (to > from) {
        TEMPORARY_TEXT(m)
        TEMPORARY_TEXT(c)
        for (int i=from; i<to; i++) {
            inchar32_t ch = Str::get_at(matter, i);
            inchar32_t cc = Str::get_at(colouring, i);
            if (ch == '\t') {
                int spaces =
                    SPACES_PER_TAB_IN_WOVEN_CODE - (i % SPACES_PER_TAB_IN_WOVEN_CODE);
                for (int j=0; j<spaces; j++) {
                    PUT_TO(m, ' '); PUT_TO(c, cc);
                }
            } else {
                PUT_TO(m, ch); PUT_TO(c, cc);
            }
        }
        if (Str::len(c) != Str::len(m)) internal_error("bad source segment");
        tree_node *SC = WeaveTree::source_code(tree, m, c);
        Trees::make_child(SC, ap);
        DISCARD_TEXT(m)
        DISCARD_TEXT(c)
    }
}

§8. Endnotes. Paragraphs can optionally be given endnotes, which are automatically added rather than being free-form content by the author. We do so for five reasons:

void Weaver::weave_endnotes(heterogeneous_tree *tree, weave_order *wv, tree_node *ap,
    ls_paragraph *par) {
    tree_node *E = NULL;
    TEMPORARY_TEXT(NB)
    Show endnote on use of Preform8.1;
    Show endnote on conditional compilation affecting the paragraph8.2;
    Show endnote on where named holon is used8.3;
    Show endnotes on where any functions defined here are used8.4;
    Show endnotes on where any types defined here are used8.5;
    DISCARD_TEXT(NB)
}

§8.1. Show endnote on use of Preform8.1 =

    if (ParagraphTags::is_tagged_with(par, I"Preform"))
        WRITE_TO(NB, "This is Preform grammar, not regular C code.");
    Finish endnote8.1.1;

This code is used in §8.

§8.2. Show endnote on conditional compilation affecting the paragraph8.2 =

    IfdefTags::show_endnote_on_ifdefs(NB, par);
    Finish endnote8.1.1;

This code is used in §8.

§8.3. Show endnote on where named holon is used8.3 =

    if (LiterateSource::par_contains_named_holon(par)) {
        ls_section *PS = par->owning_unit->owning_section;
        WRITE_TO(NB, "This code is ");
        int at = LinkedLists::len(par->holon->addenda);
        if (at > 0) {
            WRITE_TO(NB, "continued in ");
            ls_holon *holon;
            int k = 0;
            LOOP_OVER_LINKED_LIST(holon, ls_holon, par->holon->addenda) {
                if (k > 0) {
                    if (k < at-1) WRITE_TO(NB, ", ");
                    else WRITE_TO(NB, " and ");
                }
                ls_paragraph *loc = holon->corresponding_chunk->owner;
                ls_line *finer = NULL;
                Insert a link node to paragraph loc8.3.1;
                k++;
            }
        }

        int ct = 0;
        holon_usage *mu;
        LOOP_OVER_LINKED_LIST(mu, holon_usage, par->holon->holon_usages) ct++;

        if (par->holon->addendum_to) {
            if (at > 0) WRITE_TO(NB, ", and ");
            WRITE_TO(NB, "a continuation of ");
            ls_paragraph *loc = par->holon->addendum_to->corresponding_chunk->owner;
            ls_line *finer = NULL;
            Insert a link node to paragraph loc8.3.1;
        } else if (ct == 0) {
            if (par->holon->file_form) {
                if (at > 0) WRITE_TO(NB, ", and ");
                WRITE_TO(NB, "written to an external file");
            } else if (par->holon->webwide == FALSE) {
                if (at > 0) WRITE_TO(NB, ", and ");
                WRITE_TO(NB, "never used");
            }
        } else {
            if (at > 0) WRITE_TO(NB, ", and ");
            int k = 0;
            holon_usage *mu;
            LOOP_OVER_LINKED_LIST(mu, holon_usage, par->holon->holon_usages)
                if (par != mu->used_in_paragraph) {
                    if (k++ == 0) {
                        WRITE_TO(NB, "used in ");
                    } else {
                        if (k < ct-1) WRITE_TO(NB, ", ");
                        else WRITE_TO(NB, " and ");
                    }
                    ls_paragraph *loc = mu->used_in_paragraph; ls_line *finer = NULL;
                    Insert a link node to paragraph loc8.3.1;
                    switch (mu->multiplicity) {
                        case 1: break;
                        case 2: WRITE_TO(NB, " (twice)"); break;
                        case 3: WRITE_TO(NB, " (three times)"); break;
                        case 4: WRITE_TO(NB, " (four times)"); break;
                        case 5: WRITE_TO(NB, " (five times)"); break;
                        default: WRITE_TO(NB, " (%d times)", mu->multiplicity); break;
                    }
                }
        }
        WRITE_TO(NB, ".");
    }
    Finish endnote8.1.1;

This code is used in §8.

§8.4. Show endnotes on where any functions defined here are used8.4 =

    language_function *fn;
    ls_paragraph_analysis *P = (ls_paragraph_analysis *) par->analysis_ref;
    LOOP_OVER_LINKED_LIST(fn, language_function, P->functions)
        if (fn->usage_described == FALSE)
            Weaver::show_function_usage(tree, wv, ap, par, fn, TRUE);

This code is used in §8.

§8.5. Show endnotes on where any types defined here are used8.5 =

    language_type *st;
    ls_paragraph_analysis *P = (ls_paragraph_analysis *) par->analysis_ref;
    LOOP_OVER_LINKED_LIST(st, language_type, P->structures) {
        WRITE_TO(NB, "The structure %S", st->structure_name);

        ls_section *S;
        LOOP_OVER(S, ls_section) S->scratch_flag = FALSE;
        structure_element *elt;
        LOOP_OVER_LINKED_LIST(elt, structure_element, st->elements) {
            hash_table_entry *hte =
                CodeAnalysis::find_hash_entry_for_section(elt->element_created_at,
                    elt->element_name, FALSE);
            if (hte) {
                hash_table_entry_usage *hteu;
                LOOP_OVER_LINKED_LIST(hteu, hash_table_entry_usage, hte->usages)
                    if (hteu->form_of_usage & ELEMENT_ACCESS_USAGE)
                        LiterateSource::section_of_par(hteu->usage_recorded_at)->scratch_flag = TRUE;
            }
        }

        int usage_count = 0, external = 0;
        LOOP_OVER(S, ls_section)
            if (S->scratch_flag) {
                usage_count++;
                if (S != LiterateSource::section_of_par(par)) external++;
            }
        if (external == 0) WRITE_TO(NB, " is private to this section");
        else {
            WRITE_TO(NB, " is accessed in ");
            int c = 0;
            LOOP_OVER(S, ls_section)
                if ((S->scratch_flag) && (S != LiterateSource::section_of_par(par))) {
                    if (c++ > 0) WRITE_TO(NB, ", ");
                    WRITE_TO(NB, "%S", WebRanges::of(S));
                }
            if (LiterateSource::section_of_par(par)->scratch_flag) WRITE_TO(NB, " and here");
        }
        WRITE_TO(NB, ".");
        Finish endnote8.1.1;
    }

This code is used in §8.

§9. Because this is needed not just in paragraph endnotes, but also when annotating code excerpts, it's extracted as a function. This is why the check for usage_described is made: if the weave has been able to work the function usage into an annotation, then it needn't give it as an endnote as well.

void Weaver::show_function_usage(heterogeneous_tree *tree, weave_order *wv,
    tree_node *ap, ls_paragraph *par, language_function *fn, int as_endnote) {
    tree_node *E = NULL;
    TEMPORARY_TEXT(NB)

    fn->usage_described = TRUE;
    hash_table_entry *hte =
        CodeAnalysis::find_hash_entry_for_section(fn->function_section,
            fn->function_name, FALSE);
    if (as_endnote) WRITE_TO(NB, "The function %S", fn->function_name);
    int used_flag = FALSE;
    hash_table_entry_usage *hteu = NULL;
    ls_section *last_cited_in = NULL;
    int count_under = 0;
    LOOP_OVER_LINKED_LIST(hteu, hash_table_entry_usage, hte->usages)
        if ((LiterateSource::section_of_par(par) ==
            LiterateSource::section_of_par(hteu->usage_recorded_at)) &&
            (hteu->finer_positioning != fn->function_header_at))
            Cite usage of function here9.1;
    LOOP_OVER_LINKED_LIST(hteu, hash_table_entry_usage, hte->usages)
        if (LiterateSource::section_of_par(par) !=
            LiterateSource::section_of_par(hteu->usage_recorded_at))
            Cite usage of function here9.1;
    if (used_flag == FALSE) {
        if (as_endnote) {
            WRITE_TO(NB, " appears nowhere else");
        } else {
            WRITE_TO(NB, "none");
        }
    }
    if (as_endnote) {
        if ((last_cited_in != LiterateSource::section_of_par(par)) && (last_cited_in))
            WRITE_TO(NB, ")");
        WRITE_TO(NB, ".");
    }
    Finish endnote8.1.1;
    DISCARD_TEXT(NB)
}

§9.1. Cite usage of function here9.1 =

    if ((as_endnote) && (used_flag == FALSE)) {
        WRITE_TO(NB, " is used in ");
        used_flag = TRUE;
    }
    ls_section *PS = LiterateSource::section_of_par(hteu->usage_recorded_at);
    if ((PS != last_cited_in) && (PS != LiterateSource::section_of_par(par))) {
        count_under = 0;
        if (last_cited_in) {
            if (as_endnote) {
                if (last_cited_in != LiterateSource::section_of_par(par))
                    WRITE_TO(NB, "), ");
                else WRITE_TO(NB, ", ");
            } else {
                WRITE_TO(NB, "\n");
            }
        }
        WRITE_TO(NB, "%S", PS->sect_title);
        if (as_endnote) WRITE_TO(NB, " ("); else WRITE_TO(NB, " - ");
    }
    if (count_under++ > 0) WRITE_TO(NB, ", ");
    ls_paragraph *loc = hteu->usage_recorded_at;
    ls_line *finer = hteu->finer_positioning;
    Insert a link node to paragraph loc8.3.1;
    last_cited_in = PS;

This code is used in §9 (twice).

§8.3.1. Lastly, then, the mechanism for spooling endnotes. Text accumulates in NB, and is only lazily emitted as endnote text nodes: that is, we minimise the number of such nodes needed. This also means that an endnote node E is only created if it will definitely be needed.

Insert a link node to paragraph loc8.3.1 =

    Flush the text so far, if any, into an endnote text node8.3.1.1;
    Guarantee an endnote node8.3.1.2;
    int distant = FALSE;
    if ((PS) && (loc->owning_unit->owning_section) &&
        (PS != loc->owning_unit->owning_section))
        distant = TRUE;
    Trees::make_child(WeaveTree::locale(tree, loc, finer, NULL, distant), E);

This code is used in §8.3 (three times) and §9.1.

§8.3.1.1. Flush the text so far, if any, into an endnote text node8.3.1.1 =

    if (Str::len(NB) > 0) {
        Guarantee an endnote node8.3.1.2;
        Trees::make_child(WeaveTree::endnote_text(tree, NB), E);
        Str::clear(NB);
    }

This code is used in §8.3.1 and §8.1.1.

§8.3.1.2. Guarantee an endnote node8.3.1.2 =

    if (E == NULL) {
        E = WeaveTree::endnote(tree); Trees::make_child(E, ap);
    }

This code is used in §8.3.1 and §8.3.1.1.

§8.1.1. Finish endnote8.1.1 =

    Flush the text so far, if any, into an endnote text node8.3.1.1;
    E = NULL;

This code is used in §8.1, §8.2, §8.3 and §8.5 and §9.