Streams

Home
foundation
Chapter 2: Memory, Streams and Collections
Streams

Support for writing structured textual output, perhaps to the screen, to a file, or to a flexible-sized wide string.

§1. About streams. The Inform tools produce textual output in many formats (HTML, EPS, plain text, Inform 6 code, XML, and so on), writing to a variety of files, and often need to juggle and rearrange partially written segments. These texts tend to be structured with running indentation, and may eventually need to be written to a disc file with either ISO Latin-1 or UTF-8 encodings, perhaps escaped for XML.

"Streams" are an abstraction to make it easy to handle all of this. The writer to a stream never needs to know where the text will come out, or how it should be indented or encoded.

Moreover, streams unify text files and strings, and can hold arbitrary Unicode text. This text is encoded internally as a sequence of 32-bit Unicode code points, in a form we might call semi-composed, in that all possible compositions of ISO Latin-1 characters are made: e.g., E plus acute accent is composed to a single code point as E-acute.

We give just one character value a non-Unicode meaning:

define NEWLINE_IN_STRING ((char) 0x7f)  Within quoted text, all newlines are converted to this

§2. The text_stream type began as a generalisation of the standard C library's FILE, and it is used in mostly similar ways. The user — the whole program outside of this section — deals only with text_stream * pointers to represent streams in use.

All stream handling is defined via macros. While many operations could be handled by ordinary functions, others cannot. text_stream cannot have exactly the semantics of FILE since we cannot rely on the host operating system to allocate and deallocate the structures behind text_stream * pointers; and we cannot use our own memory system, either, since we need stream handling to work both before the memory allocator starts and after it has finished. Our macros allow us to hide all this. Besides that, a macro approach makes it easier to retrofit new implementations as necessary. (The present implementation is the second stab at it.)

§3. We'll define a few variadic macros here, because there are awkward issues with code ordering if we leave them until later. They are written in the old-fashioned way, for compatibility with old copies of GCC, and avoid the need for comma deletion around empty tokens, as that is a point of incompatibility between implementations of the C preprocessor cpp. All the same, if you're porting this code, you may need to rewrite the macro with ... in place of args... in the header, and then __VA_ARGS__ in place of args in the definition: that being the modern way, apparently.

WRITE is essentially sprintf and fprintf combined, since it prints formatted text to the current stream, which could be either a string or a file. PRINT does the same but to STDOUT, and is thus essentially printf.

#define WRITE(args...) Writers::printf(OUT, args)

#define PRINT(args...) Writers::printf(STDOUT, args)

#define WRITE_TO(stream, args...) Writers::printf(stream, args)

#define LOG(args...) Writers::printf(DL, args)

#define LOGIF(aspect, args...) { \
    if (Log::aspect_switched_on(aspect##_DA)) Writers::printf(DL, args); \
}

§4. The main purpose of many functions is to write textual material to some file. Such functions almost always have a special argument in their prototypes: OUTPUT_STREAM. This tells them where to pipe their output, which is always to a "current stream" called OUT. What this leads to, and who will see that it's properly opened and closed, are not their concern.

define OUTPUT_STREAM text_stream *OUT  used only as a function prototype argument

§5. Three output streams are always open. One is NULL, that is, its value as a text_stream * pointer is NULL, the generic C null pointer. This represents an oubliette: it is entirely valid to use it, but output sent to NULL will never be seen again.

The others are STDOUT and STDERR. As the names suggest these are wrappers for stdout and stderr, the standard console output and error messages "files" provided by the C library.

We should always use PRINT(...) instead of printf(...) for console output, so that there are no uses of printf anywhere in the program.

define STDOUT Streams::get_stdout()
define STDERR Streams::get_stderr()

§6. PUT and PUT_TO similarly print single characters, which are specified as unsigned integer values. In practice, WRITE_TO and PUT_TO are seldom needed because there is almost always only one stream of interest at a time — OUT, the current stream.

define PUT(c) Streams::putc(c, OUT)
define PUT_TO(stream, c) Streams::putc(c, stream)

§7. Each stream has a current indentation level, initially 0. Lines of text will be indented by one tab stop for each level; it's an error for the level to become negative.

define INDENT Streams::indent(OUT);
define STREAM_INDENT(x) Streams::indent(x);
define OUTDENT Streams::outdent(OUT);
define STREAM_OUTDENT(x) Streams::outdent(x);
define SET_INDENT(N) Streams::set_indentation(OUT, N);

§8. Other streams only exist when explicitly created, or "opened". A function is only allowed to open a new stream if it can be proved that this stream will always subsequently be "closed". (Except for the possibility of the tool halting with an internal error, and therefore an exit(1), while the stream is still open.) A stream can be opened and closed only once, and outside that time its state is undefined: it must not be used at all.

The simplest way is to make a temporary stream, which can be used as a sort of clipboard. For instance, suppose we have to compile X before Y, but have to ensure Y comes before X in the eventual output. We create a temporary stream, compile X into it, then compile Y to OUT, then copy the temporary stream into OUT and dispose of it.

Temporary streams are always created in memory, held in C's local stack frame rather than allocated and freed via malloc and free. It must always be possible to prove that execution passes from TEMPORARY_TEXT to DISCARD_TEXT, unless the program makes a fatal exit in between. The stream, let's call it TEMP, exists only between those macros. We can legitimately create a temporary stream many times in one function (for instance inside a loop body) because each time TEMP is created as a new stream, overwriting the old one. TEMP is a different stream each time it is created, so it does not violate the rule that every stream is opened and closed once only.

define TEMPORARY_TEXT(T)
    inchar32_t T##_dest[2048];
    text_stream T##_stream_structure = Streams::new_buffer(2048, T##_dest);
    text_stream *T = &T##_stream_structure;
define DISCARD_TEXT(T)
    STREAM_CLOSE(T);

§9. Otherwise we can create new globally existing streams, provided we take on the responsibility for seeing that they are properly closed. There are two choices: a stream in memory, allocated via malloc and freed by free when the stream is closed; or a file written to disc, opened via fopen and later closed by fclose. Files are always written in text mode, that is, "w" not "wb", for those platforms where this makes a difference.

We use streams to handle all of our text file output, so there should be no calls to fprintf anywhere in the program except for binary files.

define STREAM_OPEN_TO_FILE(new, fn, enc) Streams::open_to_file(new, fn, enc)
define STREAM_OPEN_TO_FILE_APPEND(new, fn, enc) Streams::open_to_file_append(new, fn, enc)
define STREAM_OPEN_IN_MEMORY(new) Streams::open_to_memory(new, 20480)
define STREAM_CLOSE(stream) Streams::close(stream)

§10. The following operation is equivalent to fflush and makes it more likely (I put it no higher) that the text written to a stream has all actually been copied onto the disc, rather than sitting in some operating system buffer. This helps ensure that any debugging log is up to the minute, in case of a crash, but its absence wouldn't hurt our normal function. Flushing a memory stream is legal but does nothing.

define STREAM_FLUSH(stream) Streams::flush(stream)

§11. A piece of information we can read for any stream is the number of characters written to it: its "extent". In fact, UTF-8 multi-byte encoding schemes, together with differing platform interpretations of C's '\n', mean that this extent is not necessarily either the final file size in bytes or the final number of human-readable characters. We will only actually use it to detect whether text has, or has not, been written to a stream between two points in time, by seeing whether or not it has increased.

define STREAM_EXTENT(x) Streams::get_position(x)

§12. The remaining operations are available only for streams in memory (well, and for NULL, but of course they do nothing when applied to that). While they could be provided for file streams, this would be so inefficient that we will pretend it is impossible. Any function which might need to use one of these operations should open with the following sentinel macro:

define STREAM_MUST_BE_IN_MEMORY(x)
    if ((x != NULL) && (x->write_to_memory == NULL))
        internal_error("text_stream not in memory");

§13. First, we can erase one or more recently written characters:

define STREAM_BACKSPACE(x) Streams::set_position(x, Streams::get_position(x) - 1)
define STREAM_ERASE_BACK_TO(start_position) Streams::set_position(OUT, start_position)

§14. Second, we can look at the text written. The minimal form is to look at just the most recent character, but we can also copy one entire memory stream into another stream (where the target can be either a memory or file stream).

define STREAM_MOST_RECENT_CHAR(x) Streams::latest(x)
define STREAM_COPY(to, from) Streams::copy(to, from)

§15. So much for the definition; now the implementation.

Here is the text_stream structure. Open memory streams are represented by structures where write_to_memory is valid, open file streams by those where write_to_file is valid. That counts every open stream except NULL, which of course doesn't point to a text_stream structure at all.

Any stream can have USES_XML_ESCAPES_STRF set or cleared. When this is set, the XML (and HTML) escapes of & for ampersand, and < and > for angle brackets, will be used automatically on writing. By default this flag is clear, that is, no conversion is made.

define MALLOCED_STRF            0x00000001  was the write_to_memory pointer claimed by malloc?
define USES_XML_ESCAPES_STRF    0x00000002  see above
define USES_LOG_ESCAPES_STRF    0x00000004  WRITE to this stream supports $ escapes
define INDENT_PENDING_STRF      0x00000008  we have just ended a line, so further text should indent
define FILE_ENCODING_ISO_STRF   0x00000010  relevant only for file streams
define FILE_ENCODING_UTF8_STRF  0x00000020  relevant only for file streams
define ECHO_BYTES_STRF          0x00000080  for debugging only
define FOR_RE_STRF              0x00000100  for debugging only
define FOR_TT_STRF              0x00000200  for debugging only
define FOR_CO_STRF              0x00000400  for debugging only
define FOR_FI_STRF              0x00000800  for debugging only
define FOR_OM_STRF              0x00001000  for debugging only
define USES_I6_ESCAPES_STRF     0x00002000  as if an Inform 6 string
define READ_ONLY_STRF           0x00008000
define INDENTATION_BASE_STRF    0x00010000  number of tab stops in from the left margin
define INDENTATION_MASK_STRF    0x0FFF0000  (held in these bits)

typedef struct text_stream {
    int stream_flags;  bitmap of the *_STRF values above
    FILE *write_to_file;  for an open stream, exactly one of these is NULL
    struct HTML_file_state *as_HTML;  relevant only to the HTML:: section
    inchar32_t *write_to_memory;
    struct filename *file_written;  ditto
    int chars_written;  number of characters sent, counting \n as 1
    int chars_capacity;  maximum number the stream can accept without claiming more resources
    struct text_stream *stream_continues;  if one memory stream is extended by another
} text_stream;

The structure text_stream is accessed in 2/wal, 4/pm and here.

§16. A theological question: what is the text encoding for the null stream?

define STREAM_USES_UTF8(x) ((x)?((x->stream_flags) & FILE_ENCODING_UTF8_STRF):FALSE)

§17. When text is stored at write_to_memory, it is kept as a zero-terminated C wide string, with one word per Unicode code point. It turns out to be efficient to preserve a small margin of clear space at the end of the space, so out of the chars_capacity space, the following amount will be kept clear:

define SPACE_AT_END_OF_STREAM 6

§18. A statistic we keep, since it costs little:

int total_file_writes = 0;  number of text files opened for writing during the run

§19. Initialising the stream structure. Note that the following fills in sensible defaults for every field, but the result is not a valid open stream; it's a blank form ready to be filled in.

By default the upper limit on file size is 2 GB. It's very hard to see this ever being approached for any tool associated with the Inform project, where text files have a size which is proportionate to the result of human writing. The only output file with a sorceror's-apprentice-like ability to grow and grow is the debugging file, and if it should reach 2 GB then it deserves to be truncated and we will shed no tears.

void Streams::initialise(text_stream *stream, int from) {
    if (stream == NULL) internal_error("tried to initialise NULL stream");
    stream->stream_flags = from;
    stream->write_to_file = NULL;
    stream->write_to_memory = NULL;
    stream->chars_written = 0;
    stream->chars_capacity = 2147483647;
    stream->stream_continues = NULL;
    stream->as_HTML = NULL;
    stream->file_written = NULL;
}

§20. Any stream can have the following flag set or cleared. When this is set, the XML (and HTML) escapes of & for ampersand, and < and > for angle brackets, will be used automatically on writing. By default this flag is clear, that is, no conversion is made.

void Streams::enable_XML_escapes(text_stream *stream) {
    if (stream) stream->stream_flags |= USES_XML_ESCAPES_STRF;
}

void Streams::disable_XML_escapes(text_stream *stream) {
    if ((stream) && (stream->stream_flags & USES_XML_ESCAPES_STRF))
        stream->stream_flags -= USES_XML_ESCAPES_STRF;
}

int I6_escapes_globally_enabled = FALSE;
void Streams::enable_I6_escapes(text_stream *stream) {
    if (stream) stream->stream_flags |= USES_I6_ESCAPES_STRF;
    I6_escapes_globally_enabled = TRUE;
}

void Streams::disable_I6_escapes(text_stream *stream) {
    if ((stream) && (stream->stream_flags & USES_I6_ESCAPES_STRF))
        stream->stream_flags -= USES_I6_ESCAPES_STRF;
    I6_escapes_globally_enabled = FALSE;
}

int Streams::I6_escapes_enabled(text_stream *stream) {
    return I6_escapes_globally_enabled;
}

void Streams::enable_debugging(text_stream *stream) {
    if (stream) stream->stream_flags |= USES_LOG_ESCAPES_STRF;
}

void Streams::disable_debugging(text_stream *stream) {
    if ((stream) && (stream->stream_flags & USES_LOG_ESCAPES_STRF))
        stream->stream_flags -= USES_LOG_ESCAPES_STRF;
}

void Streams::mark_as_read_only(text_stream *stream) {
    if (stream) stream->stream_flags |= READ_ONLY_STRF;
}

void Streams::declare_as_HTML(text_stream *stream, HTML_file_state *hs) {
    if (stream) stream->as_HTML = hs;
}

HTML_file_state *Streams::get_HTML_file_state(text_stream *stream) {
    return stream->as_HTML;
}

§21. Logging.

void Streams::log(OUTPUT_STREAM, void *vS) {
    text_stream *stream = (text_stream *) vS;
    if (stream == NULL) {
        WRITE("NULL");
    } else if (stream->write_to_file) {
        WRITE("F'%f'(%d)", stream->file_written, stream->chars_written);
    } else {
        WRITE("S(");
        while (stream) {
            WRITE("%d/%d", stream->chars_written, stream->chars_capacity);
            if (stream->stream_continues) WRITE("+");
            stream = stream->stream_continues;
        }
        WRITE(")");
    }
}

§22. Standard I/O wrappers. The first call to Streams::get_stdout() creates a suitable wrapper for stdout and returns a text_stream * pointer to it; subsequent calls just return this wrapper.

text_stream STDOUT_struct; int stdout_wrapper_initialised = FALSE;
text_stream *Streams::get_stdout(void) {
    if (stdout_wrapper_initialised == FALSE) {
        Streams::initialise(&STDOUT_struct, 0); STDOUT_struct.write_to_file = stdout;
        stdout_wrapper_initialised = TRUE;
        #ifdef STDOUT_LOCALE_TEST
        STDOUT_struct.stream_flags |= ECHO_BYTES_STRF;
        #endif
        if (Locales::get(CONSOLE_LOCALE) == FILE_ENCODING_ISO_STRF)
            STDOUT_struct.stream_flags |= FILE_ENCODING_ISO_STRF;
        if (Locales::get(CONSOLE_LOCALE) == FILE_ENCODING_UTF8_STRF)
            STDOUT_struct.stream_flags |= FILE_ENCODING_UTF8_STRF;
    }
    return &STDOUT_struct;
}

§23. And similarly for the standard error file.

text_stream STDERR_struct; int stderr_wrapper_initialised = FALSE;
text_stream *Streams::get_stderr(void) {
    if (stderr_wrapper_initialised == FALSE) {
        Streams::initialise(&STDERR_struct, 0); STDERR_struct.write_to_file = stderr;
        stderr_wrapper_initialised = TRUE;
        if (Locales::get(CONSOLE_LOCALE) == FILE_ENCODING_ISO_STRF)
            STDERR_struct.stream_flags |= FILE_ENCODING_ISO_STRF;
        if (Locales::get(CONSOLE_LOCALE) == FILE_ENCODING_UTF8_STRF)
            STDERR_struct.stream_flags |= FILE_ENCODING_UTF8_STRF;
    }
    return &STDERR_struct;
}

§24. Creating file streams. Note that this can fail, if the host filing system refuses to open the file, so we return TRUE if and only if successful.

int Streams::open_to_file(text_stream *stream, filename *name, int encoding) {
    if (stream == NULL) internal_error("tried to open NULL stream");
    if (name == NULL) internal_error("stream_open_to_file on null filename");
    Streams::initialise(stream, FOR_FI_STRF);
    switch(encoding) {
        case UTF8_ENC: stream->stream_flags |= FILE_ENCODING_UTF8_STRF; break;
        case ISO_ENC: stream->stream_flags |= FILE_ENCODING_ISO_STRF; break;
        default: internal_error("stream has unknown text encoding");
    }
    stream->write_to_file = Filenames::fopen(name, "w");
    if (stream->write_to_file == NULL) return FALSE;
    stream->file_written = name;
    total_file_writes++;
    return TRUE;
}

§25. Similarly for appending:

int Streams::open_to_file_append(text_stream *stream, filename *name, int encoding) {
    if (stream == NULL) internal_error("tried to open NULL stream");
    if (name == NULL) internal_error("stream_open_to_file on null filename");
    Streams::initialise(stream, FOR_FI_STRF);
    switch(encoding) {
        case UTF8_ENC: stream->stream_flags |= FILE_ENCODING_UTF8_STRF; break;
        case ISO_ENC: stream->stream_flags |= FILE_ENCODING_ISO_STRF; break;
        default: internal_error("stream has unknown text encoding");
    }
    stream->write_to_file = Filenames::fopen(name, "a");
    if (stream->write_to_file == NULL) return FALSE;
    total_file_writes++;
    return TRUE;
}

§26. Creating memory streams. Here we have a choice. One option is to use Memory::calloc to allocate memory to hold the text of the stream; this too can fail for host platform reasons, so again we return a success code.

int Streams::open_to_memory(text_stream *stream, int capacity) {
    if (stream == NULL) internal_error("tried to open NULL stream");
    capacity += SPACE_AT_END_OF_STREAM;
    Streams::initialise(stream, FOR_OM_STRF);
    stream->write_to_memory = Memory::calloc(capacity, sizeof(inchar32_t), STREAM_MREASON);
    if (stream->write_to_memory == NULL) return FALSE;
    (stream->write_to_memory)[0] = 0;
    stream->stream_flags |= MALLOCED_STRF;
    stream->chars_capacity = capacity;
    return TRUE;
}

§27. The other option avoids fresh memory allocqtion by using specific storage already available. If called validly, this cannot fail.

text_stream Streams::new_buffer(int capacity, inchar32_t *at) {
    if (at == NULL) internal_error("tried to make stream wrapper for NULL string");
    if (capacity < SPACE_AT_END_OF_STREAM)
        internal_error("memory stream too small");
    text_stream stream;
    Streams::initialise(&stream, FOR_TT_STRF);
    stream.write_to_memory = at;
    (stream.write_to_memory)[0] = 0;
    stream.chars_capacity = capacity - SPACE_AT_END_OF_STREAM;
    return stream;
}

§28. Converting from C strings. We then have three ways to open a stream whose initial contents are given by a C string. First, a wide string (a sequence of 32-bit Unicode code points, null terminated):

int Streams::open_from_wide_string(text_stream *stream, const inchar32_t *c_string, int capacity) {
    if (stream == NULL) internal_error("tried to open NULL stream");
    Ensure a capacity large enough to hold the initial string in one frame28.3;
    if (c_string) Streams::write_wide_string(stream, c_string);
    return TRUE;
}

void Streams::write_wide_string(text_stream *stream, const inchar32_t *c_string) {
    for (int i=0; c_string[i]; i++) Streams::putc(c_string[i], stream);
}

§28.1. Similarly, an ISO string (a sequence of 8-bit code points in the first page of the Unicode set, null terminated):

int Streams::open_from_ISO_string(text_stream *stream, const char *c_string) {
    if (stream == NULL) internal_error("tried to open NULL stream");
    int capacity = (c_string)?((int) strlen(c_string)):0;
    Ensure a capacity large enough to hold the initial string in one frame28.3;
    if (c_string) Streams::write_ISO_string(stream, c_string);
    return TRUE;
}

void Streams::write_ISO_string(text_stream *stream, const char *c_string) {
    for (int i=0; c_string[i]; i++) Streams::putci(c_string[i], stream);
}

§28.2. Finally, a UTF-8 encoded C string:

int Streams::open_from_UTF8_string(text_stream *stream, const char *c_string) {
    if (stream == NULL) internal_error("tried to open NULL stream");
    int capacity = (c_string)?((int) strlen(c_string)):0;
    Ensure a capacity large enough to hold the initial string in one frame28.3;
    if (c_string) Streams::write_UTF8_string(stream, c_string);
    return TRUE;
}

void Streams::write_UTF8_string(text_stream *stream, const char *c_string) {
    unicode_file_buffer ufb = TextFiles::create_ufb();
    inchar32_t c;
    while ((c = TextFiles::utf8_fgetc(NULL, &c_string, &ufb)) != 0)
        Streams::putc(c, stream);
}

§28.3. ...all of which use:

Ensure a capacity large enough to hold the initial string in one frame28.3 =

    if (capacity < 8) capacity = 8;
    capacity += 1+SPACE_AT_END_OF_STREAM;
    int rv = Streams::open_to_memory(stream, capacity);
    if (rv == FALSE) return FALSE;

This code is used in §28, §28.1, §28.2.

§29. Converting to C strings. Now for the converse problem.

void Streams::write_as_wide_string(inchar32_t *C_string, text_stream *stream, int buffer_size) {
    if (buffer_size == 0) return;
    if (stream == NULL) { C_string[0] = 0; return; }
    if (stream->write_to_file) internal_error("stream_get_text on file stream");
    int i = 0;
    while (stream) {
        for (int j=0; j<stream->chars_written; j++) {
            if (i >= buffer_size-1) break;
            C_string[i++] = stream->write_to_memory[j];
        }
        stream = stream->stream_continues;
    }
    C_string[i] = 0;
}

§30. Unicode code points outside the first page are flattened to '?' in an ISO string:

void Streams::write_as_ISO_string(char *C_string, text_stream *stream, int buffer_size) {
    if (buffer_size == 0) return;
    if (stream == NULL) { C_string[0] = 0; return; }
    if (stream->write_to_file) internal_error("stream_get_text on file stream");
    int i = 0;
    while (stream) {
        for (int j=0; j<stream->chars_written; j++) {
            if (i >= buffer_size-1) break;
            inchar32_t c = stream->write_to_memory[j];
            if (c < 256) C_string[i++] = (char) c; else C_string[i++] = '?';
        }
        stream = stream->stream_continues;
    }
    C_string[i] = 0;
}

§31.

void Streams::write_as_UTF8_string(char *C_string, text_stream *stream, int buffer_size) {
    if (buffer_size == 0) return;
    if (stream == NULL) { C_string[0] = 0; return; }
    if (stream->write_to_file) internal_error("stream_get_text on file stream");
    unsigned char *to = (unsigned char *) C_string;
    int i = 0;
    while (stream) {
        for (int j=0; j<stream->chars_written; j++) {
            unsigned int c = (unsigned int) stream->write_to_memory[j];
            if (c >= 0x200000) {  invalid Unicode
                if (i >= buffer_size-1) break;
                to[i++] = '?';
            } else if (c >= 0x10000) {
                if (i >= buffer_size-4) break;
                to[i++] = 0xF0 + (unsigned char) (c >> 18);
                to[i++] = 0x80 + (unsigned char) ((c >> 12) & 0x3f);
                to[i++] = 0x80 + (unsigned char) ((c >> 6) & 0x3f);
                to[i++] = 0x80 + (unsigned char) (c & 0x3f);
            } else if (c >= 0x800) {
                if (i >= buffer_size-3) break;
                to[i++] = 0xE0 + (unsigned char) (c >> 12);
                to[i++] = 0x80 + (unsigned char) ((c >> 6) & 0x3f);
                to[i++] = 0x80 + (unsigned char) (c & 0x3f);
            } else if (c >= 0x80) {
                if (i >= buffer_size-2) break;
                to[i++] = 0xC0 + (unsigned char) (c >> 6);
                to[i++] = 0x80 + (unsigned char) (c & 0x3f);
            } else {
                if (i >= buffer_size-1) break;
                to[i++] = (unsigned char) c;
            }
        }
        stream = stream->stream_continues;
    }
    to[i] = 0;
}

§32. Locale versions.

int Streams::open_from_locale_string(text_stream *stream, const char *C_string) {
    if (Locales::get(SHELL_LOCALE) == FILE_ENCODING_UTF8_STRF)
        return Streams::open_from_UTF8_string(stream, C_string);
    if (Locales::get(SHELL_LOCALE) == FILE_ENCODING_ISO_STRF)
        return Streams::open_from_ISO_string(stream, C_string);
    Errors::fatal("unknown command line locale");
    return FALSE;
}

void Streams::write_as_locale_string(char *C_string, text_stream *stream, int buffer_size) {
    if (Locales::get(SHELL_LOCALE) == FILE_ENCODING_UTF8_STRF)
        Streams::write_as_UTF8_string(C_string, stream, buffer_size);
    else if (Locales::get(SHELL_LOCALE) == FILE_ENCODING_ISO_STRF)
        Streams::write_as_ISO_string(C_string, stream, buffer_size);
    else Errors::fatal("unknown command line locale");
}

void Streams::write_locale_string(text_stream *stream, char *C_string) {
    if (Locales::get(SHELL_LOCALE) == FILE_ENCODING_UTF8_STRF)
        Streams::write_UTF8_string(stream, C_string);
    else if (Locales::get(SHELL_LOCALE) == FILE_ENCODING_ISO_STRF)
        Streams::write_ISO_string(stream, C_string);
    else Errors::fatal("unknown command line locale");
}

§33. Flush and close. Note that flush is an operation which can be performed on any stream, including NULL:

void Streams::flush(text_stream *stream) {
    if (stream == NULL) return;
    if (stream->write_to_file) fflush(stream->write_to_file);
}

§34. But closing is not allowed for NULL or the standard I/O wrappers:

void Streams::close(text_stream *stream) {
    if (stream == NULL) internal_error("tried to close NULL stream");
    if (stream == &STDOUT_struct) internal_error("tried to close STDOUT stream");
    if (stream == &STDERR_struct) internal_error("tried to close STDERR stream");
    if (stream->chars_capacity == -1) internal_error("stream closed twice");
    if (stream->stream_continues) {
        Streams::close(stream->stream_continues);
        stream->stream_continues = NULL;
    }
    stream->chars_capacity = -1;  mark as closed
    if (stream->write_to_file) Take suitable action to close the file stream34.1;
    if (stream->write_to_memory) Take suitable action to close the memory stream34.2;
}

§34.1. Note that we need do nothing to close a memory stream when the storage was supplied by our client; it only needs freeing if we were the ones who allocated it.

Inscrutably, fclose returns EOF to report any failure.

Take suitable action to close the file stream34.1 =

    if ((ferror(stream->write_to_file)) || (fclose(stream->write_to_file) == EOF))
        Errors::fatal("The host computer reported an error trying to write a text file");
    if (stream != DL)
        LOGIF(TEXT_FILES, "Text file '%f' (%s): %d characters written\n",
            stream->file_written,
            (stream->stream_flags & FILE_ENCODING_UTF8_STRF)?"UTF8":"ISO",
            stream->chars_written);
    stream->write_to_file = NULL;

This code is used in §34.

§34.2. Note that we need do nothing to close a memory stream when the storage was supplied by our client; it only needs freeing if we were the ones who allocated it. free is a void function; in theory it cannot fail, if supplied a valid argument.

We have to be very careful once we have called free, because that memory may well contain the text_stream structure to which stream points — see how continuations are made, below.

Take suitable action to close the memory stream34.2 =

    if ((stream->stream_flags) & MALLOCED_STRF) {
        inchar32_t *mem = stream->write_to_memory;
        stream->write_to_memory = NULL;
        Memory::I7_free(mem, STREAM_MREASON, stream->chars_capacity*((int) sizeof(inchar32_t)));
        stream = NULL;
    }

This code is used in §34.

§35. Writing. Our equivalent of fputc reads:

void Streams::putc(inchar32_t c, text_stream *stream) {
    if (stream == NULL) return;
    text_stream *first_stream = stream;
    if (c != '\n') Insert indentation if this is pending35.2;
    if (stream->stream_flags & READ_ONLY_STRF) internal_error("modifying read-only stream");
    if ((stream->stream_flags) & USES_XML_ESCAPES_STRF) {
        switch(c) {
            case NEWLINE_IN_STRING: Streams::literal(stream, "<br>"); return;
            case '&': Streams::literal(stream, "&amp;"); return;
            case '<': Streams::literal(stream, "&lt;"); return;
            case '>': Streams::literal(stream, "&gt;"); return;
        }
    }
    while (stream->stream_continues) stream = stream->stream_continues;
    Ensure there is room to expand the escape sequence into35.3;
    if (stream->write_to_file) {
        if (stream->stream_flags & FILE_ENCODING_UTF8_STRF)
            Put a UTF8-encoded character into the underlying file35.1
        else if (stream->stream_flags & FILE_ENCODING_ISO_STRF) {
            if (c >= 0x100) c = '?';
            fputc((int) c, stream->write_to_file);
        } else internal_error("stream has unknown text encoding");
        if (stream->stream_flags & ECHO_BYTES_STRF) {
            fputc(' ', stream->write_to_file);
            fputc('0'+(c/100), stream->write_to_file);
            fputc('0'+(c%100)/10, stream->write_to_file);
            fputc('0'+(c%10), stream->write_to_file);
            if (stream->stream_flags & FILE_ENCODING_UTF8_STRF)
                fputc('u', stream->write_to_file);
            if (stream->stream_flags & FILE_ENCODING_ISO_STRF)
                fputc('i', stream->write_to_file);
            fputc(' ', stream->write_to_file);
        }
    } else if (stream->write_to_memory) {
        if ((c >= 0x0300) && (c <= 0x036F) && (stream->chars_written > 0)) {
            unsigned int newc = (unsigned int) Characters::combine_accent(
                c, (stream->write_to_memory)[stream->chars_written - 1]);
            if (newc) {
                c = newc;
                stream->chars_written--;
            }
        }
        (stream->write_to_memory)[stream->chars_written] = c;
    }
    if (c == '\n') first_stream->stream_flags |= INDENT_PENDING_STRF;
    stream->chars_written++;
}

void Streams::putci(int c_int, text_stream *stream) {
    inchar32_t c;
    if (c_int >= 0) c = (inchar32_t) c_int; else c = (inchar32_t) (c_int + 256);
    Streams::putc(c, stream);
}

§35.1. Where we pack large character values, up to 65535, as follows.

Put a UTF8-encoded character into the underlying file35.1 =

    if (c >= 0x200000) {  invalid Unicode
        fputc('?', stream->write_to_file);
    } else if (c >= 0x10000) {
        fputc(0xF0 + (c >> 18), stream->write_to_file);
        fputc(0x80 + ((c >> 12) & 0x3f), stream->write_to_file);
        fputc(0x80 + ((c >> 6) & 0x3f), stream->write_to_file);
        fputc(0x80 + (c & 0x3f), stream->write_to_file);
    } else if (c >= 0x800) {
        fputc(0xE0 + (c >> 12), stream->write_to_file);
        fputc(0x80 + ((c >> 6) & 0x3f), stream->write_to_file);
        fputc(0x80 + (c & 0x3f), stream->write_to_file);
    } else if (c >= 0x80) {
        fputc(0xC0 + (c >> 6), stream->write_to_file);
        fputc(0x80 + (c & 0x3f), stream->write_to_file);
    } else fputc((int) c, stream->write_to_file);

This code is used in §35.

§35.2. Insert indentation if this is pending35.2 =

    if (first_stream->stream_flags & INDENT_PENDING_STRF) {
        first_stream->stream_flags -= INDENT_PENDING_STRF;
        int L = (first_stream->stream_flags & INDENTATION_MASK_STRF)/INDENTATION_BASE_STRF;
        for (int i=0; i<L; i++) {
            Streams::putc(' ', first_stream); Streams::putc(' ', first_stream);
            Streams::putc(' ', first_stream); Streams::putc(' ', first_stream);
        }
    }

This code is used in §35.

§35.3. The following is checked before any numerical printf-style escape is expanded into the stream, or before any single character is written. Thus we cannot overrun our buffers unless the expansion of a numerical escape exceeds SPACE_AT_END_OF_STREAM plus 1 in size. Since no outside influence gets to choose what formatting escapes we use (so that %3000d, say, can't occur), we can be pretty confident.

The interesting case occurs when we run out of memory in a memory stream. We make a continuation to a fresh text_stream structure, which points to twice as much memory as the original, allocated via malloc. (We will actually need a little more memory than that because we also have to make room for the text_stream structure itself.) We then set stream to the continuation. Given that malloc was successful — and it must have been or we would have stopped with a fatal error — the continuation is guaranteed to be large enough, since it's twice the size of the original, which itself was large enough to hold any escape sequence when opened.

Ensure there is room to expand the escape sequence into35.3 =

    if (stream->chars_written + SPACE_AT_END_OF_STREAM >= stream->chars_capacity) {
        if (stream->write_to_file) return;  write nothing further
        if (stream->write_to_memory) {
            int offset = (32 + 2*(stream->chars_capacity))*((int) sizeof(inchar32_t));
            int needed = offset + ((int) sizeof(text_stream)) + 32;
            void *further_allocation = Memory::malloc(needed, STREAM_MREASON);
            if (further_allocation == NULL) Errors::fatal("Out of memory");
            text_stream *continuation = (text_stream *) (further_allocation + offset);
            Streams::initialise(continuation, FOR_CO_STRF);
            continuation->write_to_memory = further_allocation;
            continuation->chars_capacity = 2*stream->chars_capacity;
            (continuation->write_to_memory)[0] = 0;
            stream->stream_continues = continuation;
            stream = continuation;
        }
    }

This code is used in §35.

§36. Literal printing is just printing with XML escapes switched off:

void Streams::literal(text_stream *stream, char *p) {
    if (stream == NULL) return;
    int i, x = ((stream->stream_flags) & USES_XML_ESCAPES_STRF);
    if (x) stream->stream_flags -= USES_XML_ESCAPES_STRF;
    for (i=0; p[i]; i++) Streams::putci(p[i], stream);
    if (x) stream->stream_flags += USES_XML_ESCAPES_STRF;
}

§37. Shifting indentation. For every indent there must be an equal and opposite outdent, but error conditions can cause some compilation routines to issue problem messages and then leave what they're doing incomplete, so we will be a little cautious about assuming that a mismatch means an error.

void Streams::indent(text_stream *stream) {
    if (stream == NULL) return;
    stream->stream_flags += INDENTATION_BASE_STRF;
}

void Streams::outdent(text_stream *stream) {
    if (stream == NULL) return;
    if ((stream->stream_flags & INDENTATION_MASK_STRF) == 0) {
        if (Errors::have_occurred() == FALSE) internal_error("stream indentation negative");
        return;
    }
    stream->stream_flags -= INDENTATION_BASE_STRF;
}

void Streams::set_indentation(text_stream *stream, int N) {
    if (stream == NULL) return;
    int B = stream->stream_flags & INDENTATION_MASK_STRF;
    stream->stream_flags -= B;
    stream->stream_flags += N*INDENTATION_BASE_STRF;
}

int Streams::get_indentation(text_stream *stream) {
    if (stream == NULL) return 0;
    return (stream->stream_flags & INDENTATION_MASK_STRF)/INDENTATION_BASE_STRF;
}

§38. We can read the position for any stream, including NULL, but no matter how much is written to NULL this position never budges.

Because of continuations, this is not as simple as returning the chars_written field.

int Streams::get_position(text_stream *stream) {
    int t = 0;
    while (stream) {
        t += stream->chars_written;
        stream = stream->stream_continues;
    }
    return t;
}

§39. Memory-stream-only functions. While it would be easy enough to implement this for file streams too, there's no point, since it is used only in concert with backspacing.

int Streams::latest(text_stream *stream) {
    if (stream == NULL) return 0;
    if (stream->write_to_file) internal_error("stream_latest on file stream");
    while ((stream->stream_continues) && (stream->stream_continues->chars_written > 0))
        stream = stream->stream_continues;
    if (stream->write_to_memory) {
        if (stream->chars_written > 0)
            return (int) ((stream->write_to_memory)[stream->chars_written - 1]);
    }
    return 0;
}

§40. Accessing characters by index. Note that the stream terminates at the first zero byte found, so that putting a zero truncates it.

inchar32_t Streams::get_char_at_index(text_stream *stream, int position) {
    if (stream == NULL) internal_error("examining null stream");
    if (stream->write_to_file) internal_error("examining file stream");
    while (position >= stream->chars_written) {
        position = position - stream->chars_written;
        stream = stream->stream_continues;
        if (stream == NULL) return 0;
    }
    if (stream->write_to_memory == NULL) return 0;
    return (stream->write_to_memory)[position];
}

void Streams::put_char_at_index(text_stream *stream, int position, inchar32_t C) {
    if (stream == NULL) internal_error("modifying null stream");
    if (stream->write_to_file) internal_error("modifying file stream");
    if (stream->stream_flags & READ_ONLY_STRF) internal_error("modifying read-only stream");
    while (position >= stream->chars_written) {
        position = position - stream->chars_written;
        stream = stream->stream_continues;
        if (stream == NULL) internal_error("overrun memory stream");
    }
    (stream->write_to_memory)[position] = C;
    if (C == 0) {
        stream->chars_written = position;
        if (stream->stream_continues) {
            Streams::close(stream->stream_continues);
            stream->stream_continues = NULL;
        }
    }
}

§41. Now for what is the trickiest function, because the position may be moved back so that later continuations fall away. This will very rarely happen, so we won't worry about the inefficiency of freeing up the memory saved by closing such continuation blocks (which would be inefficient if we immediately had to open similar ones again).

void Streams::set_position(text_stream *stream, int position) {
    if (stream == NULL) return;
    if (position < 0) position = 0;  to simplify the implementation of backspacing
    if (stream->write_to_file) internal_error("stream_set_position on file stream");
    if (stream->stream_flags & READ_ONLY_STRF) internal_error("modifying read-only stream");
    if (stream->write_to_memory) {
        while (position > stream->chars_written) {
            position = position - stream->chars_written;
            stream = stream->stream_continues;
            if (stream == NULL) internal_error("can't set position forwards");
        }
        stream->chars_written = position;
        (stream->write_to_memory)[stream->chars_written] = 0;
        if (stream->stream_continues) {
            Streams::close(stream->stream_continues);
            stream->stream_continues = NULL;
        }
    }
}

§42. Lastly, our copying function, where from has to be a memory stream (or NULL) but to can be anything (including NULL).

void Streams::copy(text_stream *to, text_stream *from) {
    if ((from == NULL) || (to == NULL)) return;
    if (from == to) internal_error("tried to copy a stream to itself");
    if (from->write_to_file) internal_error("stream_copy from file stream");
    if (from->write_to_memory) {
        for (int i=0; i<from->chars_written; i++) {
            inchar32_t c = (from->write_to_memory)[i];
            Streams::putc(c, to);
        }
        if (from->stream_continues) Streams::copy(to, from->stream_continues);
    }
}

§43. Writer. This writes one stream into another one, which implements %S.

void Streams::writer(OUTPUT_STREAM, char *format_string, void *vS) {
    text_stream *S = (text_stream *) vS;
    Streams::copy(OUT, S);
}