-- TEncode / "Transterpreter Encoding"

Objectives:
  * Provide an extensible encoding format which can easily be extended.
  * Requiring no dynamic memory to parse typical files, support in-place decoding.
  * Be decodable as a stream of bytes, not just a disk file.

TEncode is binary markup language, a modified version of IFF.

TEncode streams operate in either 16 bit or 32 bit mode, the mode affects the size of integers used.  A stream is made up of elements, each element consists of a 4 byte identifier, followed by an integer, then 0 or more bytes of data.  Elements always begin on integer aligned boundaries, previous elements are padded with null (0) bytes to maintain this.  All integers are big-endian encoded, and of consistent size (2 bytes / 4 bytes) throughout a stream.  Integers are unsigned unless otherwise stated.

  Identifier := 4 * BYTE
  Integer    := INT16 / INT32
  Data       := { BYTE }
  Padding    := { "\0" }
  Element    := Identifier, Integer, Data, Padding

Identifiers are made up of 4 ASCII encoded characters, and are case-sensitive.  The last byte indicates the type of data held in the element.  The following types are defined:

  B = byte string, 
      Integer field encodes number of bytes in Data field, excluding padding null-bytes.
  I = signed integer, 
      Integer field encodes signed numeric value, no Data bytes follow.
  L = element list,
      Integer field encodes number of bytes in Data field which will be a multiple of the integer size,
      Data field contains nested Elements, and may be parsed as a new or sub-Stream.
  S = UTF8 null-terminated string, 
      Integer field encodes number of bytes in Data including null terminator, but not padding.
  U = unsigned integer, 
      Integer field encodes unsigned numeric value, no Data bytes follow.

With the exception of signed and unsigned integer types, the Integer of all Elements defines the unpadded size of the Data which follows.  Decoders may use this relationship to skip unknown element types, therefore this relationship must be preserved when adding new types.

A TEncode stream begins with the special 'tenc' or 'TEnc' Element, the Integer of which indicates the number of bytes which follow in the rest of the stream.  A lower-case 'tenc' indicates that integers are small (16 bit), whereas an upper-case 'TEnc' indicates integers are large (32 bits).  The TEnc Element contains all other elements in the stream.

  Stream        := { Element }

  ByteString    := 3 * BYTE, "B", Integer, Data [, Padding ]
  SignedInt     := 3 * BYTE, "I", Integer
  ElementList   := 3 * BYTE, "L", Integer, Stream
  UTF8String    := 3 * BYTE, "S", Integer, {<ASCII character>}, "\0" [, Padding ]
  UnsignedInt   := 3 * BYTE, "U", Integer

  Element := ByteString / SignedInt / ElementList / UTF8String / UnsignedInt
  Header  := ("tenc", INT16) / ("TEnc", INT32)
  Stream  := { Element }
  TEncode := Header, Stream

Decoders should ignore elements they do not understand or care about.  If multiple elements with the same identifier exist in the same stream and the decoder does not expect multiple instances, then the decode should use the first encountered.  When defining a stream, new elements may be freely added to its definition across versions; however, the order of elements must be maintained in order to keep parsing simple.

-- Transterpreter Byte Code (TBC)

Transterpreter byte code is a TEncode stream defined as follows:

TEnc <stream length>
  tbcL <length>
    endU <endian (0=little, 1=big)>
    ws U <workspace size (words)>
    vs U <vectorspace size (words)>
    padB <length> <padding>
    bc B <length> <bytecode>

    // Top-Level-Process descriptor (optional)
    tlpL <length>
      fmtS <length> <top-level-format> [<padding>]
      symS <length> <top-level-process symbol name> [<padding>]

    // FFI table (optional)
    ffiL <length>
      // Library names
      libL <length> 
        libS <length> <library name> [<padding>]
	... further libS elements ...
      // Symbols
      symL <length> 
        symS <length> <symbol> [<padding>]
	... further symS elements ...
      // Symbol to library mapping
      mapL <length>
        idxI <library index of symbol (-1 = unknown)>
	... further idxI elements (one element per symS) ...
   
    // Symbol Table (optional)
    stbL <length>
      symL <length>
        offU <bytecode offset>
        symS <length> <symbol name> [<padding>]
	// (optionally)
	defS <length> <definition> [<padding>]
	ws U <workspace words>
	vs U <vectorspace words>

    // Debug Information (optional)
    dbgL <length>
      // File names
      fn L <length>
        fn S <length> <file name>
      // Line Numbering Data
      lndB <length>
        <bytecode offset> <file index> <line number>
	... further elements ...

  ... further tbcL elements (e.g. alternate endian) ...
