Improve (fix) the T·S·V parser

[Shushe] / README.markdown

diff --git a/README.markdown b/README.markdown
index 51aca8d383ac5162e1c8f6265298708f74d80323..f6a08642353a226246ec11eaa60b873e764cfb1d 100644 (file)
--- a/README.markdown
+++ b/README.markdown
@@ -1,6 +1,6 @@
 # ⛩️📰 书社
 
 # ⛩️📰 书社
 

-<b>An X·S·L·T‐based static site generator.</b>
+<b>A make·file for X·M·L.</b>

 
 <dfn>⛩️📰 书社</dfn> aims to make it easy to generate websites with
   X·S·L·T and G·N·U Make.
 
 <dfn>⛩️📰 书社</dfn> aims to make it easy to generate websites with
   X·S·L·T and G·N·U Make.


@@ -18,6 +18,15 @@ It makes things easier by :⁠—
 
 It aims to do this with zero dependencies beyond the programs already
   installed on your computer.
 
 It aims to do this with zero dependencies beyond the programs already
   installed on your computer.

+(On Linux machines, you may need to install `libxml2-utils` to get the
+  commandline programs from `libxml2`.)
+
+**Note:**
+⛩️📰 书社 requires functionality present in G·N·U Make 3.81 (or later)
+  and will not work in previous versions, or other implementations of
+  Make.
+Compatibility with later versions of G·N·U Make is assumed, but not
+  tested.

 
 ## Nomenclature
 
 
 ## Nomenclature
 


@@ -91,19 +100,29 @@ In every case, you may supply your own implementation by overriding the
   corresponding (allcaps) variable (e·g, set `MKDIR` to supply your own
   `mkdir` implementation).
 
   corresponding (allcaps) variable (e·g, set `MKDIR` to supply your own
   `mkdir` implementation).
 

+- `awk`

 - `cat`
 - `cat`

+- `cksum`

 - `cp`
 - `cp`

+- `date`

 - `echo`
 - `file`
 - `find`
 - `echo`
 - `file`
 - `find`

+- `git` (optional; set `GIT=` to disable)
+- `ln`

 - `mkdir` (requires support for `-p`)
 - `mv`
 - `mkdir` (requires support for `-p`)
 - `mv`

+- `od` (requires support for `-t x1`)

 - `printf`
 - `printf`

+- `rm`

 - `sed`
 - `sed`

+- `sleep`
+- `stat`

 - `test`
 - `touch`
 - `tr` (requires support for `-d`)
 - `uuencode` (requires support for `-m` and `-r`)
 - `test`
 - `touch`
 - `tr` (requires support for `-d`)
 - `uuencode` (requires support for `-m` and `-r`)

+- `xargs` (requires support for `-0`)

 - `xmlcatalog` (provided by `libxml2`)
 - `xmllint` (provided by `libxml2`)
 - `xsltproc` (provided by `libxslt`)
 - `xmlcatalog` (provided by `libxml2`)
 - `xmllint` (provided by `libxml2`)
 - `xsltproc` (provided by `libxslt`)


@@ -113,16 +132,32 @@ The following additional variables can be used to control the behaviour
 
 - **`SRCDIR`:**
   The location of the source files (default: `sources`).
 
 - **`SRCDIR`:**
   The location of the source files (default: `sources`).

+  Multiple source directories can be provided, so long as the same
+    file subpath doesn’t exist in more than one of them.

 
 - **`INCLUDEDIR`:**
 
 - **`INCLUDEDIR`:**

-  The location of the source files (default: `sources/includes`).
+  The location of source includes (default: `sources/includes`).

   This can be inside of `SRCDIR`, but needn’t be.
   This can be inside of `SRCDIR`, but needn’t be.

+  Multiple include directories can be provided, so long as the same
+    file subpath doesn’t exist in more than one of them.

 
 - **`BUILDDIR`:**
   The location of the (temporary) build directory (default: `build`).
 
 - **`BUILDDIR`:**
   The location of the (temporary) build directory (default: `build`).

+  `make clean` will delete this, and it is recommended that it not be
+    used for programs aside from ⛩️📰 书社.

 
 - **`DESTDIR`:**
   The location of directory to output files to (default: `public`).
 
 - **`DESTDIR`:**
   The location of directory to output files to (default: `public`).

+  `make install` will overwrite files in this directory which
+    correspond to those in `SRCDIR`.
+  It *will not* touch other files, including those generated from files
+    in `SRCDIR` which have since been deleted.
+
+  Files are first compiled to `$(BUILDDIR)/public` before they are
+    copied to `DESTDIR`, so this folder is relatively quick and
+    inexpensive to re·create.
+  It’s reasonable to simply delete it before every `make install` to
+    ensure stale content is removed.

 
 - **`THISDIR`:**
   The location of the ⛩️📰 书社 `GNUmakefile`.
 
 - **`THISDIR`:**
   The location of the ⛩️📰 书社 `GNUmakefile`.


@@ -131,29 +166,62 @@ The following additional variables can be used to control the behaviour
   This variable is used to find the ⛩️📰 书社 `lib/` folder, which is
     expected to be in the same location.
 
   This variable is used to find the ⛩️📰 书社 `lib/` folder, which is
     expected to be in the same location.
 

-- **`MAGICDIR`:**
-  The location of the magic files to use (default: `$(THISDIR)/magic`).
+- **`MAGIC`:**
+  A white·space‐separated list of magic files to use (default:
+    `$(THISDIR)/magic/*`).

 
 

-- **`FINDOPTS`:**
-  Options to pass to `find` when searching for source files (default:
-    `-LE`).
+- **`EXTRAMAGIC`:**
+  The value of this variable is appended to `MAGIC` by default, to
+    enable additional magic files without overriding the existing ones.

 
 - **`FINDRULES`:**
 
 - **`FINDRULES`:**

-  Rules to use with `find` when searching for source files (default:
-    `-flags -nohidden -and -not -name '.*'`).
+  Rules to use with `find` when searching for source files.
+  The default ignores files that start with a period or hyphen‐minus,
+    those which end with a cloparen, and those which contain a hash,
+    buck, percent, asterisk, colon, semi, eroteme, bracket, backslash,
+    or pipe.
+
+- **`EXTRAFINDRULES`:**
+  The value of this variable is appended to `FINDRULES` by default, to
+    enable additional rules without overriding the existing ones.
+
+- **`FINDINCLUDERULES`:**
+  Rules to use with `find` when searching for includes (default:
+    `$(FINDRULES)`).
+
+- **`EXTRAFINDINCLUDERULES`:**
+  The value of this variable is appended to `FINDINCLUDERULES` by
+    default, to enable additional rules without overriding the existing
+    ones.

 
 - **`PARSERS`:**
   A white·space‐separated list of parsers to use (default:
     `$(THISDIR)/parsers/*.xslt`).
 
 
 - **`PARSERS`:**
   A white·space‐separated list of parsers to use (default:
     `$(THISDIR)/parsers/*.xslt`).
 

+- **`EXTRAPARSERS`:**
+  The value of this variable is appended to `PARSERS` by default, to
+    enable additional parsers without overriding the existing ones.
+

 - **`TRANSFORMS`:**
   A white·space‐separated list of transforms to use (default:
     `$(THISDIR)/transforms/*.xslt`).
 
 - **`TRANSFORMS`:**
   A white·space‐separated list of transforms to use (default:
     `$(THISDIR)/transforms/*.xslt`).
 

+- **`EXTRATRANSFORMS`:**
+  The value of this variable is appended to `TRANSFORMS` by default, to
+    enable additional transforms without overriding the existing ones.
+

 - **`XMLTYPES`:**
   A white·space‐separated list of media types to consider X·M·L
     (default: `application/xml text/xml`).
 
 - **`XMLTYPES`:**
   A white·space‐separated list of media types to consider X·M·L
     (default: `application/xml text/xml`).
 

+- **`THISREV`:**
+  The current version of ⛩️📰 书社 (default: derived from the current
+    git tag/branch/commit).
+
+- **`SRCREV`:**
+  The current version of the source files (default: derived from the
+    current git tag/branch/commit).
+

 - **`VERBOSE`:**
   If this variable has a value, every recipe instruction will be
     printed when it runs (default: empty).
 - **`VERBOSE`:**
   If this variable has a value, every recipe instruction will be
     printed when it runs (default: empty).


@@ -173,6 +241,8 @@ Supported magic numbers include :⁠—
 - `#!js` for `text/javascript` files
 - `@charset "` for `text/css` files
 - `#!tsv` for `text/tab-separated-values` files
 - `#!js` for `text/javascript` files
 - `@charset "` for `text/css` files
 - `#!tsv` for `text/tab-separated-values` files

+- `%%` for `text/record-jar` files (unregistered; see
+    [[draft-phillips-record-jar-01][]])

 
 Text formats with associated X·S·L·T parsers are wrapped in a H·T·M·L
   `<script>` element whose `@type` gives its media type, and then
 
 Text formats with associated X·S·L·T parsers are wrapped in a H·T·M·L
   `<script>` element whose `@type` gives its media type, and then


@@ -180,18 +250,16 @@ Text formats with associated X·S·L·T parsers are wrapped in a H·T·M·L
 Source files whose media type does not have an associated X·S·L·T
   parser are considered “assets” and will not be transformed.
 
 Source files whose media type does not have an associated X·S·L·T
   parser are considered “assets” and will not be transformed.
 

-For compatibility with this program, source filenames should conform to
-  the following rules :⁠—
-
-- They should not start with a hyphen‐minus.
-  This is to prevent confusion between filenames and options on the
-    commandline.
-
-- They should not contain spaces, colons, percent signs, backticks,
-    question marks, hashes, or backslashes.
-
-In general, filenames should be such that they do not require
-  percent‐encoding in the path component of an i·r·i.
+**☡ For compatibility with this program, source file·names must not
+  contain Ascii white·space, colons (`:`), semis (`;`), pipes (`|`),
+  bucks (`$`), percents (`%`), hashes (`#`), asterisks (`*`), brackets
+  (`[` or `]`), erotemes (`?`), backslashes (`\`), or control
+  characters, must not begin with a hyphen‐minus (`-`), and must not
+  end with a cloparen (`)`).**
+The former characters have the potential to conflict with make syntax,
+  a leading hyphen‐minus is confusable for a command‐line argument, and
+  a trailing cloparen [activates a bug in G·N·U Make
+  3.81](https://stackoverflow.com/questions/17148468/capturing-filenames-including-parentheses-with-gnu-makes-wildcard-function#comment24825307_17148894).

 
 ## Parsers
 
 
 ## Parsers
 


@@ -203,12 +271,17 @@ Parsers are used to convert plaintext files into X·M·L trees, as well
 - **`parsers/plain.xslt`:**
   Wraps `text/plain` contents in a `<html:pre>` element.
 
 - **`parsers/plain.xslt`:**
   Wraps `text/plain` contents in a `<html:pre>` element.
 

+- **`parsers/record-jar.xslt`:**
+  Converts `text/record-jar` contents into a `<html:div>` of
+    `<html:dl>` elements (one for each record).
+

 - **`parsers/tsv.xslt`:**
   Converts `text/tab-separated-values` contents into an `<html:table>`
     element.
 
 - **`parsers/tsv.xslt`:**
   Converts `text/tab-separated-values` contents into an `<html:table>`
     element.
 

-New ⛩️📰 书社 parsers should have a `<xslt:template>` element with no
-  `@name` or `@mode` and whose `@match` attribute…
+New ⛩️📰 书社 parsers which target plaintext formats should have an
+  `<xslt:template>` element with no `@name` or `@mode` and whose
+  `@match` attribute…

 
 - Starts with an appropriately‐namespaced qualified name for a
     `<html:script>` element.
 
 - Starts with an appropriately‐namespaced qualified name for a
     `<html:script>` element.


@@ -226,13 +299,15 @@ For example, the trivial `text/plain` parser is defined as follows :⁠—
 ```xml
 <?xml version="1.0"?>
 <transform
 ```xml
 <?xml version="1.0"?>
 <transform

-       xmlns="http://www.w3.org/1999/XSL/Transform"
-       xmlns:html="http://www.w3.org/1999/xhtml"
-       version="1.0"
+  xmlns="http://www.w3.org/1999/XSL/Transform"
+  xmlns:html="http://www.w3.org/1999/xhtml"
+  xmlns:书社="urn:fdc:ladys.computer:20231231:Shu1She4"
+  version="1.0"

 >
 >

-       <template match="html:script[@type='text/plain']">
-               <html:pre><value-of select="."/></html:pre>
-       </template>
+  <书社:id>example:text/plain</书社:id>
+  <template match="html:script[@type='text/plain']">
+    <html:pre><value-of select="."/></html:pre>
+  </template>

 </transform>
 ```
 
 </transform>
 ```
 


@@ -240,10 +315,53 @@ For example, the trivial `text/plain` parser is defined as follows :⁠—
   the set of allowed plaintext file types.
 Multiple such `<xslt:template>` elements may be provided in a single
   parser, for example if the parser supports multiple media types.
   the set of allowed plaintext file types.
 Multiple such `<xslt:template>` elements may be provided in a single
   parser, for example if the parser supports multiple media types.

-
-It is **strongly recommended** that all templates in parsers other than
-  those described above be namespaced (by `@name` or `@mode`), to avoid
-  conflicts between templates in multiple parsers.
+Alternatively, you can set the `@书社:supported-media-types` attribute
+  on the root element of the parser to override media type support
+  detection.
+
+Even when `@书社:supported-media-types` is set, it is a requirement
+  that each parser transform any `<html:script>` elements with a
+  `@type` which matches their registered types into something else.
+Otherwise the parser will be stuck in an endless loop.
+The result tree of applying the transform to the `<html:script>`
+  element will be reparsed (in case any new `<html:script>` elements
+  were added in its subtree), and a `@书社:parsed-by` attribute will be
+  added to each toplevel element in the result.
+The value of this attribute will be the value of the `<书社:id>`
+  toplevel element in the parser.
+
+It is possible for parsers to support zero plaintext types.
+This is useful when targeting specific dialects of X·M·L; parsers in
+  this sense operate on the same basic principles as transforms
+  (described below).
+The major distinction between X·M·L parsers and transforms is where in
+  the process the transformation happens:
+Parsers are applied *prior* to embedding (and can be used to generate
+  embeds); transforms are applied *after*.
+
+It is **strongly recommended** that auxillary templates in parsers be
+  namespaced (by `@name` or `@mode`) whenever possible, to avoid
+  conflicts between parsers.
+
+### Attributes added during parsing
+
+⛩️📰 书社 will add a few attributes to the output of the parsing step,
+  namely :⁠—
+
+- A `@书社:cksum` attribute on toplevel result elements, giving the
+    `cksum` checksum of the corresponding source file.
+
+- For the elements which result from parsing plaintext `<html:script>`
+    elements :⁠—
+
+  - A `@书社:parsed-by` attribute, giving a space‐separated list of
+      parsers which parsed the node.
+    (Generally, this will be a list of one, but it is possible for the
+      result of a parse to be another plaintext node, which may be
+      parsed by a different parser.)
+
+  - A `@书社:media-type` attribute, giving the identified media type of
+      the plaintext node.

 
 ## Embedding
 
 
 ## Embedding
 


@@ -261,6 +379,8 @@ Embeds are replaced with the parsed contents of a file, unless the file
   is an asset, in which case an `<html:object>` element is produced
   instead (with the contents of the asset file provided as a base64
   `data:` u·r·i).
   is an asset, in which case an `<html:object>` element is produced
   instead (with the contents of the asset file provided as a base64
   `data:` u·r·i).

+Embed replacements will be given a `@书社:identifier` attribute whose
+  value will match the `@xlink:href` of the embed.

 
 Embedding takes place after parsing but before transformation, so
   parsers are able to generate their own embeds.
 
 Embedding takes place after parsing but before transformation, so
   parsers are able to generate their own embeds.


@@ -268,6 +388,15 @@ Embedding takes place after parsing but before transformation, so
   and update them accordingly; it will signal an error if the
   dependencies are recursive.
 
   and update them accordingly; it will signal an error if the
   dependencies are recursive.
 

+## Output Redirection
+
+By default, ⛩️📰 书社 installs files to the same location in `DESTDIR`
+  as they were placed in their `SRCDIR`.
+This behaviour can be customized by setting the `@书社:destination`
+  attribute on the root element, whose value can give a different path.
+This attribute is read after parsing, but before transformation (where
+  it is silently dropped).
+

 ## Transforms
 
 Transforms are used to convert X·M·L files into their final output,
 ## Transforms
 
 Transforms are used to convert X·M·L files into their final output,


@@ -275,25 +404,24 @@ Transforms are used to convert X·M·L files into their final output,
 ⛩️📰 书社 comes with some transforms; namely :⁠—
 
 - **`transforms/asset.xslt`:**
 ⛩️📰 书社 comes with some transforms; namely :⁠—
 
 - **`transforms/asset.xslt`:**

-  Converts `<html:object type="text/css">` elements into corresponding
-    `<html:link rel="stylesheet">` elements and
-    `<html:object type="text/javascript">` elements into corresponding
-    `<html:script>` elements.
-  This transform enables embedding of `text/css` and `text/javascript`
-    files, which ordinarily are considered assets (as they lack
-    associated parsers).
+  Converts `<html:object>` elements which correspond to recognized
+    media types into the appropriate H·T·M·L elements, and deletes
+    `<html:style>` elements from the body of the document and moves
+    them to the head.

 
 - **`transforms/metadata.xslt`:**
   Provides basic `<html:head>` metadata.
 
 - **`transforms/metadata.xslt`:**
   Provides basic `<html:head>` metadata.

-  This metadata is generated from `<html:meta>` descendants of the
-    first element with an `@itemscope` attribute (recommended to just
-    be the root element).
-  Such elements can provide metadata using the following `@itemprop`
-    attributes :⁠—
+  This metadata is generated from `<html:meta>` elements with one of
+    the following `@itemprop` attributes :⁠—

 
   - **`urn:fdc:ladys.computer:20231231:Shu1She4:title`:**
     Provides the title of the page.
 
 
   - **`urn:fdc:ladys.computer:20231231:Shu1She4:title`:**
     Provides the title of the page.
 

+  ⛩️📰 书社 automatically encapsulates H·T·M·L embeds so that their
+    metadata does not propogate up to the embedding document.
+  To undo this behaviour, remove the `@itemscope` and `@itemtype`
+    attributes from the embed during the transformation phase.
+

 The following are recommendations on effective creation of
   transforms :⁠—
 
 The following are recommendations on effective creation of
   transforms :⁠—
 


@@ -306,18 +434,118 @@ The following are recommendations on effective creation of
 - Set `@exclude-result-prefixes` on the root `xslt:transform` element
     to reduce the number of declared namespaces in the final result.
 
 - Set `@exclude-result-prefixes` on the root `xslt:transform` element
     to reduce the number of declared namespaces in the final result.
 

+## Global Params
+
+The following params are made available globally in parsers and
+  transforms :⁠—
+
+- **`BUILDTIME`:**
+  The current time.
+
+- **`CKSUM`:**
+  The checksum of the source file (⅌ `cksum`).
+
+- **`IDENTIFIER`:**
+  The ⛩️📰 书社 identifier of the source file (a u·r·i beginning with
+    `about:shushe`).
+
+- **`SRCREV`:**
+  The value of the `SRCREV` variable (if present).
+
+- **`SRCTIME`:**
+  The time at which the source file was last modified.
+
+- **`THISREV`:**
+  The value of the `THISREV` variable (if present).
+
+The following params are only available in transforms :⁠—
+
+- **`CATALOG`:**
+  The path of the catalog file (within `BUILDDIR`).
+
+- **`PATH`:**
+  The path of the output file (within `DESTDIR`).
+

 ## Output Wrapping
 
 ⛩️📰 书社 will wrap the final output of the transforms in appropriate
   `<html:html>` and `<html:body>` elements, so it is not necessary for
   transforms to do this explicitly.
 ## Output Wrapping
 
 ⛩️📰 书社 will wrap the final output of the transforms in appropriate
   `<html:html>` and `<html:body>` elements, so it is not necessary for
   transforms to do this explicitly.

-The `<html:head>` of the output will contain the result tree generated
-  by matching the root node in the `书社:metadata` mode; the provided
-  `transforms/metadata.xslt` transform uses this mode to generate basic
-  metadata, but it is possible for other transforms to add their own.
+After performing the initial transform, ⛩️📰 书社 will match the root
+  node of the result in the following modes to fill in areas of the
+  wrapper :⁠—
+
+- **`书社:header`:**
+  The result of matching in this mode is prepended into the
+    `<html:body>` of the output (before the transformation result).
+
+- **`书社:footer`:**
+  The result of matching in this mode is appended into the
+    `<html:body>` of the output (after the transformation result).
+
+- **`书社:metadata`:**
+  The result of matching in this mode is inserted into the
+    `<html:head>` of the output.
+
+In addition to being called with the transform result, each of these
+  modes will additionally be called with a `<xslt:include>` element
+  corresponding to each transform.
+If a transform has a `<书社:id>` top‐level element whose value is an
+  i·r·i, its `<xslt:include>` element will have a corresponding
+  `@书社:id` attribute.
+This mechanism can be used to allow transforms to insert content
+  without matching any elements in the result; for example, the
+  following transform adds a link to a stylesheet to the `<html:head>`
+  of every page :⁠—
+
+```xml
+<?xml version="1.0"?>
+<transform
+  xmlns="http://www.w3.org/1999/XSL/Transform"
+  xmlns:html="http://www.w3.org/1999/xhtml"
+  xmlns:xslt="http://www.w3.org/1999/XSL/Transform"
+  xmlns:书社="urn:fdc:ladys.computer:20231231:Shu1She4"
+  exclude-result-prefixes="书社"
+  version="1.0"
+>
+  <书社:id>example:add-stylesheet-links.xslt</书社:id>
+  <template match="xslt:include[@书社:id='example:add-stylesheet-links.xslt']" mode="书社:metadata">
+    <html:link rel="stylesheet" type="text/css" href="/style.css"/>
+  </template>
+</transform>
+```
+
+Output wrapping can be entirely disabled by adding a
+  `@书社:disable-output-wrapping` attribute to the top‐level element in
+  the result tree.
+This attribute will also prevent wrapping non‐H·T·M·L embeds with an
+  `<html:div>`.
+
+## Applying Attributes
+
+The `<书社:apply-attributes>` element will apply any attributes on the
+  element to the element(s) it wraps.
+It is especially useful in combination with embeds.
+
+The `<书社:apply-attributes-to-root>` element will apply any attributes
+  on the element to the root node of the final transformation result.
+It is especially useful in combination with output wrapping.
+
+In both cases, attributes from various sources are combined with
+  white·space between them.
+Attribute application takes place after all ordinary transforms have
+  completed.
+
+Both elements ignore attributes in the `xml:` namespace, except for
+  `@xml:lang`, which ignores all but the first definition (including
+  any already present on the root element).
+On H·T·M·L and S·V·G elements, `@lang` has the same behaviour as
+  `@xml:lang`.

 
 ## License
 
 Source files are licensed under the terms of the <cite>Mozilla Public
   License, version 2.0</cite>.
 For more information, see [LICENSE](./LICENSE).
 
 ## License
 
 Source files are licensed under the terms of the <cite>Mozilla Public
   License, version 2.0</cite>.
 For more information, see [LICENSE](./LICENSE).

+
+[draft-phillips-record-jar-01]: <https://datatracker.ietf.org/doc/html/draft-phillips-record-jar-01>