Enable two‐stage builds using a “data” directory

One pattern for building sites is to use a large number of data files
which are all compiled into a single large file, which is then used to
derive the various pages of the website (using expanded archive
functionality). While this can be done using ordinary includes, it is
expensive to do so, because inclusions and transformations on the
“single large file” will re‐happen every time that it is included. What
is actually desired in this case is an intermediate data file, which
has already had all transformations applied, which can then be included
into other files.

This commit establishes a pattern for doing so, where·by certain files
(conventionally, by file extension; by default, `.rdf`) can be
designated as “data files” and built during an initial pass using
includes from a “data directory”, and then themselves be available as
includes for the second stage. Installing installs files from both
stages. Other commands (such as `listout`) currently ignore data files
because they don’t follow common assumptions (they are not built to
`$(BUILDDIR)/stage2/public`, for example); variants or flags to enable
listing these files may be fruitful future work.

diff --git a/GNUmakefile b/GNUmakefile
index 3d4bf87ed64f813841a8eeedbab006f1bafdd2e0..105b403510b9fe98fc60b935af5ab4d48d2aec97 100644 (file)
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -149,11 +149,21 @@ SRCDIR := sources
 # Multiple directories can be given so long as files with the same name do not exist in each.
 #
 # These can be inside of `SRCDIR´ directories if desired.
+#
+# This variable is also used by the two‐stage `MODE´, where it gives the include directory for the second stage.
 INCLUDEDIR := $(foreach dir,$(SRCDIR),$(dir)/includes)
 
+# The directory which contains the data files.
+#
+# Only used in the two‐stage `MODE´, and defaults to that `MODE´ if this directory exists.
+# This directory is used as the include directory in the first stage.
+#
+# Multiple directories can be given so long as files with the same name do not exist in each.
+DATADIR :=
+
 # The directory in which to generate temporary buildfiles.
 #
-# This variable is also used by the archiving `MODE´.
+# This variable is also used by the archiving and two‐stage `MODE´s.
 BUILDDIR := build
 
 # The directory into which to output files on `make install´.
@@ -165,17 +175,28 @@ DESTDIR := public
 #
 # By default, this is inferred from the variable `MAKEFILE_LIST´.
 #
-# This variable is also used by the archiving `MODE´.
+# This variable is also used by the archiving and two‐stage `MODE´s.
 THISDIR := $(patsubst %/,%,$(dir $(lastword $(MAKEFILE_LIST))))
 
 # Configuration of `find´.
 #
 # By default, `find´ will ignore files which begin with a period and those which are likely to cause problems for `make´.
+#
+# These variables are also used by the two‐stage `MODE´.
 EXTRAFINDRULES :=
 EXTRAFINDINCLUDERULES :=
-FINDRULES := ! '(' '(' -name '[.-]*' -a ! -name '.' -o -name '*[][*?:|$$%\#\\; ]*' -o -name '*[)]' ')' -a -prune ')'$(if $(EXTRAFINDRULES), -a '(' $(EXTRAFINDRULES) ')',)
+FINDRULES := '!' '(' '(' -name '[.-]*' -a '!' -name '.' -o -name '*[][*?:|$$%\#\\; ]*' -o -name '*[)]' ')' -a -prune ')'$(if $(EXTRAFINDRULES), -a '(' $(EXTRAFINDRULES) ')',)
 FINDINCLUDERULES := $(FINDRULES)$(if $(EXTRAFINDINCLUDERULES), -a '(' $(EXTRAFINDINCLUDERULES) ')',)
 
+# File extensions which indicate files in `SRCDIR´ which should be built as part of the first, rather than second, stage of the two‐stage `MODE´.
+DATAEXT := rdf
+
+# Rules for identifying files in `SRCDIR´ which should be built as part of the first, rather than second, stage of the two‐stage `MODE´.
+#
+# By default, this just constructs rules from `DATAEXT´.
+EXTRAFINDDATARULES :=
+FINDDATARULES := -name '.' $(foreach ext,$(DATAEXT), -o -name '$(subst ','"'"',[!.]*.$(ext))')$(if $(EXTRAFINDDATARULES), -a '(' $(EXTRAFINDDATARULES) ')',)
+
 # The list of magic files to use when determining media types.
 #
 # Some are provided as part of this repository, but you can add more if you need different media type detection.
@@ -233,11 +254,14 @@ endif
 # The following modes are available :—
 #
 # • ‹ urn:fdc:ladys.computer:20231231:Shu1She4:mode:default ›:
-#   The default mode; typical Shushe behaviours.
+#   The default mode; typical ⛩📰 书社 behaviours.
 #
 # • ‹ urn:fdc:ladys.computer:20231231:Shu1She4:mode:archive ›:
 #   Generates archive files from parse results.
-MODE := urn:fdc:ladys.computer:20231231:Shu1She4:mode:default
+#
+# • ‹ urn:fdc:ladys.computer:20231231:Shu1She4:mode:_2stage ›:
+#   Two‐stage build; runs ⛩📰 书社 twice.
+MODE := urn:fdc:ladys.computer:20231231:Shu1She4:mode:$(if $(DATADIR),$(shell if $(TEST) -d $(call quote,$(DATADIR)); then $(PRINTF) '%s\n' '_2stage'; else $(PRINTF) '%s\n' 'default'; fi),default)
 
 # Set to a non·empty value to silence informative messages.
 QUIET :=
@@ -589,10 +613,10 @@ FORCE : ;
 .SUFFIXES : ;
 
 # Don’t delete these files even if Make is stopped in the process of rebuilding them.
-.PRECIOUS : GNUmakefile ;
+.PRECIOUS : $(THISDIR)/GNUmakefile ;
 
 # Phony rules; always consider these out·of·date.
-.PHONY : FORCE all default clean gone info install list listout uninstall $(call built,$(recursivefiles)) ;
+.PHONY : FORCE all clean gone help install list listout uninstall $(call built,$(recursivefiles)) ;
 
 ifeq ($(notbuilding),)
 # Reload this make·file if the magic file, parser, dependencies, or destinations have changed.
@@ -707,6 +731,12 @@ endif
 
 # ─ ¶ Special Targets ─────────────────────────────────────────────────
 
+# Don’t use any implicit rules.
+.SUFFIXES : ;
+
+# Don’t delete these files even if Make is stopped in the process of rebuilding them.
+.PRECIOUS : $(THISDIR)/GNUmakefile ;
+
 # Reload this make·file if the archive components have changed.
 $(THISDIR)/GNUmakefile : $(BUILDDIR)/index
        $(silent)$(TOUCH) $(THISDIR)/GNUmakefile
@@ -746,6 +776,82 @@ $(DESTDIR)/% : $(SRC) $(foreach file,$(archivefiles),$(BUILDDIR)/files/$(file))
        $(silent)$(RM) -f -R $(call quote,$@)
        $(silent)$(CD) $(call quote,$(BUILDDIR)/files); if $(call xpath,/*/@*[local-name()="expanded" and namespace-uri()="urn:fdc:ladys.computer:20231231:Shu1She4"],$(abspath $<)); then $(MKDIR) -p $(call quote,$(abspath $@)); $(PRINTF) '%s\n' $(foreach file,$(archivefiles),$(call quote,$(file))) | $(PAX) -r -w $(call quote,$(abspath $@)); else $(PRINTF) '%s\n' $(foreach file,$(archivefiles),$(call quote,$(file))) | $(PAX) -w -x ustar >|$(call quote,$(abspath $@)); fi
 
+# ━ § BEGIN TWO‐STEP MAKE·FILE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+else ifeq ($(MODE),urn:fdc:ladys.computer:20231231:Shu1She4:mode:_2stage)
+
+# ─ ¶ Non‐Recipe Variable Definitions ─────────────────────────────────
+
+override makefile := $(abspath $(THISDIR)/GNUmakefile)
+
+# (overridable) Options to use when calling ⛩📰 书社 the first time.
+shushedataopts := INCLUDEDIR=$(call quote,$(DATADIR)) BUILDDIR=$(call quote,$(BUILDDIR)/stage1) DESTDIR=$(call quote,$(BUILDDIR)/data) FINDRULES=$(call quote,$(FINDRULES) -a '(' $(FINDDATARULES) ')') FINDINCLUDERULES=$(call quote,$(FINDINCLUDERULES)) MODE='urn:fdc:ladys.computer:20231231:Shu1She4:mode:default'
+
+# (overridable) Options to use when calling ⛩📰 书社 the second time.
+shushesiteopts := INCLUDEDIR=$(call quote,$(INCLUDEDIR) $(BUILDDIR)/data) BUILDDIR=$(call quote,$(BUILDDIR)/stage2) FINDRULES=$(call quote,$(FINDRULES) -a '!' '(' $(FINDDATARULES) ')') FINDINCLUDERULES=$(call quote,$(FINDINCLUDERULES)) MODE='urn:fdc:ladys.computer:20231231:Shu1She4:mode:default'
+
+# ─ ¶ Recipe Variable Definitions ─────────────────────────────────────
+
+# ─ ¶ Phony Targets ───────────────────────────────────────────────────
+
+# Compile all files, or error if any are recursive.
+all : data
+       @$(MAKE) -f $(call quote,$(makefile)) $(shushesiteopts)
+
+# Destroy buildfiles.
+clean :
+       $(if $(BUILDDIR),$(silent)$(RM) -f -R $(call quote,$(BUILDDIR)/),)
+
+# Build the data and remove outdated data files.
+data : $(BUILDDIR)/data.out
+       @$(MAKE) -f $(call quote,$(makefile)) install $(shushedataopts)
+       $(silent)$(FIND) $(call quote,$(BUILDDIR)/data) '!' -exec $(GREP) -F -q -x '{}' $(call quote,$<) ';' -a '(' -type d -o -print ')' | $(xargsmultiquote) | $(XARGS) -E '' $(RM)
+
+# Destroy build directory and installed files.
+gone : clean uninstall ;
+
+# Install the compiled files into `DESTDIR´.
+install : all
+       @$(MAKE) -f $(call quote,$(makefile)) $@ $(shushesiteopts)
+       @$(MAKE) -f $(call quote,$(makefile)) $(foreach dest,$(patsubst $(BUILDDIR)/data/%,$(DESTDIR)/%,$(shell $(CAT) $(call quote,$(BUILDDIR)/data.out))),$(call quote,$(dest)))
+
+# List all source files and includes and their computed types.
+list listout : data
+       @$(MAKE) -f $(call quote,$(makefile)) $@ $(shushesiteopts)
+
+# Destroy installed files.
+uninstall :
+       @$(MAKE) -f $(call quote,$(makefile)) $@ $(shushesiteopts)
+
+# Add as a prerequisite to treat the target as tho it were phony.
+FORCE : ;
+
+# ─ ¶ Special Targets ─────────────────────────────────────────────────
+
+# Don’t use any implicit rules.
+.SUFFIXES : ;
+
+# Phony rules; always consider these out·of·date.
+.PHONY : FORCE all data clean gone install list listout uninstall ;
+
+# ─ ¶ Build Targets ───────────────────────────────────────────────────
+
+$(BUILDDIR)/data.out : FORCE
+       @$(MAKE) -f $(call quote,$(makefile)) $(shushedataopts)
+       @$(MAKE) -s -f $(call quote,$(makefile)) listout QUIET=1 $(shushedataopts) | $(TR) ' ' '\n' | $(xargsmultiquote) | $(XARGS) -E '' $(PRINTF) $(call quote,$(BUILDDIR)/data/%s\n) >$(call quote,$(BUILDDIR)/data.out)
+
+$(BUILDDIR)/stage1/% : FORCE
+       @$(MAKE) -f $(call quote,$(makefile)) $@ $(shushedataopts)
+
+$(BUILDDIR)/data/% : FORCE
+       @$(MAKE) -f $(call quote,$(makefile)) $@ $(shushedataopts)
+
+$(BUILDDIR)/stage2/% : data
+       @$(MAKE) -f $(call quote,$(makefile)) $@ $(shushesiteopts)
+
+$(DESTDIR)/% : data
+       $(if $(shell $(GREP) -F -x $(call quote,$(BUILDDIR)/data/$*) $(BUILDDIR)/data.out),$(call inform $(PRINTF) '%s\n' $(call quote,Copying over </$*>…) >&2)$(newline),$(CP) $(call quote,$(BUILDDIR)/data/$*) $(call quote,$@),@$(MAKE) -f $(call quote,$(makefile)) $@ $(shushesiteopts))
+
 # ━ § END DEFINED MAKE·FILE MODES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 
 else

diff --git a/README.markdown b/README.markdown
index 8ae0a04f5bad55f6d2ca8d604ca5ac1594d6c6bd..503a16338dda83163e59c655e8c9a02cf00d66ec 100644 (file)
--- a/README.markdown
+++ b/README.markdown
@@ -244,6 +244,14 @@ The following additional variables can be used to control the behaviour
   Multiple include directories can be provided, so long as the same
     file subpath doesn’t exist in more than one of them.
 
+- **`DATADIR`:**
+  If set to the location of a directory, ⛩📰 书社 will run a two‐stage build.
+  In the first stage, only files in `SRCDIR` which match `FINDDATARULES` (see below) will be built, with files in `DATADIR` serving as includes.
+  In the second stage, the remaining files in `SRCDIR` will be built, with the files built during the first stage, in addition to any files in `INCLUDEDIR`, serving as includes.
+  Files built during the first stage are copied into `DESTDIR` alongside those from the second stage when installing.
+
+  This functionality is intended for sites where the bulk of the site can be built from a few data files which are expensive to create.
+
 - **`BUILDDIR`:**
   The location of the (temporary) build directory (default: `build`).
   `make clean` will delete this, and it is recommended that it not be
@@ -299,6 +307,18 @@ The following additional variables can be used to control the behaviour
     default, to enable additional rules without overriding the existing
     ones.
 
+- **`DATAEXT`:**
+  A list of file extensions which signify “data” files during a two‐stage build using `DATADIR`.
+
+- **`FINDDATARULES`:**
+  Rules to use with `find` when searching for data files.
+  By default, these rules are derived from `DATAEXT`.
+
+- **`EXTRAFINDDATARULES`:**
+  The value of this variable is appended to `FINDDATARULES` by
+    default, to enable additional rules without overriding the existing
+    ones.
+
 - **`PARSERS`:**
   A white·space‐separated list of parsers to use (default:
     `$(THISDIR)/parsers/*.xslt`).