Introduction

This document is the source of truth for Phase 8 of the Arcology Hypermedia Publishing Platform: deploying the Arcology Web Server (arcology serve) and the Syncthing watcher (arcology sync) on the wobserver as systemd services behind nginx.

Two artifacts live here:

A generic NixOS module tangled to ../nix/module.nix, exposed as nixosModules.server from the repo flake. It knows nothing about my machines — it wraps the two CLI commands in systemd units and optionally fronts them with an nginx virtualHost.
A home-manager module tangled to ../nix/hm-module.nix, exposed as homeManagerModules.sync: the user-level sibling of the NixOS sync watcher for microVM hosts, so ~/org/arcology.db stays fresh for arcology2 notes consumers without a root-level service.
My configuration, a separate heading below, tangled to ~/nix/nixos/arcology2go.nix and exported to the Arroyo System Flake Generator via the ARROYO_NIXOS_MODULE keyword in its properties drawer.

The Arroyo System Flake Generator collects ./modulePath lines from arroyo.nixos_role_modules(role) for every ARROYO_NIXOS_MODULE entry (see arroyo.org), so adding this file's heading property is all that's needed for the host module list to pick up my configuration; the module itself is imported from the arcology2go flake input which is already declared in app/build.org's Arroyo System Integration.

The Generic Module

The module takes pkgs and exposes services.arcology2go. It does not reference the flake self — the package comes from an overridable package option so the module stays plain-importable even without the flake wrapper.

Options

services.arcology2go.enable: master switch, pulls in both services and the nginx block.
services.arcology2go.package: the arcology2 CLI derivation, best to just use the one provided by the package itself.
services.arcology2go.environmentFile: systemd EnvironmentFile; must define ARCOLOGY_SYNCTHING_API_KEY (SyncCommand errors out without it) and may define ARCOLOGY_MAPTILER_KEY, the MapTiler API key the web map pages fall back to when their ARCOLOGY_MAP property carries no key knob (knob wins when present; see maps/web.org's key-resolution order). This keeps the key out of the published org files entirely.

nix#+name: module-options:noweb-ref module-options

enable = mkEnableOption "Arcology web publishing server and sync watcher";

package = mkOption {
  type = types.package;
  # Relative to this file (nix/), so the repo-root default.nix.
  default = pkgs.callPackage ../default.nix {};
  defaultText = literalExpression "pkgs.callPackage ../default.nix {}";
  description = "The arcology2 CLI derivation.";
};

environmentFile = mkOption {
  type = types.nullOr types.path;
  default = null;
  description = ''
    EnvironmentFile loaded by both services. Must define
    ARCOLOGY_SYNCTHING_API_KEY for the sync watcher. May define
    ARCOLOGY_MAPTILER_KEY, the MapTiler API key the web map pages
    fall back to when their ARCOLOGY_MAP property has no `key=`
    knob — setting it here keeps the key out of the org files.
  '';
};

services.arcology2go.web.enable: start arcology serve.
services.arcology2go.web.orgDir: the org directory
services.arcology2go.web.dbPath: path to arcology.db.
services.arcology2go.web.domainsFile: path to a domains.json (see Arcology's Domain Map Configuration); no default, the consumer points it at their own copy.
services.arcology2go.web.port: listen port (default 8080; nginx proxies to it).
services.arcology2go.web.cacheDir — HTML cache directory (default /var/cache/arcology/html); ServeCommand's --cache-dir.
services.arcology2go.web.cacheCleanupAge — systemd-tmpfiles cleanup age on the HTML cache directory; see Module Source for the semantics. Set to "-" to disable eviction.
services.arcology2go.web.attachmentDir — crushed attachment cache (default /var/cache/arcology/attachments); maps to $ARCOLOGY_ATTACHMENT_DIR which both the indexer plugin and the server read (AttachmentCrusher.defaultCacheDir).

nix#+name: module-options:noweb-ref module-options

web = {
  enable = mkOption {
    type = types.bool;
    default = true;
    description = "Start the arcology serve web server.";
  };
  port = mkOption {
    type = types.port;
    default = 8080;
    description = "Port the web server listens on.";
  };
  dbPath = mkOption {
    type = types.str;
    default = "/var/lib/arcology/arcology.db";
    description = "Path to the arcology SQLite database.";
  };
  orgDir = mkOption {
    type = types.str;
    description = "Root org-mode directory (the Syncthing folder).";
  };
  domainsFile = mkOption {
    type = types.nullOr types.path;
    default = null;
    description = "Path to domains.json (SITE to domain mapping).";
  };
  cacheDir = mkOption {
    type = types.str;
    default = "/var/cache/arcology/html";
    description = "Rendered HTML cache directory.";
  };
  cacheCleanupAge = mkOption {
    type = types.str;
    default = "-";
    description = ''
      Cleanup age for the HTML cache directory, passed to
      systemd-tmpfiles: file entries older than this are evicted
      during systemd-tmpfiles-clean runs (age scores mtime, atime and
      ctime; a cache hit refreshes atime, so in-use entries
      effectively survive). "-" disables aging.
    '';
  };
  attachmentDir = mkOption {
    type = types.str;
    default = "/var/cache/arcology/attachments";
    description = "Crushed attachment cache directory (ARCOLOGY_ATTACHMENT_DIR).";
  };
  enableMetrics = mkOption {
    type = types.bool;
    default = true;
    description = "Enable Prometheus collector on this host.";
  };
};

services.arcology2go.sync.enable — start arcology sync.
services.arcology2go.sync.apiUrl — Syncthing REST API base URL (default http://127.0.0.1:8384).
services.arcology2go.sync.folderId — optional explicit Syncthing folder ID override; when null the watcher resolves the folder whose path matches orgDir.
services.arcology2go.sync.pollTimeoutSeconds — event long-poll timeout (default 60).

nix#+name: module-options:noweb-ref module-options

sync = {
  enable = mkOption {
    type = types.bool;
    default = true;
    description = "Start the arcology sync Syncthing watcher.";
  };
  apiUrl = mkOption {
    type = types.str;
    default = "http://127.0.0.1:8384";
    description = "Syncthing REST API base URL.";
  };
  folderId = mkOption {
    type = types.nullOr types.str;
    default = null;
    description = "Explicit Syncthing folder ID (overrides path matching).";
  };
  pollTimeoutSeconds = mkOption {
    type = types.int;
    default = 60;
    description = "Syncthing event long-poll timeout in seconds.";
  };
};

services.arcology2go.nginx.enable — add the nginx virtualHost.
services.arcology2go.nginx.virtualHost — virtualHost name; also the server_name.
services.arcology2go.nginx.serverAliases — additional server_name entries on the virtualHost, i.e. the other domains from the domain map so every published site routes here.

nix#+name: module-options:noweb-ref module-options

nginx = {
  enable = mkOption {
    type = types.bool;
    default = false;
    description = "Add an nginx virtualHost proxying to the web server.";
  };
  virtualHost = mkOption {
    type = types.nullOr types.str;
    default = null;
    description = "nginx virtualHost name (server_name).";
  };
  serverAliases = mkOption {
    type = types.listOf types.str;
    default = [];
    description = ''
      Additional server_names on the virtualHost — the extra domains
      from the domain map (arcology.garden, thelionsrear.com, etc).
    '';
  };
};

NEXT automatically populate nginx server_aliases from the domains.json or the table

Module Source

nix#+name: arcology2go-module:tangle ../nix/module.nix:mkdirp yes:noweb yes

{ config, lib, pkgs, ... }:

with lib;

let
  cfg = config.services.arcology2go;
in {
  options.services.arcology2go = {
    <<module-options>>
  };
  
  config = mkIf cfg.enable {
    # The web service owns the db and cache dirs; the sync watcher writes
    # the db (WAL mode, see roam/indexer.org DatabaseFactory) while serve
    # reads it, so both run as the same dedicated user.
    users.users.arcology = {
      isSystemUser = true;
      group = "arcology";
      description = "Arcology web publishing services";
      # A shell and home so `sudo -u arcology -i` works for manual CLI
      # maintenance runs (backfills, ad-hoc indexing) as the service user;
      # /var/lib/arcology is StateDirectory-owned by the services.
      shell = pkgs.bash;
      home = "/var/lib/arcology";
    };
    users.groups.arcology = { };

    # Put the CLI on the system PATH so `sudo -u arcology arcology2 …`
    # resolves through sudo's secure_path (/run/current-system/sw/bin)
    # instead of the invoking user's profile, which the service user
    # cannot traverse.
    environment.systemPackages = [ cfg.package ];

    systemd.services.arcology-web = mkIf cfg.web.enable {
      description = "Arcology web publishing server";
      wantedBy = [ "multi-user.target" ];
      after = [ "network.target" ];
      serviceConfig = {
        Type = "simple";
        User = "arcology";
        Group = "arcology";
        EnvironmentFile = mkIf (cfg.environmentFile != null) [ cfg.environmentFile ];
        StateDirectory = "arcology";
        CacheDirectory = [ "arcology/html" "arcology/attachments" ];
        # serve never writes outside cacheDir/attachmentDir/db; ProtectSystem
        # keeps the org dir read-only which is exactly the sync-to-publish
        # contract: org files are the source of truth.
        ProtectSystem = "strict";
        ReadWritePaths = [
          cfg.web.cacheDir
          cfg.web.attachmentDir
          (dirOf cfg.web.dbPath)
        ];
        NoNewPrivileges = true;
        PrivateTmp = true;
        Restart = "on-failure";
        RestartSec = 5;
      };
      script = ''
        ${cfg.package}/bin/arcology2 serve \
          --db ${cfg.web.dbPath} \
          --port ${toString cfg.web.port} \
          --org-dir ${cfg.web.orgDir} \
          ${optionalString (cfg.web.domainsFile != null)
            "--domains ${cfg.web.domainsFile}"} \
          --cache-dir ${cfg.web.cacheDir} \
          --attachment-dir ${cfg.web.attachmentDir}
      '';
    };

    services.prometheus.scrapeConfigs = mkIf cfg.web.enableMetrics [
      {
        job_name = "arcology2go";
        static_configs = [{ targets = [ "127.0.0.1:${toString cfg.web.port}" ]; }];
      }
    ];

    systemd.services.arcology-sync = mkIf cfg.sync.enable {
      description = "Arcology Syncthing watch indexer";
      wantedBy = [ "multi-user.target" ];
      after = [ "network.target" ];
      serviceConfig = {
        Type = "simple";
        User = "arcology";
        Group = "arcology";
        EnvironmentFile = mkIf (cfg.environmentFile != null) [ cfg.environmentFile ];
        StateDirectory = "arcology";
        ReadWritePaths = [ (dirOf cfg.web.dbPath) ];
        NoNewPrivileges = true;
        PrivateTmp = true;
        Restart = "always";
        RestartSec = 10;
      };
      script = ''
        ${cfg.package}/bin/arcology2 sync \
          --db ${cfg.web.dbPath} \
          --org-dir ${cfg.web.orgDir} \
          --api-url ${cfg.sync.apiUrl} \
          --poll-timeout ${toString cfg.sync.pollTimeoutSeconds} \
          ${optionalString (cfg.sync.folderId != null)
            "--folder ${cfg.sync.folderId}"}
      '';
    };

    systemd.tmpfiles.settings  = {
      "10-arcology" = {
        "${cfg.web.attachmentDir}" = {
          d = { group = "arcology"; mode = "0755"; user = "arcology"; };
        };
        "${cfg.web.cacheDir}" = {
          d = {
            age = cfg.web.cacheCleanupAge;
            group = "arcology";
            mode = "0755";
            user = "arcology";
          };
        };
      };
    };

    services.nginx = mkIf (cfg.nginx.enable && cfg.web.enable) {
      enable = true;
      virtualHosts.${cfg.nginx.virtualHost} = {
        serverAliases = cfg.nginx.serverAliases;
        locations."/" = {
          proxyPass = "http://127.0.0.1:${toString cfg.web.port}";
          proxyWebsockets = true;
          extraConfig = ''
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # XXX in my case we want to hardcode this because it may already have SSL stripped and the response is through TS, for now...
            # proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Proto "https";
            proxy_set_header X-Forwarded-Host $http_host;
            proxy_set_header Host $host;
          '';
        };
        # /metrics is internal-only (metrics.org); blocked at the nginx layer.
        locations."/metrics" = {
          return = "403";
        };
        locations."/health" = {
          proxyPass = "http://127.0.0.1:${toString cfg.web.port}/health";
        };
      };
    };
  };
}

Note the ../default.nix in the package default: ../nix/module.nix sits one directory below the repo root, so the callPackage path climbs one level. Consumers overriding package (like my configuration below) never hit it.

Home-Manager Module — the sync watcher for agent hosts

The wobserver runs this repo's NixOS module; the microVMs (Hermes harness hosts, per the notes-tools freshness requirement in rrix's note 20261001T212917.593002) need the same freshness guarantee on the user level. This module is the home-manager sibling of the NixOS module's arcology-sync service: a user-level systemd unit wrapping the existing arcology sync verb so ~/org/arcology.db stays current for arcology2 notes consumers without a root-level NixOS service. It shares the same verb — no second watcher implementation exists.

services.arcology2goSync.enable — start the user-level sync watcher.
services.arcology2goSync.package — the arcology2 CLI derivation (defaults to the flake consumer's override, or callPackage from the repo).
services.arcology2goSync.dbPath — default %h/org/arcology.db (the notes tools' default).
services.arcology2goSync.orgDir — default %h/org.
services.arcology2goSync.apiUrl — Syncthing REST endpoint.
services.arcology2goSync.apiKeyFile — path to a file whose content is the key value (rendered by sops-nix in my hosts); the unit exports ARCOLOGY_SYNCTHING_API_KEY read from it at start.
services.arcology2goSync.folderId, pollTimeoutSeconds — pass-through to the sync verb.

nix#+name: arcology2go-hm-module:tangle ../nix/hm-module.nix:mkdirp yes

{ config, lib, pkgs, ... }:

with lib;

let
  cfg = config.services.arcology2goSync;

  # The sync verb requires ARCOLOGY_SYNCTHING_API_KEY in its environment.
  # With apiKeyFile set, a tiny wrapper execs the real binary with the key
  # read out of the file; otherwise the CLI is invoked directly and the
  # user is expected to export the key through their session environment.
  keyWrapper = pkgs.writeShellScript "arcology-sync-keyed" ''
    export ARCOLOGY_SYNCTHING_API_KEY="$(cat ${cfg.apiKeyFile})"
    exec ${cfg.package}/bin/arcology2 sync "$@"
  '';
  syncArgs = concatStringsSep " " (
    [ "--db ${cfg.dbPath}"
      "--org-dir ${cfg.orgDir}"
      "--api-url ${cfg.apiUrl}"
    ] ++ optionals (cfg.folderId != null) [ "--folder ${cfg.folderId}" ]
      ++ [ "--poll-timeout ${toString cfg.pollTimeoutSeconds}" ]
  );
  syncExec =
    if cfg.apiKeyFile != null
    then "${keyWrapper} ${syncArgs}"
    else "${cfg.package}/bin/arcology2 sync ${syncArgs}";
in {
  options.services.arcology2goSync = {
    enable = mkEnableOption "Arcology Syncthing watch indexer (user-level)";

    package = mkOption {
      type = types.package;
      default = pkgs.callPackage ../default.nix {};
      defaultText = literalExpression "pkgs.callPackage ../default.nix {}";
      description = "The arcology2 CLI derivation.";
    };

    dbPath = mkOption {
      type = types.str;
      default = "%h/org/arcology.db";
      description = "Path to the arcology SQLite database (%h expands to the user's home).";
    };

    orgDir = mkOption {
      type = types.str;
      default = "%h/org";
      description = "Root org-mode directory (the Syncthing folder).";
    };

    apiUrl = mkOption {
      type = types.str;
      default = "http://127.0.0.1:8384";
      description = "Syncthing REST API base URL.";
    };

    apiKeyFile = mkOption {
      type = types.nullOr types.path;
      default = null;
      description = ''
        File whose content is the Syncthing API key, exported as
        ARCOLOGY_SYNCTHING_API_KEY for the sync watcher. Render with
        sops-nix or write by hand; the unit cats it at start. Without it
        the watcher relies on ARCOLOGY_SYNCTHING_API_KEY already being
        set in the user's environment.
      '';
    };

    folderId = mkOption {
      type = types.nullOr types.str;
      default = null;
      description = "Explicit Syncthing folder ID (overrides path matching).";
    };

    pollTimeoutSeconds = mkOption {
      type = types.int;
      default = 60;
      description = "Syncthing event long-poll timeout in seconds.";
    };
  };

  config = mkIf cfg.enable {
    systemd.user.services.arcology-sync = {
      Unit = {
        Description = "Arcology Syncthing watch indexer (keeps ~/org/arcology.db fresh for arcology2 notes)";
        After = [ "default.target" ];
      };
      Service = {
        Type = "simple";
        ExecStart = syncExec;
        Restart = "always";
        RestartSec = 10;
      };
      Install = { WantedBy = [ "default.target" ]; };
    };
  };
}

Running the CLI as the service user

The module adds the arcology2 derivation to environment.systemPackages and gives the arcology user a bash shell with home /var/lib/arcology, so manual maintenance runs (attachment backfills, ad-hoc indexing, cache inspections) work as the service user on the wobserver:

sudo -u arcology -i
# or one-shot, resolved through sudo's secure_path:
sudo -u arcology arcology2 crush-attachments \
  --db /var/lib/arcology/arcology.db \
  --org-dir /media/org \
  --attachment-dir /var/cache/arcology/attachments

Before this module change, sudo -u arcology arcology2 failed two ways: the account had no shell (sudo -i → "This account is currently not available"), and the bare command name resolved against the invoking user's ~/.nix-profile, which the arcology user cannot traverse. The environment.systemPackages entry puts the binary at /run/current-system/sw/bin/arcology2 where =sudo='s secure_path and the service user's PATH both find it.

My Configuration

The host module imports the flake's nixosModules.server output and configures it for the wobserver. The domains come from the domain map's site-meta table ride along as nginx.serverAliases on one virtualHost, proxied to the same backend. environmentFile points at a secret file providing ARCOLOGY_SYNCTHING_API_KEY and ARCOLOGY_MAPTILER_KEY (the MapTiler key the map pages use when their ARCOLOGY_MAP knob carries no key); the values themselves are managed outside this file.

domainsFile points at the live repo checkout rather than a copied domains.json: arcology2 tangle regenerates it in place from the domain map tables, and the server only reads it at startup, so a regenerated file takes effect on the next restart.

environmentFile comes from sops-nix, the house pattern (same as vaultwarden_env): sops.secrets entries in the server secrets file with owner set to the arcology service user the module creates, rendered secret file passed to both services as the EnvironmentFile. The yaml carries arcology-syncthing-api-key holding =ARCOLOGY_SYNCTHING_API_KEY=<key>= and arcology-maptiler-key holding =ARCOLOGY_MAPTILER_KEY=<key>= (both =KEY=value= lines in one file, or two files listed in EnvironmentFile — either shape works for systemd).

nix:tangle ~/nix/nixos/arcology2go.nix:mkdirp yes

{ inputs, pkgs, config, ... }:

{
  imports = [ inputs.arcology2go.nixosModules.server ];

  sops.secrets.arcology-syncthing-api-key.owner = "arcology";

  services.arcology2go = {
    enable = true;
    package = inputs.arcology2go.packages.${pkgs.stdenv.hostPlatform.system}.default;
    environmentFile = config.sops.secrets.arcology-syncthing-api-key.path;
    web = {
      enable = true;
      port = 8377;
      orgDir = "/media/org";
      # dbPath = "/media/org/arcology.db";
      domainsFile = "/media/org/arcology2go/web/domains.json";
      cacheDir = "/var/cache/arcology/html";
      cacheCleanupAge = "7d";
      attachmentDir = "/var/cache/arcology/attachments";
    };
    sync = {
      enable = true;
      apiUrl = "http://127.0.0.1:8384";
      pollTimeoutSeconds = 60;
    };
    nginx = {
      enable = true;
      virtualHost = "arcology.whatthefuck.computer";
      serverAliases = [
        "rix.si"
        "thelionsrear.com"
        "whatthefuck.computer"
        "arcology.garden"
        "engine.arcology.garden"
        "cce.whatthefuck.computer"
        "thechanceencounter.com"
      ];
    };
  };
  services.nginx.virtualHosts."arcology.whatthefuck.computer" = {
    addSSL = true;
    useACMEHost = "fontkeming.fail";
    locations."~ ^/~(.+?)(/.*)?$".extraConfig = ''
      index index.html index.htm;
      alias /home/$1/public_html$2;
      autoindex on;
    '';
  };
}

Known Gaps

The users.users.arcology is created unconditionally under cfg.enable even if only nginx is enabled; a refinement would tie user creation to web.enable || sync.enable.
The nginx block assumes services.nginx.enable is already on (true on the wobserver); a weaker module would also mkIf the nginx option into existence.
The wobserver must have a syncthing service sharing the orgDir; the module =requires=/=wants= it but does not configure it.
The sops.secrets.arcology-syncthing-api-key entry must exist in the target host's sops file (secrets/server.yaml for the wobserver) before the first rebuild; sops-nix renders it to /run/secrets/arcology-syncthing-api-key and the module passes its path to both services.

Footnotes

This file is tangled with arcology2 tangle web/deployment.org; the ~/nix targets expand tilde via the tangle tool's $HOME expansion (arroyo.org tangle knobs).

Arcology Engine

Arcology CCE Deployment

Introduction

The Generic Module

Options

NEXT automatically populate nginx server_aliases from the domains.json or the table

Module Source

Home-Manager Module — the sync watcher for agent hosts

Running the CLI as the service user

My Configuration

Known Gaps

Footnotes