Introduction

This document is the source of truth for Phase 8 of the Arcology Hypermedia Publishing Platform: deploying the Arcology Web Server (arcology serve) and the Syncthing watcher (arcology sync) on the wobserver as systemd services behind nginx.

Two artifacts live here:

A generic NixOS module tangled to ../nix/module.nix, exposed as nixosModules.server from the repo flake. It knows nothing about my machines — it wraps the two CLI commands in systemd units and optionally fronts them with an nginx virtualHost.
My configuration, a separate heading below, tangled to ~/nix/nixos/arcology2go.nix and exported to the Arroyo System Flake Generator via the ARROYO_NIXOS_MODULE keyword in its properties drawer.

The Arroyo System Flake Generator collects ./modulePath lines from arroyo.nixos_role_modules(role) for every ARROYO_NIXOS_MODULE entry (see arroyo.org), so adding this file's heading property is all that's needed for the host module list to pick up my configuration; the module itself is imported from the arcology2go flake input which is already declared in app/build.org's Arroyo System Integration.

The Generic Module

The module takes pkgs and exposes services.arcology2go. It does not reference the flake self — the package comes from an overridable package option so the module stays plain-importable even without the flake wrapper.

Options

services.arcology2go.enable: master switch, pulls in both services and the nginx block.
services.arcology2go.package: the arcology2 CLI derivation, best to just use the one provided by the package itself.
services.arcology2go.environmentFile: systemd EnvironmentFile; must define ARCOLOGY_SYNCTHING_API_KEY (SyncCommand errors out without it).

nix#+name: module-options:noweb-ref module-options

enable = mkEnableOption "Arcology web publishing server and sync watcher";

package = mkOption {
  type = types.package;
  # Relative to this file (nix/), so the repo-root default.nix.
  default = pkgs.callPackage ../default.nix {};
  defaultText = literalExpression "pkgs.callPackage ../default.nix {}";
  description = "The arcology2 CLI derivation.";
};

environmentFile = mkOption {
  type = types.nullOr types.path;
  default = null;
  description = ''
    EnvironmentFile loaded by both services. Must define
    ARCOLOGY_SYNCTHING_API_KEY for the sync watcher.
  '';
};

services.arcology2go.web.enable: start arcology serve.
services.arcology2go.web.orgDir: the org directory
services.arcology2go.web.dbPath: path to arcology.db.
services.arcology2go.web.domainsFile: path to a domains.json (see Arcology's Domain Map Configuration); no default, the consumer points it at their own copy.
services.arcology2go.web.port: listen port (default 8080; nginx proxies to it).
services.arcology2go.web.cacheDir — HTML cache directory (default /var/cache/arcology/html); ServeCommand's --cache-dir.
services.arcology2go.web.attachmentDir — crushed attachment cache (default /var/cache/arcology/attachments); maps to $ARCOLOGY_ATTACHMENT_DIR which both the indexer plugin and the server read (AttachmentCrusher.defaultCacheDir).

nix#+name: module-options:noweb-ref module-options

web = {
  enable = mkOption {
    type = types.bool;
    default = true;
    description = "Start the arcology serve web server.";
  };
  port = mkOption {
    type = types.port;
    default = 8080;
    description = "Port the web server listens on.";
  };
  dbPath = mkOption {
    type = types.str;
    default = "/var/lib/arcology/arcology.db";
    description = "Path to the arcology SQLite database.";
  };
  orgDir = mkOption {
    type = types.str;
    description = "Root org-mode directory (the Syncthing folder).";
  };
  domainsFile = mkOption {
    type = types.nullOr types.path;
    default = null;
    description = "Path to domains.json (SITE to domain mapping).";
  };
  cacheDir = mkOption {
    type = types.str;
    default = "/var/cache/arcology/html";
    description = "Rendered HTML cache directory.";
  };
  attachmentDir = mkOption {
    type = types.str;
    default = "/var/cache/arcology/attachments";
    description = "Crushed attachment cache directory (ARCOLOGY_ATTACHMENT_DIR).";
  };
};

services.arcology2go.sync.enable — start arcology sync.
services.arcology2go.sync.apiUrl — Syncthing REST API base URL (default http://127.0.0.1:8384).
services.arcology2go.sync.folderId — optional explicit Syncthing folder ID override; when null the watcher resolves the folder whose path matches orgDir.
services.arcology2go.sync.pollTimeoutSeconds — event long-poll timeout (default 60).

nix#+name: module-options:noweb-ref module-options

sync = {
  enable = mkOption {
    type = types.bool;
    default = true;
    description = "Start the arcology sync Syncthing watcher.";
  };
  apiUrl = mkOption {
    type = types.str;
    default = "http://127.0.0.1:8384";
    description = "Syncthing REST API base URL.";
  };
  folderId = mkOption {
    type = types.nullOr types.str;
    default = null;
    description = "Explicit Syncthing folder ID (overrides path matching).";
  };
  pollTimeoutSeconds = mkOption {
    type = types.int;
    default = 60;
    description = "Syncthing event long-poll timeout in seconds.";
  };
};

services.arcology2go.nginx.enable — add the nginx virtualHost.
services.arcology2go.nginx.virtualHost — virtualHost name; also the server_name.
services.arcology2go.nginx.serverAliases — additional server_name entries on the virtualHost, i.e. the other domains from the domain map so every published site routes here.

nix#+name: module-options:noweb-ref module-options

nginx = {
  enable = mkOption {
    type = types.bool;
    default = false;
    description = "Add an nginx virtualHost proxying to the web server.";
  };
  virtualHost = mkOption {
    type = types.nullOr types.str;
    default = null;
    description = "nginx virtualHost name (server_name).";
  };
  serverAliases = mkOption {
    type = types.listOf types.str;
    default = [];
    description = ''
      Additional server_names on the virtualHost — the extra domains
      from the domain map (arcology.garden, thelionsrear.com, etc).
    '';
  };
};

NEXT automatically populate nginx server_aliases from the domains.json or the table

Module Source

nix#+name: arcology2go-module:tangle ../nix/module.nix:mkdirp yes:noweb yes

{ config, lib, pkgs, ... }:

with lib;

let
  cfg = config.services.arcology2go;
in {
  options.services.arcology2go = {
    <<module-options>>
  };
  
  config = mkIf cfg.enable {
    # The web service owns the db and cache dirs; the sync watcher writes
    # the db (WAL mode, see roam/indexer.org DatabaseFactory) while serve
    # reads it, so both run as the same dedicated user.
    users.users.arcology = {
      isSystemUser = true;
      group = "arcology";
      description = "Arcology web publishing services";
      # A shell and home so `sudo -u arcology -i` works for manual CLI
      # maintenance runs (backfills, ad-hoc indexing) as the service user;
      # /var/lib/arcology is StateDirectory-owned by the services.
      shell = pkgs.bash;
      home = "/var/lib/arcology";
    };
    users.groups.arcology = { };

    # Put the CLI on the system PATH so `sudo -u arcology arcology2 …`
    # resolves through sudo's secure_path (/run/current-system/sw/bin)
    # instead of the invoking user's profile, which the service user
    # cannot traverse.
    environment.systemPackages = [ cfg.package ];

    systemd.services.arcology-web = mkIf cfg.web.enable {
      description = "Arcology web publishing server";
      wantedBy = [ "multi-user.target" ];
      after = [ "network.target" ];
      serviceConfig = {
        Type = "simple";
        User = "arcology";
        Group = "arcology";
        EnvironmentFile = mkIf (cfg.environmentFile != null) [ cfg.environmentFile ];
        StateDirectory = "arcology";
        CacheDirectory = [ "arcology/html" "arcology/attachments" ];
        # serve never writes outside cacheDir/attachmentDir/db; ProtectSystem
        # keeps the org dir read-only which is exactly the sync-to-publish
        # contract: org files are the source of truth.
        ProtectSystem = "strict";
        ReadWritePaths = [
          cfg.web.cacheDir
          cfg.web.attachmentDir
          (dirOf cfg.web.dbPath)
        ];
        NoNewPrivileges = true;
        PrivateTmp = true;
        Restart = "on-failure";
        RestartSec = 5;
      };
      script = ''
        ${cfg.package}/bin/arcology2 serve \
          --db ${cfg.web.dbPath} \
          --port ${toString cfg.web.port} \
          --org-dir ${cfg.web.orgDir} \
          ${optionalString (cfg.web.domainsFile != null)
            "--domains ${cfg.web.domainsFile}"} \
          --cache-dir ${cfg.web.cacheDir} \
          --attachment-dir ${cfg.web.attachmentDir}
      '';
    };

    systemd.services.arcology-sync = mkIf cfg.sync.enable {
      description = "Arcology Syncthing watch indexer";
      wantedBy = [ "multi-user.target" ];
      after = [ "network.target" ];
      serviceConfig = {
        Type = "simple";
        User = "arcology";
        Group = "arcology";
        EnvironmentFile = mkIf (cfg.environmentFile != null) [ cfg.environmentFile ];
        StateDirectory = "arcology";
        ReadWritePaths = [ (dirOf cfg.web.dbPath) ];
        NoNewPrivileges = true;
        PrivateTmp = true;
        Restart = "always";
        RestartSec = 10;
      };
      script = ''
        ${cfg.package}/bin/arcology2 sync \
          --db ${cfg.web.dbPath} \
          --org-dir ${cfg.web.orgDir} \
          --api-url ${cfg.sync.apiUrl} \
          --poll-timeout ${toString cfg.sync.pollTimeoutSeconds} \
          ${optionalString (cfg.sync.folderId != null)
            "--folder ${cfg.sync.folderId}"}
      '';
    };

    systemd.tmpfiles.settings  = {
      "10-arcology" = {
        "${cfg.web.attachmentDir}" = {
          d = { group = "arcology"; mode = "0755"; user = "arcology"; };
        };
        "${cfg.web.cacheDir}" = {
          d = { group = "arcology"; mode = "0755"; user = "arcology"; };
        };
      };
    };

    services.nginx = mkIf (cfg.nginx.enable && cfg.web.enable) {
      enable = true;
      virtualHosts.${cfg.nginx.virtualHost} = {
        serverAliases = cfg.nginx.serverAliases;
        locations."/" = {
          proxyPass = "http://127.0.0.1:${toString cfg.web.port}";
          proxyWebsockets = true;
          extraConfig = ''
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # XXX in my case we want to hardcode this because it may already have SSL stripped and the response is through TS, for now...
            # proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Proto "https";
            proxy_set_header X-Forwarded-Host $http_host;
            proxy_set_header Host $host;
          '';
        };
        # /metrics is internal-only (metrics.org); blocked at the nginx layer.
        locations."/metrics" = {
          return = "403";
        };
        locations."/health" = {
          proxyPass = "http://127.0.0.1:${toString cfg.web.port}/health";
        };
      };
    };
  };
}

Note the ../default.nix in the package default: ../nix/module.nix sits one directory below the repo root, so the callPackage path climbs one level. Consumers overriding package (like my configuration below) never hit it.

Running the CLI as the service user

The module adds the arcology2 derivation to environment.systemPackages and gives the arcology user a bash shell with home /var/lib/arcology, so manual maintenance runs (attachment backfills, ad-hoc indexing, cache inspections) work as the service user on the wobserver:

sudo -u arcology -i
# or one-shot, resolved through sudo's secure_path:
sudo -u arcology arcology2 crush-attachments \
  --db /var/lib/arcology/arcology.db \
  --org-dir /media/org \
  --attachment-dir /var/cache/arcology/attachments

Before this module change, sudo -u arcology arcology2 failed two ways: the account had no shell (sudo -i → "This account is currently not available"), and the bare command name resolved against the invoking user's ~/.nix-profile, which the arcology user cannot traverse. The environment.systemPackages entry puts the binary at /run/current-system/sw/bin/arcology2 where =sudo='s secure_path and the service user's PATH both find it.

My Configuration

The host module imports the flake's nixosModules.server output and configures it for the wobserver. The domains come from the domain map's site-meta table ride along as nginx.serverAliases on one virtualHost, proxied to the same backend. environmentFile points at a secret file providing ARCOLOGY_SYNCTHING_API_KEY (and nothing else); the value itself is managed outside this file.

domainsFile points at the live repo checkout rather than a copied domains.json: arcology2 tangle regenerates it in place from the domain map tables, and the server only reads it at startup, so a regenerated file takes effect on the next restart.

environmentFile comes from sops-nix, the house pattern (same as vaultwarden_env): a sops.secrets.arcology-syncthing-api-key entry in the server secrets file, with owner set to the arcology service user the module creates, and the rendered secret file passed to both services as the EnvironmentFile. The key in the yaml file is arcology-syncthing-api-key holding =ARCOLOGY_SYNCTHING_API_KEY=<key>=.

nix:tangle ~/nix/nixos/arcology2go.nix:mkdirp yes

{ inputs, pkgs, config, ... }:

{
  imports = [ inputs.arcology2go.nixosModules.server ];

  sops.secrets.arcology-syncthing-api-key.owner = "arcology";

  services.arcology2go = {
    enable = true;
    package = inputs.arcology2go.packages.${pkgs.stdenv.hostPlatform.system}.default;
    environmentFile = config.sops.secrets.arcology-syncthing-api-key.path;
    web = {
      enable = true;
      port = 8377;
      orgDir = "/media/org";
      # dbPath = "/media/org/arcology.db";
      domainsFile = "/media/org/arcology2go/web/domains.json";
      cacheDir = "/var/cache/arcology/html";
      attachmentDir = "/var/cache/arcology/attachments";
    };
    sync = {
      enable = true;
      apiUrl = "http://127.0.0.1:8384";
      pollTimeoutSeconds = 60;
    };
    nginx = {
      enable = true;
      virtualHost = "arcology.whatthefuck.computer";
      serverAliases = [
        "rix.si"
        "thelionsrear.com"
        "whatthefuck.computer"
        "arcology.garden"
        "engine.arcology.garden"
        "cce.whatthefuck.computer"
        "thechanceencounter.com"
      ];
    };
  };
  services.nginx.virtualHosts."arcology.whatthefuck.computer" = {
    addSSL = true;
    useACMEHost = "fontkeming.fail";
    locations."~ ^/~(.+?)(/.*)?$".extraConfig = ''
      index index.html index.htm;
      alias /home/$1/public_html$2;
      autoindex on;
    '';
  };
}

Known Gaps

The users.users.arcology is created unconditionally under cfg.enable even if only nginx is enabled; a refinement would tie user creation to web.enable || sync.enable.
The nginx block assumes services.nginx.enable is already on (true on the wobserver); a weaker module would also mkIf the nginx option into existence.
The wobserver must have a syncthing service sharing the orgDir; the module =requires=/=wants= it but does not configure it.
The sops.secrets.arcology-syncthing-api-key entry must exist in the target host's sops file (secrets/server.yaml for the wobserver) before the first rebuild; sops-nix renders it to /run/secrets/arcology-syncthing-api-key and the module passes its path to both services.

Footnotes

This file is tangled with arcology2 tangle web/deployment.org; the ~/nix targets expand tilde via the tangle tool's $HOME expansion (arroyo.org tangle knobs).

Arcology Engine

Arcology CCE Deployment

Introduction

The Generic Module

Options

NEXT automatically populate nginx server_aliases from the domains.json or the table

Module Source

Running the CLI as the service user

My Configuration

Known Gaps

Footnotes