The Arcology Site Engine

Arcology Poetry Pyproject

• Life • Tech • Emacs • Topics • Arcology •

Okay so the Arcology FastAPI package is built with poetry. I run the commands, look at the output, and copy it back in here… This is not very ergonomic right now, but I don't have a better idea on how to manage these literately.

[tool.poetry]
name = "arcology"
version = "0.1.0"
description = "The Arcology is a Multi-domain Web Site Engine for Org Roam Files"
authors = ["Ryan Rix <code@whatthefuck.computer>"]

include = ["static", "templates", "pandoc"]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Arcology Python Dependencies

Here's what I use:

I don't have a good workflow for bringing this literate doc in sync to the tools output regularly, to de-tangle changes made by poetry add --lock'ing new dependencies or updating their versions. The best bet is to run poetry add --lock and then nix-shell in – this will update the package environment and lock file, and then the updated pyproject.toml will need to be de-tangled back to here by-hand…

[tool.poetry.dependencies]
python = "^3.9"
fastapi = "^0.70"
uvicorn = "^0.16"
sqlmodel = "^0.0.5"
# https://github.com/tiangolo/sqlmodel/issues/315
sqlalchemy = "1.4.35"
sexpdata = "^0.0.3"
pypandoc = "^1.7"
Jinja2 = "^3.0"
prometheus-fastapi-instrumentator = "^5.7"
asyncinotify = "^2.0.2"
transitions = "^0.8.11"
graphviz = "^0.19.1"
pygraphviz = "^1.9"
async-lru = "^1.0.3"

[tool.poetry.dev-dependencies]
ipdb = "^0.13"

Poetry creates Startup Scripts in PATH

Poetry can create wrapper scripts for modules in the package which will be automatically stuck in to PATH or the appropriate spot.

[tool.poetry.scripts]
arcology-inotify = 'arcology.inotify:start'
arcology-fastapi = 'arcology.server:start'

The arcology inotify-watcher is invoked simply, right? it's just a little thing, doesn't even need command line arguments since it's configured in the environment. Same with the Arcology FastAPI server!

Nix Derivations

poetry2nix will package the Arcology application up based on Poetry TOML in default.nix

The Poetry application is factored out so that it can be used in the nix-shell below. Using poetry2nix to extract the package information from pyproject.toml is a pretty simple affair with poetry2nix bundled with nixpkgs.

{ pkgs ? import <nixpkgs> {},
  poetry2nix ? pkgs.poetry2nix,
  stdenv ? pkgs.stdenv,
  python ? pkgs.python3 }:

let
  mkPoetryApplication' =
    <<mkPoetryApplicationPrime>>
  ;
in
mkPoetryApplication' {
  inherit python;
  projectDir = ./.;
  propagatedBuildInputs = [
    pkgs.coreutils
    pkgs.pandoc
  ];
  overrides = [
    (poetry2nix.defaultPoetryOverrides.overrideOverlay (
      self: super: {
        sqlmodel = super.sqlmodel.overridePythonAttrs (
          old: {
            buildInputs = (old.buildInputs or [ ]) ++ [self.poetry-core];
            patchPhase = (old.patchPhase or "") + ''
              # fix pyproject.toml version?
              substituteInPlace pyproject.toml --replace 'version = "0"' 'version = "${old.version}"'
            '';
          }
        );
        traitlets = super.traitlets.overridePythonAttrs (
          old: {
            buildInputs = (old.buildInputs or [ ]) ++ [self.hatchling];
          }
        );
        pypandoc = super.pypandoc.overridePythonAttrs (
          old: {
            buildInputs = (old.buildInputs or [ ]) ++ [self.poetry-core];
          }
        );
        asyncinotify = super.asyncinotify.overridePythonAttrs (
          old: {
            buildInputs = (old.buildInputs or [ ]) ++ [self.setuptools];
          }
        );
        sqlalchemy2-stubs = super.sqlalchemy2-stubs.overridePythonAttrs (
          old: {
            buildInputs = (old.buildInputs or [ ]) ++ [self.setuptools];
          }
        );
      }
    ))
  ];
  editablePackageSources = {
    arcology = ./.;
  };
}

mkPoetryApplication' exists to allow for unified editable source shell and application

From nix-community/poetry2nix#423:

I am developing a few applications using this awesome projects. One thing that bothered me for a while that it takes a lot of code duplication to maintain a derivation for packaging using mkPoetryApplication and a second one for nix-shell using e.g. mkPoetryEnv. Unfortunately there is no easy way to override the mkPoetryPackages call in mkPoetryApplication to pass in editablePackageSources and mkPoetryEnv doesn't support the same hooks that buildPythonPackage supports. In my packages I e.g. need to call makeWrapper on the generated executables which cannot be easily done currently.

I use the following hack to get around this limitation:

and so do i. This can be included as a noweb (or, i guess, properly imported but we hackin' here.)

{ projectDir,
  editablePackageSources,
  overrides ? poetry2nix.defaultPoetryOverrides,
  ... }@args:
let
  # pass all args which are not specific to mkPoetryEnv
  app = poetry2nix.mkPoetryApplication (builtins.removeAttrs args [ "editablePackageSources" ]);

  # pass args specific to mkPoetryEnv and all remaining arguments to mkDerivation
  editableEnv = stdenv.mkDerivation (
    {
      name = "editable-env";
      src = poetry2nix.mkPoetryEnv {
        inherit projectDir editablePackageSources overrides;
      };

      # copy all the output of mkPoetryEnv so that patching and wrapping of outputs works
      installPhase = ''
     mkdir -p $out
     cp -a * $out
   '';
    } // builtins.removeAttrs args [ "projectDir" "editablePackageSources" "overrides" ]
  );
in
app.overrideAttrs (super: {
  passthru = super.passthru // { inherit editableEnv; };
})

poetry2nix composes with nix-shell to get a development environment in shell.nix

This imports myApp from above so that it has a common gcroot – nothing more annoying than opening a file in this project and having direnv block while pytest runs. This is a fnord, but it gets me to the point where I can use direnv-mode to have lsp-org to work with pyright LSP Mode.

{ pkgs ? import <nixpkgs> {} }:

let
  myApp = import ./default.nix { inherit pkgs; };
  myAppEnv = myApp.editableEnv;
in pkgs.mkShell {
  packages = [
    myAppEnv
    pkgs.pandoc
    pkgs.poetry
    # inotify-tools
  ];
}

This gets me to having a thing i can nix-shell in to and have dependencies available, where I can run uvicorn arcology.server:app

arcology-with-assets can be bundled in to a simple Docker container with dockerTools.buildImage

{
  arroyo ? import <arroyo> {},
  emacsOverlay ? arroyo.lib.pkgVersions.emacsOverlay {},
  pkgs ? import <nixpkgs> { overlays = [
                              (import emacsOverlay)
                              (import <arroyo/overlay.nix>)
                            ];
                          },
  python ? pkgs.python3
}:

let
  app = import ./default.nix { inherit pkgs; inherit python; };
  env = app.dependencyEnv;
  myEmacs = (import /home/rrix/arroyo-nix/pkgs/emacs.nix { inherit pkgs; });
in pkgs.dockerTools.buildLayeredImage {
  name = "arcology";
  tag = "latest";

  contents = [ app myEmacs pkgs.pandoc pkgs.coreutils ];
  config = {
    Env = [
      "ARCOLOGY_DIRECTORY=/data"
      "ARCOLOGY_SRC=/data/arcology-fastapi"
      "ARROYO_SRC=/data/arroyo"
      "ARCOLOGY_DB=/databases/arcology.db"
      "ORG_ROAM_DB=/databases/org-roam.db"
      "ARCOLOGY_ENV=prod"
    ];

    Volumes = {
      "/data"={};
      "/databases"={};
    };
    WorkingDir = "${app}/lib/python${python.pythonVersion}/site-packages/";
    ExposedPorts = {
      "8000/tcp" = {};
    };
    # Cmd = ["${app}/bin/arcology-fastapi" ];
  };
}

These dockerfiles modify the starting command – i couldn't figure out how to do this with dockerTools' fromImage argument, the invocation was giving me some wonky error within poetry2nix?? daft.

FROM arcology:latest

CMD ["/bin/arcology-inotify" ]
FROM arcology:latest

CMD ["/bin/arcology-fastapi" ]

Build these like so:

Here's an all-in-one

set -e

nix-build docker.nix
docker load -i result

docker build -f Dockerfile-fastapi -t docker.fontkeming.fail/arcology-fastapi .
docker build -f Dockerfile-inotify -t docker.fontkeming.fail/arcology-inotify .

docker push docker.fontkeming.fail/arcology-fastapi
docker push docker.fontkeming.fail/arcology-inotify

ssh fontkeming "docker pull docker.fontkeming.fail/arcology-fastapi && docker pull docker.fontkeming.fail/arcology-inotify && sudo systemctl restart arcology-fastapi arcology-inotify"

(lol, sorry… i'll automate all this with The Wobserver's NixOS port some day) Ralphy voice i'm a systems engineer

INPROGRESS This needs to load in Arroyo Emacs built out of nix-community/emacs-overlay somehow…

Maybe easier to just rebuild the Wobserver run this on NixOS LMAO

DONE Environment configure BaseSettings

DONE Volumes mount volumes

DONE cmd – need a wrapper which can either inotify or fastapi – maybe split those in to different packages

NEXT refactor all this bullshit overlay stuff lmao

All of this can be encapsulated by a Nix Flake

Flakes are the new hotness everyone in Nix land says you should use except it's unstable and may change out from underneath you but it can make it easy to distribute your package from JitHub so you should do it. At the very least nix develop is nicer than nix-shell in theory.

In the spirit of Hey Smell This I'll provide a flake that in theory you can invoke to run the Arcology on any system with Nix installed. Probably? it probably won't work since i relative-import CCE modules but in theory this should next pull a flake-ified version of Arroyo Emacs in. some day. for now you get to keep the pieces and i get to run shell:nix run .#arcology-fastapi to start the project.

{
  description = "arcology org-mode publishing";
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = {self, nixpkgs, flake-utils}:
    flake-utils.lib.eachDefaultSystem (system:
      let pkgs = nixpkgs.legacyPackages.${system}; in
      rec {
        devShell = import ./shell.nix { inherit pkgs; };
        packages = flake-utils.lib.flattenTree {
          arcology = import ./default.nix { inherit pkgs; };
          docker = import ./docker.nix { inherit pkgs; };
        };
        defaultPackage = packages.arcology;
        apps.arcology-fastapi = flake-utils.lib.mkApp {
          drv = packages.arcology-with-assets;
          exePath = "/bin/arcology-fastapi";
        };
        apps.arcology-inotify = flake-utils.lib.mkApp {
          drv = packages.arcology-with-assets;
          exePath = "/bin/arcology-inotify";
        };
      }
    );
}

Everyone who wants to write maintainable flakes have to pull in a random 3rd party dependency first called flake-utils.

I should noweb this but for now i'm just copying it in here directly, I donno if I even want to keep using this since my systems compose on the filesystem using Syncthing.

Deploying Arcology to NixOS

Arcology is deployed to The Wobserver using Arroyo NixOS Generator, it uses the same Metadata extraction engine which powers the Arcology itself to generate a NixOS configuration file which will build my systems and deploy them through Morph. Deploying Arcology in this manner should not be so difficult, but I'm not so sure how to make this replicable by others – of course the source can be pulled from My Gitea Instance but I don't want to always be pushing code to remotes to iterate on the server…

{ config, lib, options, pkgs, ... }:

with lib;
let
  cfg = config.services.arcology;

  env = {
    ARCOLOGY_ENV = cfg.environment;
    ARCOLOGY_DIRECTORY = cfg.orgDir;

    ARCOLOGY_SRC = "${cfg.orgDir}/arcology-fastapi";
    ARROYO_SRC = "${cfg.orgDir}/arroyo";
    ARROYO_EMACS = "${cfg.packages.emacs}/bin/emacs";

    ARCOLOGY_DB = "${cfg.dataDir}/databases/arcology.db";
    ORG_ROAM_DB = "${cfg.dataDir}/databases/org-roam.db";
  };

  domainVHosts = {
    services.nginx.virtualHosts."${head cfg.domains}" = {
      serverAliases = tail cfg.domains;
      locations."/".proxyPass = "http://localhost:8000";
      extraConfig = ''
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header Host $host;
      '';
    };
  };
in {
  options = {
    services.arcology = {
      packages.arcology = mkOption {
        type = types.package;
        default = pkgs.arcology;
        description = mdDoc ''
        '';
      };

      packages.emacs = mkOption {
        type = types.package;
        default = pkgs.arroyo-emacs;
      };

      prometheus_collection.enable = mkOption {
        type = types.bool;
        default = config.services.prometheus.enable;
        description = mdDoc ''
          Set up prometheus to scrap Arcology
        '';
      };

      domains = mkOption {
        type = types.listOf types.string;
      };

      environment = mkOption {
        type = types.enum ["prod" "dev"];
        default = "prod";
      };

      dataDir = mkOption {
        type = types.path;
        default = "/var/lib/arcology";
        description = mdDoc ''
          Directory to store Arcology cache files, database, etc. Service User's home directory.
        '';
      };

      orgDir = mkOption {
        type = types.path;
        description = mdDoc ''
          Directory containing the org-mode documents.
          Arcology needs read-only access to this directory.
        '';
      };
    };
  };

  config = {
    # fix this probably...
    ids.uids.arcology = 900;
    ids.gids.arcology = 900;

    users.users.arcology = {
      group = "arcology";
      home = cfg.dataDir;
      createHome = true;
      shell = "${pkgs.bash}/bin/bash";
      isSystemUser = true;
      uid = config.ids.uids.arcology;
    };

    users.groups.arcology = {
      gid = config.ids.gids.arcology;
    };

    systemd.services.arcology-web = {
      description = "Arcology Web Site Engine's FastAPI web worker";
      after = ["network.target" "arcology-inotify.service"];
      wantedBy = ["multi-user.target"];
      environment = env;
      serviceConfig = {
        Type = "simple";
        User = "arcology";
        Group = "arcology";
        # include in pkg
        # WorkingDirectory = "${cfg.packages.arcology}/lib/python${pkgs.python3.pythonVersion}/site-packages/";
        WorkingDirectory = "${cfg.orgDir}/arcology-fastapi";
        ExecStart = "${cfg.packages.arcology}/bin/arcology-fastapi";

        Restart = "on-failure";
        UMask = "0077";
        # todo hardening
      };
    };

    systemd.services.arcology-inotify = {
      description = "Arcology Web Site Engine's indexing worker";
      after = ["network.target"];
      wantedBy = ["multi-user.target"];
      environment = env;
      serviceConfig = {
        Type = "simple";
        User = "arcology";
        Group = "arcology";
        # include in pkg
        # WorkingDirectory = "${cfg.packages.arcology}/lib/python${pkgs.python3.pythonVersion}/site-packages/";
        WorkingDirectory = "${cfg.orgDir}/arcology-fastapi";
        ExecStart = "${cfg.packages.arcology}/bin/arcology-inotify";

        Restart = "on-failure";
        UMask = "0077";
        # todo hardening
      };
    };
  } // domainVHosts;
}

Using this is pretty simple:

{ ... }:

{
  imports = [ <arroyo/nixos/arcology.nix> ];
  fileSystems."/media/org" = {
    device = "/home/rrix/org";
    options = ["bind"];
  };
  services.arcology = {
    orgDir = "/media/org";
    dataDir = "/srv/arcology";
    domains = [
      "engine.arcology.garden"
      "arcology.garden" 
      "thelionsrear.com"
      "cce.whatthefuck.computer" "cce.rix.si"
      "doc.rix.si"
    ];
  };
}

Pages which Link Here