The Arcology Site Engine

nixpkgs has pretty decent support for Elixir projects with some good helpers. In 2021 I tried an earlier incarnation of The Arcology Project in Elixir+Phoenix, got it working on my laptop but when time came to use Nix to put together a Docker container I couldn't get ecto_sqlite3 to build. Since then, I have shipped a fully function Arcology FastAPI prototype, but I am still quite interested in shipping this on the BEAM. Elixir is cool, and the Arcology Public Router could "just" be a Phoenix.Plug. Now that I have a proper database layer in the Arroyo Arcology Generator many of the design shortcomings of the original Arcology Phoenix prototype can be worked around. I'll be writing a Content-Addressed Store, too, I think, for caching the pages.

Install these packages from nixpkgs:

nix source: :noweb-ref packages
elixir_1_14
elixir_ls
inotify-tools
mix2nix

# probably will need this.... 🥴
nodePackages.node2nix

Set these basic environment variables in the shell:

nix source: :noweb-ref environment
LANG="C.UTF-8";
ERL_AFLAGS="-kernel shell_history enabled";

Put dependency files in $PWD/.nix-shell: (This is taken from elixir forum)

nix source: :noweb-ref environment
NIX_SHELL_DIR="$PWD/.nix-shell";
MIX_HOME="$NIX_SHELL_DIR/.mix";
MIX_ARCHIVES="$MIX_HOME/archives";
HEX_HOME="$NIX_SHELL_DIR/.hex";
PATH="$HEX_HOME/bin:$MIX_HOME/escripts:$MIX_HOME/bin:$PATH";
LIVEBOOK_HOME="$PWD";

Install a small wrapper script which will install hex package manager and the Phoenix mix tasks; run setup-mix-phx the first time this project is set up.

shell source: :noweb-ref packages
(writeScriptBin "setup-mix-phx" ''
 ${elixir}/bin/mix local.hex
 ${elixir}/bin/mix local.rebar
 ${elixir}/bin/mix archive.install hex phx_new
 '')

Assemble all that in to a pkgs.mkShell for nix-shell

nix source: :noweb yes :tangle shell.nix
{ ... }:

let
  pkgs = import <nixpkgs> {};
  # import <arroyo> {};
in with pkgs; mkShell {
  packages = [
    <<packages>>
  ];

  shellHook = ''
    export <<environment>>

    <<shellHook>>

    ${elixir}/bin/mix --version
    ${elixir}/bin/iex --version
  '';
}

And set up =direnv= (don't forget to direnv allow):

shell source: :tangle .envrc
use nix

This doesn't set up PostgreSQL or anything like that -- we're in roam:Sqlite country.

I had a fully functional Arcology Phoenix a number of years back but wasn't able to get ecto_sqlite3 to build in NixOS. Better start on with that now!

Start with a stub default.nix that sets up a callPackage:

nix source: :tangle default.nix
{ ... }:

let pkgs = import <nixpkgs> {};
in
  pkgs.callPackage ./arcology.nix {}

Then this thing can have its nix dependencies declared:

nix source: :tangle arcology.nix :noweb yes
{ lib, beamPackages, callPackage, ... }:

beamPackages.mixRelease rec {
  pname = "arcology";
  version = "0.0.1";

  src = ./.;

  dontStrip = true;

  <<override-mix-deps>>

  meta = with lib; {
    description = "Arcology Org-mode Web Engine";
    homepage = "https://engine.arcology.garden";
    license = licenses.unfree;
    maintainers = with maintainers; [ rrix ];
  };
}

Nix's mix deps are provided by mix2nix, run [[shell:mix2nix > mix.nix]] when the mix.exs deps are updated;

this basically works, except that exqlite which ecto_sqlite3 uses as a database driver. It will try to write "something" to XDG_CACHE_HOME ... I'm not sure why I need to define this twice -- defining it in only one of the root Arcology mixRelease or exqlite's mixRelease will cause the build to fail...

nix source: :noweb-ref override-mix-deps
XDG_CACHE_HOME = "/tmp/elixir-cache";
mixNixDeps = import ./mix.nix {
  inherit beamPackages lib;
  overrides = (final: prev: {
    exqlite = prev.exqlite.overrideAttrs (pprev: pprev // {
      XDG_CACHE_HOME = "/tmp/elixir-cache";
    });
  });
};

elixir source: :noweb-ref project
def project do
  [
    app: :arcology,
    version: "0.1.0",
    elixir: "~> 1.14",
    elixirc_paths: elixirc_paths(Mix.env()),
    start_permanent: Mix.env() == :prod,
    aliases: aliases(),
    deps: deps()
  ]
end

This instructs Mix to load roam:Arcology.Application as the entrypoint:

elixir source: :noweb-ref application
def application do
  [
    mod: {Arcology.Application, []},
    extra_applications: [:logger, :runtime_tools]
  ]
end

Specifies which paths to compile per environment.

elixir source: :noweb-ref paths
defp elixirc_paths(:test), do: ["lib", "test/support"]
defp elixirc_paths(_), do: ["lib"]

Here's what we depend on:

elixir source: :noweb-ref deps
defp deps do
  [
    {:phoenix, "~> 1.7.1"},
    {:phoenix_ecto, "~> 4.4"},
    {:ecto_sql, "~> 3.6"},
    {:ecto_sqlite3, ">= 0.0.0"},
    {:phoenix_html, "~> 3.3"},
    {:phoenix_live_reload, "~> 1.2", only: :dev},
    {:phoenix_live_view, "~> 0.18.16"},
    {:floki, ">= 0.30.0", only: :test},
    {:phoenix_live_dashboard, "~> 0.7.2"},
    {:esbuild, "~> 0.5", runtime: Mix.env() == :dev},
    {:tailwind, "~> 0.1.8", runtime: Mix.env() == :dev},
    {:swoosh, "~> 1.3"},
    {:finch, "~> 0.13"},
    {:telemetry_metrics, "~> 0.6"},
    {:telemetry_poller, "~> 1.0"},
    {:gettext, "~> 0.20"},
    {:jason, "~> 1.2"},
    {:plug_cowboy, "~> 2.5"},
    {:symbolic_expression, git: "https://github.com/rob-brown/SymbolicExpression", tag: "1.0.3"},
  ]
end

Aliases are shortcuts or tasks specific to the current project. For example, to install project dependencies and perform other setup tasks, run shell:mix setup. See the documentation for `Mix` for more info on aliases.

elixir source: :noweb-ref aliases
defp aliases do
  [
    setup: ["deps.get", "ecto.setup", "assets.setup", "assets.build"],
    "ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
    "ecto.reset": ["ecto.drop", "ecto.setup"],
    test: ["ecto.create --quiet", "ecto.migrate --quiet", "test"],
    "assets.setup": ["tailwind.install --if-missing", "esbuild.install --if-missing"],
    "assets.build": ["tailwind default", "esbuild default"],
    "assets.deploy": ["tailwind default --minify", "esbuild default --minify", "phx.digest"]
  ]
end

elixir source: :tangle mix.exs :noweb yes
defmodule Arcology.MixProject do
  use Mix.Project

  <<project>>

  <<application>>

  <<paths>>

  <<deps>>

  <<aliases>>
end

elixir source: :tangle config/config.exs
# This file is responsible for configuring your application
# and its dependencies with the aid of the Config module.
#
# This configuration file is loaded before any dependency and
# is restricted to this project.

# General application configuration
import Config

Tell Phoenix to load roam:Arcology.Repo

elixir source: :tangle config/config.exs
config :arcology,
  ecto_repos: [Arcology.Repo]

Configure =ArcologyWeb.Endpoint= , instruct it to use Arcology.PubSub as its message-passing layer.

elixir source: :tangle config/config.exs
config :arcology, ArcologyWeb.Endpoint,
  url: [host: "localhost"],
  render_errors: [
    formats: [html: ArcologyWeb.ErrorHTML, json: ArcologyWeb.ErrorJSON],
    layout: false
  ],
  pubsub_server: Arcology.PubSub,
  live_view: [signing_salt: "OgcteMC0"]

Configures the mailer

By default it uses the "Local" adapter which stores the emails locally. You can see the emails in your browser, at "/dev/mailbox".

For production it's recommended to configure a different adapter at the `config/runtime.exs`.

elixir source: :tangle config/config.exs
config :arcology, Arcology.Mailer, adapter: Swoosh.Adapters.Local

Configure esbuild for Javascript packaging. Sure beats a webpack setup!! I hope!!!!

elixir source: :tangle config/config.exs
config :esbuild,
  version: "0.14.41",
  default: [
    args:
      ~w(js/app.js --bundle --target=es2017 --outdir=../priv/static/assets --external:/fonts/* --external:/images/*),
    cd: Path.expand("../assets", __DIR__),
    env: %{"NODE_PATH" => Path.expand("../deps", __DIR__)}
  ]

Configure Tailwind for components CSS; I might just ship the same 40 CSS rules I've been using for a while, though, tbh. This stuff is auto-generated:

elixir source: :tangle config/config.exs
# Configure tailwind (the version is required)
config :tailwind,
  version: "3.2.4",
  default: [
    args: ~w(
      --config=tailwind.config.js
      --input=css/app.css
      --output=../priv/static/assets/app.css
    ),
    cd: Path.expand("../assets", __DIR__)
  ]

Configures Elixir's Logger:

elixir source: :tangle config/config.exs
config :logger, :console,
  format: "$time $metadata[$level] $message\n",
  metadata: [:request_id]

Use Jason for JSON parsing in Phoenix; in the past I used Poison which require a NIF, but this seems like a good middle-ground between fast and cheap.

elixir source: :tangle config/config.exs
config :phoenix, :json_library, Jason

Import environment specific config. This must remain at the bottom of this file so it overrides the configuration defined above.

elixir source: :tangle config/config.exs
import_config "#{config_env()}.exs"

This will override the code in the parent config.exs.

elixir source: :tangle config/dev.exs
import Config

In production this will read an environment variable, but in dev we can just use the local directory to store the Arroyo Arcology Generator 's DB.

elixir source: :tangle config/dev.exs
config :arcology, Arcology.Repo,
  database: Path.expand("~/.emacs.d/arroyo.db", Path.dirname(__ENV__.file)),
  pool_size: 5,
  stacktrace: true,
  show_sensitive_data_on_connection_error: true

Dev build will bind to http://localhost:4000 and have debug helpers installed code reloading enabled for Elixir, CSS, and Javascript.

elixir source: :tangle config/dev.exs
config :arcology, ArcologyWeb.Endpoint,
  # Binding to loopback ipv4 address prevents access from other machines.
  # Change to `ip: {0, 0, 0, 0}` to allow access from other machines.
  http: [ip: {127, 0, 0, 1}, port: 4000],
  check_origin: false,
  code_reloader: true,
  debug_errors: true,
  secret_key_base: "PkJnuIpBpXQpUqGrc7ynUHYyXvAB40b03rjay/9WuKmzEN9H9GaELBdGOok2GQih",
  watchers: [
    esbuild: {Esbuild, :install_and_run, [:default, ~w(--sourcemap=inline --watch)]},
    tailwind: {Tailwind, :install_and_run, [:default, ~w(--watch)]}
  ]

Phoenix's LiveReload is pretty slick -- it will automatically reload your browser page if CSS, JS, or Elixir Web modules, etc are changed:

elixir source: :tangle config/dev.exs
config :arcology, ArcologyWeb.Endpoint,
  live_reload: [
    patterns: [
      ~r"priv/static/.*(js|css|png|jpeg|jpg|gif|svg)$",
      ~r"priv/gettext/.*(po)$",
      ~r"lib/arcology_web/(controllers|live|components)/.*(ex|heex)$"
    ]
  ]

Enable dev routes for dashboard and mailbox

elixir source: :tangle config/dev.exs
config :arcology, dev_routes: true

Do not include metadata nor timestamps in development logs

elixir source: :tangle config/dev.exs
config :logger, :console, format: "[$level] $message\n"

Set a higher stacktrace during development. The auto-generated comment says "Avoid configuring such in production as building large stacktraces may be expensive."

elixir source: :tangle config/dev.exs
config :phoenix, :stacktrace_depth, 20

Initialize plugs at runtime for faster development compilation:

elixir source: :tangle config/dev.exs
config :phoenix, :plug_init_mode, :runtime

Disable swoosh api client as it is only required for production adapters, this is for the Mailer, and I probably don't care to set this up in prod, but yanno...

elixir source: :tangle config/dev.exs
config :swoosh, :api_client, false

I promise myself I'll figure out how to write tests for Arcology this time around.............

elixir source: :tangle config/test.exs
import Config

# Configure your database
#
# The MIX_TEST_PARTITION environment variable can be used
# to provide built-in test partitioning in CI environment.
# Run `mix help test` for more information.
config :arcology, Arcology.Repo,
  database: Path.expand("../arcology_test.db", Path.dirname(__ENV__.file)),
  pool_size: 5,
  pool: Ecto.Adapters.SQL.Sandbox

# We don't run a server during test. If one is required,
# you can enable the server option below.
config :arcology, ArcologyWeb.Endpoint,
  http: [ip: {127, 0, 0, 1}, port: 4002],
  secret_key_base: "4EDyvZHi6tCmgax6tx/v9e6L9kQmXt8qnhsqYE78k7P82uy71Ge7Zl6jPMolA+RR",
  server: false

# In test we don't send emails.
config :arcology, Arcology.Mailer, adapter: Swoosh.Adapters.Test

# Disable swoosh api client as it is only required for production adapters.
config :swoosh, :api_client, false

# Print only warnings and errors during test
config :logger, level: :warning

# Initialize plugs at runtime for faster test compilation
config :phoenix, :plug_init_mode, :runtime

This is mostly overridden by the runtime configuration below.

elixir source: :tangle config/prod.exs
import Config

# For production, don't forget to configure the url host
# to something meaningful, Phoenix uses this information
# when generating URLs.

# Note we also include the path to a cache manifest
# containing the digested version of static files. This
# manifest is generated by the `mix phx.digest` task,
# which you should run after static files are built and
# before starting your production server.
config :arcology, ArcologyWeb.Endpoint, cache_static_manifest: "priv/static/cache_manifest.json"

# Configures Swoosh API Client
config :swoosh, api_client: Swoosh.ApiClient.Finch, finch_name: Arcology.Finch

# Do not print debug messages in production
config :logger, level: :info

# Runtime production configuration, including reading
# of environment variables, is done on config/runtime.exs.

These helpers are used to set up the process with environment variables at runtime. These will be important when it comes to deploying Arcology on the Wobserver .

Only start the web server if PHX_SERVER is set; this is done automatically by the bin/server wrapper script installed by mix phx.gen.release.

elixir source: :tangle config/runtime.exs
import Config
if System.get_env("PHX_SERVER") do
  config :arcology, ArcologyWeb.Endpoint, server: true
end

Only prod is runtime configured:

elixir source: :tangle config/runtime.exs
if config_env() == :prod do

The database is pointed to with DATABASE_PATH:

elixir source: :tangle config/runtime.exs
  database_path =
    System.get_env("DATABASE_PATH") ||
      raise """
      environment variable DATABASE_PATH is missing.
      For example: /etc/arcology/arcology.db
      """
  
  config :arcology, Arcology.Repo,
    database: database_path,
    pool_size: String.to_integer(System.get_env("POOL_SIZE") || "5")

The secret key base is used to sign/encrypt cookies and other secrets. A default value is used in config/dev.exs and config/test.exs but you want to use a different value for prod and you most likely don't want to check this value into version control, so we use an environment variable instead.

elixir source: :tangle config/runtime.exs
  secret_key_base =
    System.get_env("SECRET_KEY_BASE") ||
      raise """
      environment variable SECRET_KEY_BASE is missing.
      You can generate one by calling: mix phx.gen.secret
      """

PHX_HOST and PORT are used to configure the route/URL generation. The app is configured to listen only to the loopback IPv4 interface since Nginx will be handling the actual edge and SSL and whatnot. Otherwise it would be configured according to Plug.Cowboy documentation.

elixir source: :tangle config/runtime.exs
  host = System.get_env("PHX_HOST") || "example.com"
  port = String.to_integer(System.get_env("PORT") || "4000")

  config :arcology, ArcologyWeb.Endpoint,
    url: [host: host, port: 443, scheme: "https"],
    http: [
      ip: {127, 0, 0, 1},
      port: port
    ],
    secret_key_base: secret_key_base

These aren't used right now:

elixir source: 
  # ## SSL Support
  #
  # To get SSL working, you will need to add the `https` key
  # to your endpoint configuration:
  #
  #     config :arcology, ArcologyWeb.Endpoint,
  #       https: [
  #         ...,
  #         port: 443,
  #         cipher_suite: :strong,
  #         keyfile: System.get_env("SOME_APP_SSL_KEY_PATH"),
  #         certfile: System.get_env("SOME_APP_SSL_CERT_PATH")
  #       ]
  #
  # The `cipher_suite` is set to `:strong` to support only the
  # latest and more secure SSL ciphers. This means old browsers
  # and clients may not be supported. You can set it to
  # `:compatible` for wider support.
  #
  # `:keyfile` and `:certfile` expect an absolute path to the key
  # and cert in disk or a relative path inside priv, for example
  # "priv/ssl/server.key". For all supported SSL configuration
  # options, see https://hexdocs.pm/plug/Plug.SSL.html#configure/1
  #
  # We also recommend setting `force_ssl` in your endpoint, ensuring
  # no data is ever sent via http, always redirecting to https:
  #
  #     config :arcology, ArcologyWeb.Endpoint,
  #       force_ssl: [hsts: true]
  #
  # Check `Plug.SSL` for all available options in `force_ssl`.

  # ## Configuring the mailer
  #
  # In production you need to configure the mailer to use a different adapter.
  # Also, you may need to configure the Swoosh API client of your choice if you
  # are not using SMTP. Here is an example of the configuration:
  #
  #     config :arcology, Arcology.Mailer,
  #       adapter: Swoosh.Adapters.Mailgun,
  #       api_key: System.get_env("MAILGUN_API_KEY"),
  #       domain: System.get_env("MAILGUN_DOMAIN")
  #
  # For this example you need include a HTTP client required by Swoosh API client.
  # Swoosh supports Hackney and Finch out of the box:
  #
  #     config :swoosh, :api_client, Swoosh.ApiClient.Hackney
  #
  # See https://hexdocs.pm/swoosh/Swoosh.html#module-installation for details.

elixir source: :tangle config/runtime.exs
end