The Arcology Site Engine

Navigating the Arcology Site Graph with SigmaJS

LifeTechEmacsTopicsArcology

What is SigmaJS

SigmaJS is a JavaScript library for visualizing graphs. My org-roam knowledge base is a graph and arroyo-db contains the public pages and links. As nodes and edges, this makes a fine graph and a lot more "logical" than an XML sitemap or linear walk through the sites.

Getting SigmaJS

Download and vendor the minified versions:

export GRAPHOLOGY_VER=0.24.1
export SIGMAJS_VER=v2.3.1

curl --location -v -o arcology/static/sigmajs.min.js https://github.com/jacomyal/sigma.js/releases/download/$SIGMAJS_VER/sigma.min.js

curl --location -v -o arcology/static/graphology.min.js https://github.com/graphology/graphology/releases/download/$GRAPHOLOGY_VER/graphology.min.js

curl --location -v -o arcology/static/graphology-library.min.js https://github.com/graphology/graphology/releases/download/$GRAPHOLOGY_VER/graphology-library.min.js

At this point the Arcology Page Template should be able to import these thanks to the Arcology FastAPI router exposing that static directory. I could set up a typescript dev environment and webpack and all this sort of shit, but I don't really need it…. It's fine! This Is Fine!

Generating the Nodes and Edges bundle from Arroyo Arcology

From arroyo-db load all Arcology Page as Nodes shaped like:

{ "id": "n0", "label": "A node", "x": 0, "y": 0, "size": 3 }

From arroyo-db load all Arcology Links as Edges shaped like:

{ "id": "e0", "source": "n0", "target": "n1" }

compose together and return in a single dict for JSON rendering shaped like:

{ "nodes": [ <<all nodes>> ] "edges": [ <<all edges>> ] }

Making SigmaJS Nodes

Nodes are simple:

import hashlib

from sqlmodel import Session, select

import arcology.arroyo as arroyo
from arcology.arroyo import Page

def make_loc_hash(page: Page, salt, max_q=700):
  key = page.get_file() + str(salt)
  hash = hashlib.sha224(key.encode("utf-8")).digest()
  return int.from_bytes(hash, byteorder="big") % max_q

def nodes(engine):
  # collect all pages
  with Session(engine) as s:
    pages = s.exec(select(Page)).all()

    nodes = [
      dict(
        key=page.get_key(),
        attributes=dict(
          label=page.get_title(),
          x=make_loc_hash(page, 1),
          y=make_loc_hash(page, 2),
          size=min((len(page.backlinks) + len(page.outlinks)) / 2, 10),
          color=page.get_site().node_color,
          href=page.get_arcology_key().to_url()
        )
      )
      for page in pages
    ]

    return nodes

Making SigmaJS Edges

Edges are too, we just have to make sure the IDs line up:

from sqlmodel import Session, select

import arcology.arroyo as arroyo
from arcology.arroyo import Link, engine

from arcology.parse import parse_sexp

# HACK: https://stackoverflow.com/questions/1151658/python-hashable-dicts
class hashabledict(dict):
    def __hash__(self):
        return hash(tuple(sorted(self.items())))

def edges(engine):
  # collect all pages
  with Session(engine) as s:
    q_links = s.exec(select(Link)).all()

    links = set()
    for link in q_links:
      source = link.source_page.get_key()
      dest = link.dest_page.get_key()
      links.add(
        hashabledict(
          key=f"{source}-{dest}",
          source=source,
          target=dest,
        )
      )

    return list(links)

Compose these together, let the Router handle turning it in to JSON. I clobber this in to an LRU cache like I do in a bunch of other places with a key brute-forced to map to the SHA hashes of every file in the public wiki… Woof! Glad Personal Software Can Be Shitty!

import functools

@functools.lru_cache(maxsize=20)
def sigma_lru(cache_key):
  return dict(
    # this is a different import of the same engine...
    nodes=nodes(engine),
    edges=edges(engine)
  )

def sigma(engine):
  with Session(engine) as session:
    pages = session.exec(select(Page).order_by(Page.key))
    hashes =[page.hash for page in pages]

    cache_key = ''.join(
      hashes
    )
    return sigma_lru(cache_key)

Sitemap HTML Template

This derives from the Base HTML Template for the sites. It's decomposed a little bit with noweb syntax so that the JavaScript elements can be worked on as JavaScript instead of as Jinja2 templates.

{% extends "base.html.j2" %}

{% block h1 %}
  <h1><a href='/'>{{ site.title }}</a></h1>
  <h2>The Site Map</h2>
{% endblock %}

{% block head %}
  <title>Arcology System Site Map</title>
  <script src="/static/graphology.min.js"></script>
  <script src="/static/graphology-library.min.js"></script>
  <script src="/static/sigmajs.min.js"></script>
{% endblock %}

{% block body %}
  <main>
    <section class="body">
      <p>
        This is a network graph of all the pages published in my
        Arcology network. The color of the node corresponds to its
        site, and you can click on any of them to jump to that page or
        zoom in to see more of their labels. I will be adding more
        features to this sitemap like search and highlighting
        "relative" neighbors later on. Feel free to view
        <a href="https://engine.arcology.garden/sitemaps">the implementation</a>
        within the Arcology pages, as usual.
      </p>

      <div id="sitemap-container">
      </div>

    </section>
    <script type="application/javascript">
      <<sigmaInit>>
    </script>
  </main>
{% endblock %}

JavaScript to set up and render the SigmaJS graph

The SigmaJS code is loaded in the <head>, and the graph is injected in to the #sitemap-container <div>.

const container = document.getElementById("sitemap-container");
fetch('/sitemap/json')
    .then(response => response.json())
    .then((data) => {
        const graph = new graphology.Graph.from(data);

        <<forceAtlas2-setup>>

        const renderer = new Sigma(graph, container);

        <<hoverReducer>>

        <<forceAtlas2-invoke-worker>>

        <<clickNode>>

        return {renderer, graph}
    });

A ForceAtlas2 layout algorithm is applied to the page for 5 seconds to allow the nodes to coalesce in to a stable layout.

var FA2Layout = graphologyLibrary.FA2Layout;
var forceAtlas2 = graphologyLibrary.layoutForceAtlas2;
const sensibleSettings = forceAtlas2.inferSettings(graph);
const layout = new FA2Layout(graph, {
    settings: sensibleSettings
});

The layout work is done in a background task:

layout.start();
setTimeout(() => {
    layout.stop()
}, 5000);

An event handler is defined for the clickNode event to open the URL embedded in the node's attributes added in Making SigmaJS Nodes.

renderer.on("clickNode", ({ node }) => {
    var realnode = graph._nodes.get(node);
    window.location = realnode.attributes.href;
});

Focusing on Nodes in the Graph

The <<hoverReducer>> code to "highlight" a node by de-emphasizing the nodes it does not neighbor is taken from the SigmaJS examples and converted from TypeScript to JavaScript by hand. No I will not set up a NodeJS build chain.

state = {};
function setHoveredNode(node) {
    if (node) {
        state.hoveredNode = node;
        state.hoveredNeighbors = new Set(graph.neighbors(node));
    } else {
            state.hoveredNode = undefined;
            state.hoveredNeighbors = undefined;
        }

        // Refresh rendering:
        renderer.refresh();
    }

    renderer.on("enterNode", ({ node }) => {
        setHoveredNode(node);
    });
    renderer.on("leaveNode", () => {
        setHoveredNode(undefined);
    });

    // Render nodes accordingly to the internal state:
    // 1. If a node is selected, it is highlighted
    // 2. If there is query, all non-matching nodes are greyed
    // 3. If there is a hovered node, all non-neighbor nodes are greyed
    renderer.setSetting("nodeReducer", (node, data) => {
        const res = { ...data };

        if (state.hoveredNeighbors && !state.hoveredNeighbors.has(node) && state.hoveredNode !== node) {
            res.label = "";
            res.color = "#f6f6f6";
        }

        if (state.selectedNode === node) {
            res.highlighted = true;
        } else if (state.suggestions && !state.suggestions.has(node)) {
            res.label = "";
            res.color = "#f6f6f6";
        }

        return res;
    });

    // Render edges accordingly to the internal state:
    // 1. If a node is hovered, the edge is hidden if it is not connected to the
    //    node
    // 2. If there is a query, the edge is only visible if it connects two
    //    suggestions
    renderer.setSetting("edgeReducer", (edge, data) => {
        const res = { ...data };

        if (state.hoveredNode && !graph.hasExtremity(edge, state.hoveredNode)) {
            res.hidden = true;
        }

        if (state.suggestions && (!state.suggestions.has(graph.source(edge)) || !state.suggestions.has(graph.target(edge)))) {
            res.hidden = true;
        }

        return res;
    });

NEXT Arcology References as edges…