Arcology Engine

Maps: OsmAnd ↔ Arcology2go

Contents

Last night my buddy asked our group chat if anyone had good recommendations for particular campsites in some particular plots of BLM land that he was gonna bike out to camp in. Oh, I have like a half dozen out there from when I would go out noodling around and play-acting a Forest Sage. The best way for me to get that to him, though, was a half dozen geo: URI or coordinate pairs.

The only good map exploration tool I've found is OsmAnd.net ... RideWithGPS is good if you're on a bike, but for finding weird forest roads, comparing satellite views, looking for fun places to do Dispersed Camping, etc, OsmAnd is the way to go, it beats Google Maps and maybe Google Earth, but there's not really good free exploratory mapping tool except for this one Android app. And I can run it in Waydroid on my desktop. I have a custom OsmAnd Profile that lets me easily swap between satellite and map view, measure distances and incline, etc... And so I have a shit tonne of POI in this mobile app. I could just pass around GPX files, but I can't easily comment on each of the POI if they are in a single opaque export. I can't attach images to them.

This document explores how to bridge OsmAnd with the limited mapping support I've added to arcology2go and answer a core problem: how can I share the interesting places I find with other people?

Two make a robust bi-direction sync and publish solution there are three surfaces — OsmAnd into arcology, arcology into OsmAnd, and the published web view — each holding a set of open design questions.

The design is settled; import and export shipped 2026-10-09/10, and the web surface (built early, on a user call) landed 2026-10-10. Detail lives in the cluster files now:

Cluster index

  • maps/models.org — GPX 1.1 data models, parser, deterministic node IDs, property mapping, group placement (personal bucket), fixture findings. Phase 1.

  • maps/import-export.org — GpxImportService, GpxExportService, merge/body/removal policies, transport design (share intent → AIDL), intent wiring, review screen, attachments round-trip, device experiments. Phases 2–4.

  • maps/web.org — Arcology → web: map page publishing via MapTiler SDK JS (MapStyle + terrain knobs), marker↔heading anchoring, no-JS POI fallback list, ARCOLOGY_MAP knob property, key hygiene. DONE 2026-10-10.

Build order

  1. Models + parser (models.org): GpxModels.kt + GpxParser.kt, GeoCoordinate altitude, WaypointId.kt SHA-256 deterministic IDs, real-fixture tests. DONE 2026-10-09 — see the decisions log below.

  2. Import (import-export.org): GpxImportService, manifest + parseSharedIntent wiring, ImportReviewScreen v1. DONE 2026-10-10 — service + diff/apply/merge + review screen + share-intent branch + CLI gpx-import, all gates green.

  3. Export (import-export.org): GpxExportService, deterministic byte-stable output, share route beside the KML button; property tests for round-trip identity. DONE 2026-10-10 — service + writer + app share route + CLI gpx-export; KML export removed entirely (user call).

  4. Web pages (web.org): map.peb + arcology-map.js + MapPages.kt marker builder, MapTiler SDK JS from their CDN, ARCOLOGY_PAGE_TEMPLATE: map gates, ARCOLOGY_MAP knobs. DONE 2026-10-10 — server wiring in web/server.org (=PageModel.mapDataJson=/=mapEntries=, resolveTemplate map branch, keywords-sidebar key scrub); app dropdown gained map (editor/screen.org).

  5. Device experiments (manual, on-phone): round-trip diff with our extensions attached; importGpx-to-favorites-dir silent-sync check (CVE-2026-65996 area); web map page smoke on the serve box with the real MapTiler key (domain-restrict the key first).

Workflow per phase: edit org → arcology2 tangle → make build → make test. No new Gradle deps planned → no make update-deps. Coverage floor 50%.

Decisions log

  • 2026-10-10 POI-list titled links center the map (user call): clicking a .poi-list titled link was a native anchor jump; now the anchor carries data-poi-id (map.peb) and the delegated click handler treats it as the third click-anchored surface — preventDefault + flyTo the POI, scrollIntoView + history.replaceState only when the heading is on-page (cross-site POIs center the map; the marker popup remains the way out, the row's geo: link remains the OsmAnd handoff). Without JS the links still navigate (progressive enhancement preserved).

  • 2026-10-10 custom style UUID pass-through: the user's MapTiler style is a designer-made custom style whose id is a UUID — =resolveStyle='s unknown-name fallback to STREETS silently swallowed it. arcology-map.js now passes bare UUIDs (regex, case-insensitive), maptiler://<uuid> URIs, and full style.json URLs through as plain strings — the SDK's styleToStyle accepts all three forms, normalizes them, and injects config.apiKey (README + sdk-js/examples/custom-map, pinned 2026-10-10); only named reference styles take the MapStyle proxy path.

  • 2026-10-10 ARCOLOGY_MAPTILER_KEY env fallback: the MapTiler key resolves knob-first (ARCOLOGY_MAP `key=`), then the ARCOLOGY_MAPTILER_KEY process environment variable (blank treated as unset), else "". The env var is the recommended carrier — keeps the key out of published org files entirely; the knob remains for per-page overrides. buildMapPageContext(…, envKey System.getenv(…)?)= — injectable parameter so tests stay hermetic. Documented in web/deployment.org's environmentFile (sops-rendered alongside ARCOLOGY_SYNCTHING_API_KEY).

  • 2026-10-10 map-page properties = drawer convention (smoke-test finding): file-level =#+ARCOLOGY_:= preamble keyword lines drive route insertion (ArcologyPublishingPlugin reads document.keywordLines) but do NOT populate file_properties — a #+ARCOLOGY_PAGE_TEMPLATE: keyword line routes correctly and then silently renders page.peb. The prod org files use the file-level :PROPERTIES: drawer (archive.org: :ARCOLOGY_PAGE_TEMPLATE: topic) and that is what lands in file_properties. So a map page's =ARCOLOGY_PAGE_TEMPLATE/=ARCOLOGY_MAP= must be carried in the file-level drawer (or a heading drawer on a section-level page, same as the app capture UI writes). Constraint documented in web.org's knob section; no code change — teaching the DB about keyword lines is the pre-existing publishing model's shape, not this feature's bug to fix. - 2026-10-10 serve smoke test (localhost:8099, 2-file fixture)*: map page rendered map.peb with SDK CDN includes, one arcology-map-data blob carrying =key=SMOKE-KEY/style=outdoor/terrain=true/markers x3, no-JS =<ul class="poi-list">= with anchored + geo: links, markers exactly the rendered subtree (page-node + 2 children), :noexport: POI fully elided (builder + renderer), other-file POI excluded, sibling page.peb route clean (no map artifacts), /static/arcology-map.js 200, ARCOLOGY_MAP absent from the rendered page (keywords scrub holds). Gated behind ARCOLOGY_PAGE_TEMPLATE: map as designed.

  • 2026-10-10 web surface DONE (built early on user call): MapTiler, not OpenFreeMap/AWS Terrain (the doc's opening line was always the plan; the "woven-in recommendation" paragraph was the wrong reading, now superseded — survey matrix retained as the fallback map if the account dies). Client MapTiler SDK JS v4.2.0 via cdn.maptiler.com (session-based billing; extends MapLibre so Marker/Popup/flyTo/terrain stand); NOT vendored — the map page talks to api.maptiler.com for every tile regardless, so vendoring buys nothing. Opt-in ARCOLOGY_PAGE_TEMPLATE: map; knobs in ARCOLOGY_MAP reuse GpxOrgMapping.decodeExtensions (OSMAND_EXTENSIONS escaping, one home). Popups title + link + geo: href (user call; body popups with inline crushed attachments are the follow-up). Marker set rendered subtree's =GEO_COORDS=+=:ID:= nodes via one getNodesWithLocation + node_ancestors membership (parser parentNodeId NOT hydrated by RoamRepositoryImpl, so the ancestors table is the only DB hierarchy). Markers computed per request, outside the CachedPage envelope; JSON blob neutralizes </ → <\/ so a POI title can't break out of the script tag. Key hygiene: SidebarService keywords panel excludes ARCOLOGY_MAP (would render the key knob into the page); restrict the MapTiler key to the site domains in their console.

  • 2026-10-10 disk-truth guard (fixed after device data loss): plan() used to trust repository rows alone — a user-deleted target file whose DB rows survived their reindex came back UNCHANGED, and apply() then wrote a header-only file over it (gpx_imports/favorites.org truncated to title+heading on-device). Now fileExists gates everything: a stored node whose :ID: is absent from the on-disk target file always imports NEW (covers both the deleted-file and bare-header cases), stale DB rows for a missing file never surface as orphan deletion suggestions, and a warning reports "repository still has N nodes for <file> but the file is missing". Recovery path when it bites: re-import from the sources snapshot (gpx_imports/sources/<name>.gpx), which apply() wrote alongside every successful import.

  • 2026-10-10 Koin function-type erasure crash (fixed on device): registering (Uri) -> GpxExportService as a single<T> collided with the module's (AndroidFileSystem) -> OrgDocumentEditor single — Koin keys function-typed definitions by the erased Function1 class, so the map ViewModel's get() resolved the editor factory and its fs as AndroidFileSystem cast crashed with ClassCastException: Uri$StringUri cannot be cast to AndroidFileSystem on the first export tap. Rule: never register function-typed singles alongside another same-shape single in this module; factories are built inline in their consumer's viewModel block (the FlashcardService/=createOrgEditor= convention).

  • 2026-10-10 phase-3 export DONE + KML removed: =GpxExportService=/(GpxXmlWriter hand-rolled emitter over the phase-1 model — xmlutil's writer overloads are JVM-only extensions invisible from commonMain, resolving the phase-1 "custom serializer needed" note), MapViewModel.exportToGpx share route (same filtered node set as the old KML button), gpx-export CLI. KmlModels, KmlExportService, GeoCoordinate.toKmlCoordinates deleted. User calls superseding settled-design lines: `<desc>` = raw org markup (NOT an OrgHtmlRenderer render — org truth verbatim, `osmand:arcology_org` stays the restore channel) and no `<link>` emission in v1 (published-attachments backfill is JVM-only, web surface deferred; links round-trip through the body). Groups: ARCOLOGY_GROUP ?? olp-derived ?? default bucket, appearance entries only from ARCOLOGY_POINTS_GROUPS file properties. Gates: fixture import→export→re-import fixed point (110/110 UNCHANGED), byte-identical double export, 200-sample writer↔parser identity property.

  • 2026-10-10 slugify emits underscores, not hyphens: org tag charset is [_[:alnum:]@#%:] — hyphens are invalid in org tags (Emacs rejects them), so GpxOrgMapping.slugify turns every non-tag char INTO an underscore (hyphens included), collapses runs, trims edges. Phase-1's hyphen-preserving slug was wrong; caught after the first import output. The import service's tag-span regexes now match the same charset.

  • 2026-10-10 phase-2 import DONE: GpxImportService (diff/apply, heading-scoped upsert, arcology_org body restore, orphan confirm), ImportReviewScreen v1 + ImportReviewViewModel, share-intent sniff branch, gpx-import CLI. All gates green; fixed gates kept green by 19 kotest service cases + 4 VM cases.

  • 2026-10-10 placement supersedes file-per-group: ONE org file per gpx; OsmAnd groups map to the org heading outline path (Siuslaw/campsites → nested headings); the "" default group places points at file top level (re-export round-trips them back to "") and the literal personal group becomes a = personal= heading. Group names and olp both split on / symmetrically. (User call 2026-10-10.) - 2026-10-10 FileSystemInterface.ensureFile(path, initialContent) added to the shared contract: SAF writeFile can't create documents, so AndroidFileSystem overrides with docId walking (findChildByName/createDirectory/createFileWithContent); JVM override creates parents; TestFileSystem default suffices. Keeps the service commonMain and all three entry points thin. - 2026-10-10 xmlutil fact: XmlReader delivers entity references as separate ENTITY_REF events whose text is the decoded replacement — GpxParser.readValue accumulates them or every &amp;/&lt;/&gt; in a desc goes missing. Caught by the import service tests, pinned in models.org. - 2026-10-10 DB outline-path shape: outline paths persist as =/-joined strings and re-split on /, so point TITLES containing slashes inflate the segment count; import compares against the fully-split expected shape (outline + title) instead of slicing the last segment — otherwise slash-named points churn MOVED forever. Fixture has two such points. - 2026-10-10 the indexer never deletes node rows that vanish from a file (replace-not-delete in storeParseResultBatched), so confirmed orphan removals explicitly deleteNode + deleteNodeFromFts for every id inside the removed subtree; reindex order puts source files before the target so a moved node's FTS row is dropped by the old file and re-inserted by the new one. - 2026-10-10 stale-index guard: an import id already present in the on-disk file refreshes instead of re-inserting; forced reindex even when the refreshed content is identical, so a crash between write and reindex actually rebuilds the DB. - 2026-10-10 manifest correction: AndroidManifest.xml is hand-maintained, NOT tangled from app/build.org (that file is build-system only). The import-export.org "manifest (app/build.org)" pointer was wrong; gpx SEND/VIEW filters were direct edits. - 2026-10-09 gis package: the geospatial shared layer lives in computer.whatthefuck.arcology.gis (GpxModels/GpxParser/GpxOrgMapping/WaypointId and the phase-2/3 services to come) — export was wrong since import is its first consumer. KmlModels/KmlExportService stay in export: KML is export-only today. - 2026-10-08 deterministic IDs: SHA-256 uuid5-style over group+name on existing HashUtils expect/actual, pinned in commonMain (JVM nameUUIDFromBytes rejected). (2026-10-09: formula frozen, see below.) - 2026-10-08 empty default group imports into the personal bucket (file-per-group convention; export still never targets default/Personal/Home/Work/Parking groups). - 2026-10-09 phase-1 data layer DONE: GeoCoordinate folded out of roam/models.org into maps/models.org as its own file (same-package domain/GeoCoordinate.kt split — zero churn for consumers; Models.kt shrunk, roam/models.org keeps a stub link; GeoCoordinateTest bootstrapped since ModelsTest never covered it). Two green gates: pure move, then altitude. - 2026-10-09 GeoCoordinate gains altitude: Double? null=; parse takes 2- or 3-part values, a garbage/NaN/Infinity third part invalidates the whole value; toPropertyValue() emits 3-part only when altitude is present. - 2026-10-09 WaypointId formula frozen (pinned by vectors): uuid5-style over SHA-256 of the versioned namespace osmand-wpt-v1:<group>/<name>, version nibble 5 + RFC-4122 variant, lowercase hyphenated. Changing the salt orphans minted IDs. - 2026-10-09 OSMAND_EXTENSIONS property encoding: backslash-escaped =key=value= pairs joined by ; , \\, \c, \e, \n, \r, \t; unknown escapes decode verbatim. points_groups ↔ one ARCOLOGY_POINTS_GROUPS file property: =|=-joined groups, =;=-joined name/color/icon/background/pinned/hidden fields, same escaping. Both reversible, single-line. - 2026-10-09 fixture: real-resource src/commonTest/resources/gpx/favorites.gpx (cp of scratch/favorites.gpx, NOT a tangle target), documented in roam/fixtures.org. Fixture has 12* points_groups entries: 10 named + "" default + a group literally named personal — OsmAnd spells the personal bucket two ways; phase-2 file placement merges both into the personal file.

  • 2026-10-09 GpxParser is a streaming XmlReader parser — arbitrary-key, TEXT-only, nested-dropping extensions are inexpressible in xmlutil @Serializable decode; first XML reader in the codebase. xmlutil facts learned: no @XmlAttribute annotation (simple properties are attributes by default), QName is javax's (localPart), newReader(CharSequence) overload.

  • 2026-10-08 scope: import + export only; maps/web.org deferred.

  • 2026-10-08 deterministic IDs: SHA-256 uuid5-style over group+name on existing HashUtils expect/actual, pinned in commonMain (JVM nameUUIDFromBytes rejected).

  • 2026-10-08 empty default group imports into the personal bucket (file-per-group convention; export still never targets default/Personal/Home/Work/Parking groups).

  • 2026-10-07 no plugin, no server in the sync path: AIDL is publish-only (no favorites read API), which also kills app-to-app CRDT sync; fork is parked.