Home / internal / hermes-skills / markdown-web-publishing.md

Skill: markdown-web-publishing

Markdown Web Publishing

When to Use

Use this skill when the user wants Markdown (.md) files to appear as styled web pages rather than downloads/plain text, or wants to publish/update the Hermes WWW markdown site.

Typical triggers: - "publish a markdown" - "why does .md download from Synology/Web Station?" - "make my markdown site look like yours" - "update/regenerate the Hermes WWW index" - "run the markdown renderer behind reverse proxy" - "flip this Hermes WWW page between internal and public" - "find the other result / fix the broken Hermes WWW link without changing exposure"

This is the umbrella skill for Hermes WWW publishing. Narrow publication-control guidance belongs here as a subsection/reference, not as separate micro-skills.

Core Model

A static web server alone will not render Markdown as styled HTML. It will usually download .md files or show raw text.

The working pattern is:

browser → reverse proxy → Flask renderer → markdown files under web root

The renderer: 1. serves index.html as the home page, 2. intercepts .md requests, 3. converts Markdown to HTML, 4. wraps it in the shared CSS/theme template, 5. autolinks bare http:// / https:// text in the rendered frontend so URLs remain clickable even when the markdown source used plain text instead of [label](url) syntax, 6. serves only explicitly allowed public file types. For Zico's public Hermes WWW this is currently restricted to Markdown (.md) and HTML (.html); internal files such as published_files.json, backups, scripts, keys, configs, and arbitrary attachments must return 404 even if the path is guessed.

Hermes WWW Paths

Azure VM production setup currently uses:

/home/vmubuntu/www/                       # public content root
/home/vmubuntu/www/published_files.json   # manifest/source of truth for index cards
/home/vmubuntu/www-venv/app.py            # Flask markdown renderer
/home/vmubuntu/www-venv/update_manifest.py# recursive scanner -> JSON manifest
/home/vmubuntu/www-venv/generate_index.py # manifest -> index.html

Zico's Synology setup uses a case-sensitive root:

/volume1/docker/HermesWWW/www
/volume1/docker/HermesWWW/www-venv

See references/zico-synology-hermeswww.md for Synology-specific notes, references/manifest-driven-index.md for the JSON-manifest publishing workflow, references/zico-azure-caddy-domain.md for the Azure custom-domain/no-port HTTPS setup with Caddy + Let's Encrypt, and references/rich-html-research-reports.md for self-contained HTML reports with inline SVG charts, data-source notes, and public-route verification fallbacks.

Publishing a New Markdown/HTML File

Hermes WWW is now manifest-driven. The source of truth is published_files.json, not cards embedded in index.html. See references/manifest-driven-index.md for the exact schema and migration notes.

For rich research deliverables, a standalone .html page is often better than Markdown: use inline CSS, inline SVG charts, semantic headings, tables, and source links. Generate data-heavy HTML from a script outside the public root, then write only the final .html file under /home/vmubuntu/www/. See references/rich-html-research-reports.md for the pattern and verification checklist.

Listing Unpublished Browsable Pages

When Zico asks for unpublished articles/pages, list only human-browsable Markdown/HTML entries — ignore technical files, assets, generated root index.html, source files, images, SVGs, etc.

Use the script at ~/.hermes/scripts/list_unpublished_www_pages.py (also saved under this skill as scripts/list_unpublished_www_pages.py):

/home/vmubuntu/.hermes/scripts/list_unpublished_www_pages.py
# JSON output:
/home/vmubuntu/.hermes/scripts/list_unpublished_www_pages.py --json

The script filters publish: false + kind in {MD, HTML} and excludes the system index.html. It prints Discord-friendly bullet lists with <https://...> autolinks — NOT Markdown tables, which are unreadable in Discord.

IMPORTANT — Discord output format for any list of pages/URLs: - Never output wide Markdown tables (| col | col |) for results sent to Zico via Discord — they render as unreadable plain text. - Use numbered bullet lists with <https://...> autolinks on their own line. - Example correct format: 1. **Název stránky** - typ: `HTML` - soubor: `path/to/file.html` - odkaz: <https://hermes.suchacek.com/path/to/>

Hiding a Page Without Deleting

To hide a published page from the index without deleting the file:

  1. Open /home/vmubuntu/www/published_files.json
  2. Find the entry by filepath and set "publish": false
  3. Run generate_index.py (no need to run update_manifest.py — that would reset the flag)
  4. Verify the homepage no longer contains the href
import json, pathlib
p = pathlib.Path('/home/vmubuntu/www/published_files.json')
data = json.loads(p.read_text())
for item in data['files']:
    if item.get('filepath') == 'target/path/index.html':
        item['publish'] = False
p.write_text(json.dumps(data, ensure_ascii=False, indent=2) + '\n')

Then regenerate index only:

cd /home/vmubuntu/www-venv && ./bin/python generate_index.py

Do NOT run update_manifest.py after manually setting publish: false — it will reset all flags from the filesystem scan and overwrite manual changes.

  1. Create the Markdown/HTML file under the web root.

Example:

mkdir -p /home/vmubuntu/www/ai
cat > /home/vmubuntu/www/ai/example.md <<'EOF'
# Example

Markdown content...
EOF
  1. Update the manifest from the filesystem.

Azure VM:

cd /home/vmubuntu/www-venv
./bin/python update_manifest.py

Synology:

cd /volume1/docker/HermesWWW/www-venv
./venv/bin/python update_manifest.py --root /volume1/docker/HermesWWW/www
  1. Edit published_files.json metadata for the new file if the auto-generated values are not good enough:
{
  "filepath": "ai/example.md",
  "href": "/ai/example.md",
  "kind": "MD",
  "publish": true,
  "internal": false,
  "category": "🤖 AI / agenti / Hermes",
  "title": "Example title",
  "description": "Short description."
}

internal controls which generated index lists the page: - publish: true + internal: false → visible on the main homepage / - publish: true + internal: true → hidden from the main homepage and listed on /internal/ - direct page URLs keep working in both cases

Visibility-control workflow for existing Hermes WWW artifacts

When the task is not "publish new content" but rather find, relink, unpublish, or change visibility of an existing page, treat this as a scope-sensitive workflow:

  1. Locate first, change nothing. Inspect the matching published_files.json entry and report current href, publish, and internal state.
  2. Do not treat discovery as approval. If the user asked for "the other result" or a missing page, identifying the file does not authorize flipping internal/public visibility.
  3. Only after explicit approval may you change internal, publish, or href to move a page between /... and /internal/....
  4. After any approved visibility change, regenerate indexes and verify canonical behavior:
  5. intended route returns 200
  6. obsolete competing route returns 404 when a single canonical route is desired
  7. If visibility was changed by mistake, rollback means restoring the original manifest fields, regenerating indexes, and verifying both routes explicitly.

For the absorbed narrow case history and exact rollback pattern, see references/hermes-www-visibility-controls.md.

  1. Generate index.html from the manifest.

Azure VM:

cd /home/vmubuntu/www-venv
./bin/python generate_index.py

Synology:

cd /volume1/docker/HermesWWW/www-venv
HERMES_WWW_ROOT=/volume1/docker/HermesWWW/www ./venv/bin/python generate_index.py
  1. Verify the public URL before returning it as live. For Zico, always return a Discord-friendly clickable public URL visibly on its own line (full https://... text, not only hidden behind markdown link text). If the file/page is still being built by a Kanban worker, call it a target/cílová URL instead of implying it is ready.

Pitfall: do not manually add new cards to index.html; they will be overwritten. Put durable listing metadata in published_files.json.

Manifest metadata pitfall: update_manifest.py discovers files and technical metadata, but newly added HTML/Markdown pages often get generic category/title/description values. If you already ran generate_index.py and the homepage card looks low-quality, patch only the matching published_files.json entry (category, title, description) and rerun generate_index.py. You do not need to recreate the page or rerun update_manifest.py just to improve card metadata.

For new public technical guides/articles, prefer this publication pattern: - choose a stable topical folder/slug first (for example /azure/..., /dataverse/..., /hermes/...) so the final URL is clean and reusable, - keep the filename URL-safe/ASCII for durability, - then patch the manifest metadata to a human-readable Czech title and a concise description before final verification, - verify not only the page URL but also that the generated homepage card shows the intended title/category/description.

Internal/public listing rule: the manifest also supports internal: true|false. - publish: true + internal: false → page is listed on the main homepage / - publish: true + internal: true → page is hidden from the main homepage and listed on /internal/ - internal pages should use href under /internal/... so the visible/public-facing URL matches the internal namespace - if an internal page is still reachable on the old root path, prefer redirecting the root path to /internal/... so users land on the canonical internal URL - generated index outputs themselves stay non-card entries: /index.html and /internal/index.html

Permission rule for Zico: do not flip an existing page between internal/public visibility unless he explicitly confirms that change. If he asks you to "find" a page/result and you cannot find it, stop and ask which artifact he means before changing internal, publish, or href. Treat visibility changes as scope-changing actions, not as harmless fixes.

Rollback pattern if you switched visibility by mistake: restore the manifest entry (href and internal), rerun generate_index.py, then verify both routes explicitly — old unintended public URL should return 404, intended internal URL should return 200.

Backup rule for public roots: do not create backup files inside /home/vmubuntu/www or another served web root unless they are intentionally public. The manifest scanner can publish or list them, and the Flask renderer may serve them directly by URL. If you need a rollback copy, place it outside the web root (for example /home/vmubuntu/backups/hermes-www/ or /tmp) or immediately delete it and regenerate the manifest/index. After cleanup, verify both the intended page and any accidental backup URL (curl -k -I ...backup...) so backup artifacts return 404 and the main page still returns 200.

Renderer Setup Checklist

In app.py, set the content root:

WWW_ROOT = Path("/path/to/www")

For reverse proxy deployments, prefer local HTTP bind:

if __name__ == "__main__":
    app.run(host="127.0.0.1", port=8090)

Use reverse proxy for TLS rather than embedding cert paths in app.py, unless the host is intentionally running the Flask dev server directly with a self-signed cert.

Custom Domain / No-Port HTTPS with Caddy

When a public Hermes WWW site should be accessed without a port (for example https://hermes.suchacek.com instead of https://IP:8443), put a real reverse proxy on port 443 in front of the Flask renderer. Caddy is the preferred low-ops option because it obtains and renews Let's Encrypt certificates automatically.

Minimal pattern:

browser → Caddy :443 with Let's Encrypt → Flask renderer on localhost or existing private port

For a Flask renderer that is already serving HTTPS with a self-signed cert on 127.0.0.1:8443, Caddy can proxy with upstream verification disabled:

{
    email admin@example.com
}

hermes.example.com {
    encode zstd gzip
    reverse_proxy https://127.0.0.1:8443 {
        transport http {
            tls_insecure_skip_verify
        }
    }
}

Deploy:

sudo apt-get install -y caddy
sudo cp /tmp/Caddyfile /etc/caddy/Caddyfile
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl restart caddy

Verify the exact no-port URL and certificate before reporting success:

curl -I -sS https://hermes.example.com/
echo | openssl s_client -connect hermes.example.com:443 -servername hermes.example.com 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

If Let's Encrypt HTTP-01 validation fails with Timeout during connect on port 80 but port 443 is reachable, Caddy may still succeed via TLS-ALPN-01. Port 80 is still needed if the user wants plain http:// to redirect to HTTPS; open it in the cloud firewall/NSG only if that behavior is required.

Long-term cleaner setup: bind Flask to 127.0.0.1:8090 without TLS and let Caddy terminate TLS. Avoid exposing the Flask development server directly when a production reverse proxy is in use.

Synology No-Container Pattern

If the user does not want containers on Synology:

  1. Install Python 3.11 package.
  2. Create a venv using Python 3.11 explicitly; do not make it the DSM system default.
  3. Run the Flask app through DSM Task Scheduler at boot.
  4. Expose it through DSM Reverse Proxy to 127.0.0.1:8090.
  5. Do not use Web Station Portal to run this Flask app; Web Station is fine for static/PHP/CGI patterns but commonly returns Internal Server Error for this standalone Flask app.run(...) style.

Common Failures

New file disappears from the home page after regeneration

Root cause: listing metadata was added directly to index.html instead of published_files.json.

Fix: run update_manifest.py, edit the file's manifest entry (publish, category, title, description), then run generate_index.py.

.md downloads instead of rendering

Root cause: request is going to static Web Station/static server, not the Flask renderer.

Fix: route browser traffic through reverse proxy to the renderer port.

Manual python app.py works, but LAN URL fails

If bound to 127.0.0.1, it is only reachable locally. For a direct LAN test, temporarily bind 0.0.0.0, or test locally with:

curl http://127.0.0.1:8090

For final reverse proxy setup, 127.0.0.1 is preferred.

Web Station Portal returns Internal Server Error

Root cause: Web Station is not the right execution model for a standalone Flask dev server app.

Fix: Task Scheduler starts Flask; Reverse Proxy forwards to it.

ModuleNotFoundError: No module named 'zoneinfo'

Root cause: script is running under Python < 3.9.

Fix: run via Python 3.11 venv:

cd /path/to/www-venv
python3.11 -m venv venv
source venv/bin/activate
pip install flask markdown pygments
./venv/bin/python generate_index.py

Avoid changing Synology's system default Python.

Bare URLs are visible in markdown source but not clickable in the frontend

Root cause: the markdown source used plain https://... text and the renderer did not autolink bare URLs after Markdown conversion.

Fix: update the Flask renderer to post-process rendered HTML and wrap bare http/https text in anchors while preserving existing <a> links. For Zico's current Hermes WWW, /home/vmubuntu/www-venv/app.py should autolink bare URLs in normal prose and code blocks (leave existing anchors, scripts, and styles untouched).

Verification:

curl -k -sS https://127.0.0.1:8443/path/page.md | grep -n 'href="https://'

If the user says "links aren't clickable," check the rendered HTML first before rewriting all markdown files. Often the durable fix belongs in the renderer, not in every source document.

File and manifest exist, but public URL hangs or returns no bytes

Treat this as a service-readiness issue, not as missing content. Do not recreate the page first.

  1. Confirm the file exists under the web root and has a publish: true manifest entry.
  2. Check the user service and listener:
systemctl --user is-active hermes-www.service
ss -ltnp | grep -E '(:443|:8443|:8090)'
  1. Restart only the renderer if the HTTPS/public request hangs while the file is present:
systemctl --user restart hermes-www.service
  1. Verify both renderer and public Caddy route:
printf 'GET /path/page.html HTTP/1.1\r\nHost: hermes.suchacek.com\r\nConnection: close\r\n\r\n' \
  | timeout 10 openssl s_client -quiet -connect 127.0.0.1:8443 -servername hermes.suchacek.com
curl --resolve hermes.suchacek.com:443:127.0.0.1 -I --max-time 10 https://hermes.suchacek.com/path/page.html
curl -I --max-time 10 https://hermes.suchacek.com/path/page.html

Only publish/regenerate content after these checks prove the page is genuinely absent or unpublished.

Verification

Before reporting a Hermes WWW URL as ready, verify the exact public route, not just local files. A planned slug that has not been created yet will return Werkzeug/Flask 404 NOT FOUND, which must be treated as not published.

When the user asks to check whether a page "still exists" or says to publish only if missing, do the existence checks first and do not regenerate/recreate/publish unless the checks fail. Check, in order: the expected file under /home/vmubuntu/www, the matching published_files.json entry with publish: true, the card/title in generated index.html, and the exact public URL status.

When publishing a bundle of related artifacts (for example: one HTML report plus raw .md templates/examples), verify each public URL individually instead of assuming the parent page is sufficient.

On manifest-driven HermesWWW sites, also verify the metadata in published_files.json for the new entries: title, category, description, and publish: true. If files were created programmatically and the generated metadata is too generic, patch the manifest entries and regenerate index.html again. This matters especially for research reports, template packs, and multi-file deliverables where filename-derived titles are often not good enough for homepage cards.

curl -I http://127.0.0.1:8090/

curl -I -sS https://hermes.suchacek.com/<slug>/

If using the old self-signed fallback route, curl -k -I https://132.164.99.216:8443/<slug>/ is acceptable for diagnostics only.

User-Specific Behavior for Zico