Add archive scraping, webcal feed, and scrape schedule reporting

- scraper.py: --archive flag to scrape https://fundbureau.de/archiv.html
  (past events), fixing a container-specific wait selector and a crash
  on events with no door time that this surfaced.
- api: EventStore now merges an optional archive JSON file into the
  main event list (deduped by date+name); adds GET /events/archive.
- api: GET /calendar.ics serves an RFC 5545 feed of all events for
  webcal subscriptions, with an all-day fallback when no door time is
  parseable.
- api: GET /health reports last-scrape time (from file mtime) and, via
  new SCRAPE_CRON/ARCHIVE_SCRAPE_CRON env vars, the next scheduled run
  and seconds until it, for both the regular and archive scrape.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

README.md

@@ -18,7 +18,7 @@ fundbureau.de  ──scraper──▶  fundi-scraped-output.json  ──api─
 
 - Python 3.11+
 - For the scraper: `playwright`, `beautifulsoup4` (plus `playwright install chromium`)
-- For the API: `fastapi`, `uvicorn` (see [`api/requirements.txt`](api/requirements.txt))
+- For the API: `fastapi`, `uvicorn`, `croniter` (see [`api/requirements.txt`](api/requirements.txt))
 
 ```bash
 pip install beautifulsoup4 playwright
@@ -45,6 +45,21 @@ This writes `fundi-scraped-output.json` to the current directory.
 | `--output-path PATH` | Directory for the output file | `./` |
 | `--load-local-file` | Parse a local HTML file instead of scraping the live site (pass the file path as `--url`) | off |
 | `--ignore-ticket-link` | Don't extract ticket links; leaves `event_ticket_link` empty and `event_free` null | off |
+| `--archive` | Scrape the [archive page](https://fundbureau.de/archiv.html) (past events) instead of the landing page. Changes the defaults to `--url https://fundbureau.de/archiv.html` and `--output-file fundi-archive-output.json`; both can still be overridden explicitly | off |
+
+### Archive (past events)
+
+The Fundbureau site keeps every past event on a separate page,
+[archiv.html](https://fundbureau.de/archiv.html), using the same `.event`
+markup as the landing page. Scrape it with:
+
+```bash
+python scraper/scraper.py --archive
+```
+
+This writes `fundi-archive-output.json`, in the same schema as the regular
+output. Run both commands (regularly, e.g. via a cron job) to keep an
+up-to-date pair of files — the API merges them (see below).
 
 ### Output format
 
@@ -79,8 +94,8 @@ The output is a JSON array of event objects:
 
 ## 2. REST API
 
-Located in [`api/`](api/). It loads a scraped JSON file into memory and exposes it
+Located in [`api/`](api/). It loads the scraped JSON file(s) into memory and
-over HTTP.
+exposes them over HTTP.
 
 ### Run
 
@@ -93,26 +108,32 @@ uvicorn main:app --reload
 - Interactive docs (Swagger UI): http://127.0.0.1:8000/docs
 - OpenAPI schema: http://127.0.0.1:8000/openapi.json
 
-By default the API reads `fundi-scraped-output.json` from the repo root. Override
+By default the API reads `fundi-scraped-output.json` (current/upcoming events)
-with the `DATA_FILE` environment variable:
+and `fundi-archive-output.json` (past events, from `--archive`) from the repo
+root, and merges them into one in-memory list. The archive file is optional —
+if it doesn't exist yet, the API just serves the current events. Override
+either path with an environment variable:
 
 ```bash
-DATA_FILE=/path/to/events.json uvicorn main:app
+DATA_FILE=/path/to/events.json ARCHIVE_DATA_FILE=/path/to/archive.json uvicorn main:app
 ```
 
 ### Endpoints
 
 | Method | Path | Description |
 |--------|------|-------------|
-| `GET`  | `/health` | Service status, number of events loaded, resolved data file path |
+| `GET`  | `/health` | Service status, event counts, resolved data file paths, and last/next scrape timing |
 | `GET`  | `/events` | List events, with optional filters (below) |
-| `GET`  | `/events/{index}` | A single event by its 0-based position in the file; `404` if out of range |
+| `GET`  | `/events/archive` | Past events only — shorthand for `/events?upcoming=false` |
+| `GET`  | `/events/{index}` | A single event by its 0-based position in the merged list; `404` if out of range |
 | `GET`  | `/artists` | Deduplicated, case-insensitively sorted list of all artist names |
-| `POST` | `/reload` | Re-read the JSON file from disk (call after re-running the scraper); `500` if the file is missing |
+| `GET`  | `/calendar.ics` | iCalendar/webcal feed of all events — subscribe from any calendar app |
+| `POST` | `/reload` | Re-read the data file(s) from disk (call after re-running the scraper); `500` if the main file is missing |
 
 ### `/events` query parameters
 
-All filters are optional and combine with AND.
+All filters are optional and combine with AND. `/events/archive` accepts the
+same filters except `upcoming` (it's always `false`).
 
 | Param | Type | Meaning |
 |-------|------|---------|
@@ -124,7 +145,7 @@ All filters are optional and combine with AND.
 
 ### Response shapes
 
-`GET /events` returns:
+`GET /events` and `GET /events/archive` return:
 
 ```json
 {
@@ -136,6 +157,51 @@ All filters are optional and combine with AND.
 `GET /events/{index}` returns a single event object (same schema as the scraper
 output). `GET /artists` returns a plain JSON array of strings.
 
+### Scrape schedule reporting
+
+The scraper is meant to be run periodically via cron on the box hosting the
+API. `GET /health` reports, for both the regular and archive scrape:
+
+- `last_scrape_at` / `seconds_since_last_scrape` — from the output file's
+  mtime, so this works no matter how the scrape was triggered.
+- `next_scrape_at` / `seconds_until_next_scrape` — computed from a cron
+  expression *you provide*, since the API can't reliably read another
+  process's crontab. Set it to match what's actually in cron:
+
+```bash
+SCRAPE_CRON="0 * * * *" ARCHIVE_SCRAPE_CRON="0 4 * * *" uvicorn main:app
+```
+
+`ARCHIVE_SCRAPE_CRON` falls back to `SCRAPE_CRON` if unset (handy if both
+scrapes run off the same cron line). Leaving both unset just omits the
+`next_scrape_at` fields (`null`) — `last_scrape_at` still works.
+
+```json
+"scrape": {
+  "cron_schedule": "0 * * * *",
+  "last_scrape_at": "2026-09-05T13:00:00+02:00",
+  "seconds_since_last_scrape": 42,
+  "next_scrape_at": "2026-09-05T14:00:00+02:00",
+  "seconds_until_next_scrape": 3558
+}
+```
+
+### Calendar / webcal feed
+
+`GET /calendar.ics` renders every loaded event (upcoming + archived) as an
+RFC 5545 `VCALENDAR`. Point a calendar app at it to get an
+auto-refreshing subscription, using the `webcal://` scheme so the app treats
+it as a subscription instead of a one-off download:
+
+```
+webcal://127.0.0.1:8000/calendar.ics
+```
+
+(swap in your deployed host; use `https://` instead of `webcal://` for tools
+that don't understand the `webcal:` scheme, such as `curl`). Events without a
+parseable door time (`event_starttime`) are rendered as all-day entries
+instead of a guessed time.
+
 ### Examples
 
 ```bash
@@ -151,21 +217,29 @@ curl 'http://127.0.0.1:8000/events?artist=randali&upcoming=true'
 # Everything on a given night
 curl 'http://127.0.0.1:8000/events?date=05.09.26'
 
-# First event in the file
+# Past events (archive)
+curl http://127.0.0.1:8000/events/archive
+
+# First event in the merged list
 curl http://127.0.0.1:8000/events/0
 
 # All known artists
 curl http://127.0.0.1:8000/artists
 
+# Calendar feed
+curl http://127.0.0.1:8000/calendar.ics
+
 # Refresh after re-scraping
-python scraper/scraper.py && curl -X POST http://127.0.0.1:8000/reload
+python scraper/scraper.py && python scraper/scraper.py --archive
+curl -X POST http://127.0.0.1:8000/reload
 ```
 
 ## Typical workflow
 
 ```bash
-# 1. Scrape
+# 1. Scrape upcoming events, and (occasionally) the archive
 python scraper/scraper.py
+python scraper/scraper.py --archive
 
 # 2. Serve
 cd api && uvicorn main:app --reload
@@ -180,12 +254,14 @@ curl -X POST http://127.0.0.1:8000/reload
 ```
 .
 ├── scraper/
-│   └── scraper.py            # scrapes fundbureau.de -> JSON
+│   └── scraper.py            # scrapes fundbureau.de (or --archive: archiv.html) -> JSON
 ├── api/
-│   ├── main.py               # FastAPI app + routes
+│   ├── main.py                # FastAPI app + routes
-│   ├── datasource.py         # loads/reloads the JSON, date & artist helpers
+│   ├── datasource.py           # loads/merges/reloads the JSON files, date & artist helpers
-│   ├── models.py             # Pydantic models for the event schema
+│   ├── calendar_feed.py         # builds the /calendar.ics webcal feed
+│   ├── models.py               # Pydantic models for the event schema
 │   ├── requirements.txt
 │   └── README.md
-└── fundi-scraped-output.json # example scraper output / default API datasource
+├── fundi-scraped-output.json  # scraper output: upcoming events / default API datasource
+└── fundi-archive-output.json  # scraper --archive output: past events / archive API datasource
 ```

api/README.md

@@ -19,22 +19,43 @@ uvicorn main:app --reload
 
 Interactive docs: http://127.0.0.1:8000/docs
 
-By default the API reads `../fundi-scraped-output.json` (the repo root).
+By default the API reads `../fundi-scraped-output.json` (upcoming events) and
-Point it elsewhere with an env var:
+`../fundi-archive-output.json` (past events, optional — produced by
+`scraper.py --archive`), merging both into one in-memory list. Point either
+elsewhere with an env var:
 
 ```bash
-DATA_FILE=/path/to/events.json uvicorn main:app
+DATA_FILE=/path/to/events.json ARCHIVE_DATA_FILE=/path/to/archive.json uvicorn main:app
 ```
 
+### Scrape schedule reporting
+
+`/health` reports when each scraper output file was last written (its mtime)
+and, if you tell it the cron schedule, when it's next due. This doesn't read
+your crontab — set the same expression(s) you put there as env vars:
+
+```bash
+SCRAPE_CRON="0 * * * *" ARCHIVE_SCRAPE_CRON="0 4 * * *" uvicorn main:app
+```
+
+`SCRAPE_CRON` covers the plain scrape (`fundi-scraped-output.json`);
+`ARCHIVE_SCRAPE_CRON` covers `--archive` (`fundi-archive-output.json`) and
+falls back to `SCRAPE_CRON` if unset — set it separately only if the archive
+scrape runs on its own cron line. Leave both unset to just get the
+last-scrape info with `next_scrape_at` / `seconds_until_next_scrape` as
+`null`.
+
 ## Endpoints
 
-| Method | Path             | Description |
+| Method | Path              | Description |
-|--------|------------------|-------------|
+|--------|-------------------|-------------|
-| GET    | `/health`        | Status + number of events loaded |
+| GET    | `/health`         | Status, event counts, and last/next scrape timing (see below) |
-| GET    | `/events`        | List events, with optional filters (see below) |
+| GET    | `/events`         | List events, with optional filters (see below) |
-| GET    | `/events/{index}`| Single event by its position in the file (0-based) |
+| GET    | `/events/archive` | Past events only — shorthand for `/events?upcoming=false` |
-| GET    | `/artists`       | Deduplicated, sorted list of all artist names |
+| GET    | `/events/{index}` | Single event by its position in the merged list (0-based) |
-| POST   | `/reload`        | Re-read the JSON file from disk (after a fresh scrape) |
+| GET    | `/artists`        | Deduplicated, sorted list of all artist names |
+| GET    | `/calendar.ics`   | iCalendar/webcal feed of all events — subscribe with `webcal://<host>/calendar.ics` |
+| POST   | `/reload`         | Re-read the data file(s) from disk (after a fresh scrape) |
 
 ### `/events` query parameters
 

api/calendar_feed.py

@@ -0,0 +1,126 @@
+"""Builds an RFC 5545 (iCalendar) feed from events, for webcal subscriptions.
+
+Point a calendar app at ``webcal://<host>/calendar.ics`` (or the plain
+``https://`` URL) to subscribe - the app will periodically re-fetch it.
+"""
+
+import hashlib
+import re
+from datetime import date, datetime, timedelta, timezone
+from typing import List, Optional
+
+from datasource import iter_artist_names, parse_event_date
+from models import Event
+
+_STARTTIME_RE = re.compile(r"(\d{1,2}):(\d{2})")
+_DEFAULT_DURATION = timedelta(hours=5)
+_UID_DOMAIN = "fundbureau-scraper"
+
+
+def _escape_text(value: str) -> str:
+    return (
+        value.replace("\\", "\\\\")
+        .replace(";", "\\;")
+        .replace(",", "\\,")
+        .replace("\n", "\\n")
+    )
+
+
+def _fold_line(line: str) -> str:
+    """Fold a logical line to <=75 octets per line, as RFC 5545 requires."""
+    data = line.encode("utf-8")
+    if len(data) <= 75:
+        return line
+    chunks = []
+    start = 0
+    limit = 75
+    while start < len(data):
+        end = min(start + limit, len(data))
+        # Don't split a multi-byte UTF-8 sequence in half.
+        while end < len(data) and (data[end] & 0xC0) == 0x80:
+            end -= 1
+        chunks.append(data[start:end].decode("utf-8"))
+        start = end
+        limit = 74  # continuation lines are prefixed with a single space
+    return "\r\n ".join(chunks)
+
+
+def _parse_starttime(raw: str) -> Optional[tuple]:
+    match = _STARTTIME_RE.search(raw or "")
+    if not match:
+        return None
+    hour, minute = int(match.group(1)), int(match.group(2))
+    if 0 <= hour < 24 and 0 <= minute < 60:
+        return hour, minute
+    return None
+
+
+def _event_uid(event: Event) -> str:
+    digest = hashlib.sha1(f"{event.event_date}|{event.event_name}".encode("utf-8")).hexdigest()
+    return f"{digest}@{_UID_DOMAIN}"
+
+
+def _build_vevent(event: Event, dtstamp: str) -> List[str]:
+    event_date = parse_event_date(event.event_date)
+    lines = ["BEGIN:VEVENT", f"UID:{_event_uid(event)}", f"DTSTAMP:{dtstamp}"]
+
+    if event_date is None:
+        # Can't place it on the calendar without a parseable date.
+        return []
+
+    starttime = _parse_starttime(event.event_starttime)
+    if starttime is not None:
+        hour, minute = starttime
+        dtstart = datetime(event_date.year, event_date.month, event_date.day, hour, minute)
+        dtend = dtstart + _DEFAULT_DURATION
+        lines.append(f"DTSTART:{dtstart.strftime('%Y%m%dT%H%M%S')}")
+        lines.append(f"DTEND:{dtend.strftime('%Y%m%dT%H%M%S')}")
+    else:
+        # No parseable door time - render as an all-day event instead of
+        # guessing a time.
+        next_day = event_date + timedelta(days=1)
+        lines.append(f"DTSTART;VALUE=DATE:{event_date.strftime('%Y%m%d')}")
+        lines.append(f"DTEND;VALUE=DATE:{next_day.strftime('%Y%m%d')}")
+
+    lines.append(f"SUMMARY:{_escape_text(event.event_name)}")
+    lines.append("LOCATION:" + _escape_text("Fundbureau, Hamburg"))
+
+    description_parts = []
+    lineup = ", ".join(iter_artist_names(event))
+    if lineup:
+        description_parts.append(f"Line-up: {lineup}")
+    if event.event_starttime:
+        description_parts.append(event.event_starttime)
+    if event.event_free:
+        description_parts.append("Eintritt frei")
+    elif event.event_ticket_link and event.event_ticket_link != "n/a":
+        description_parts.append(f"Tickets: {event.event_ticket_link}")
+    if description_parts:
+        lines.append(f"DESCRIPTION:{_escape_text(chr(10).join(description_parts))}")
+
+    if event.event_ticket_link and event.event_ticket_link not in ("", "n/a"):
+        lines.append(f"URL:{event.event_ticket_link}")
+
+    lines.append("END:VEVENT")
+    return lines
+
+
+def build_ics(events: List[Event]) -> str:
+    """Render a list of events as a VCALENDAR document (CRLF-terminated)."""
+    dtstamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+    lines = [
+        "BEGIN:VCALENDAR",
+        "VERSION:2.0",
+        "PRODID:-//fundi-scraper-api//Fundbureau Events//DE",
+        "CALSCALE:GREGORIAN",
+        "METHOD:PUBLISH",
+        "X-WR-CALNAME:Fundbureau Events",
+        "X-WR-CALDESC:Upcoming and past events at the Fundbureau, Hamburg",
+        "X-WR-TIMEZONE:Europe/Berlin",
+        "REFRESH-INTERVAL;VALUE=DURATION:PT12H",
+        "X-PUBLISHED-TTL:PT12H",
+    ]
+    for event in events:
+        lines.extend(_build_vevent(event, dtstamp))
+    lines.append("END:VCALENDAR")
+    return "\r\n".join(_fold_line(line) for line in lines) + "\r\n"

api/datasource.py

@@ -1,7 +1,14 @@
-"""Loads the scraped events JSON file and keeps it in memory.
+"""Loads the scraped events JSON file(s) and keeps them in memory.
 
-The file is treated as read-only. Call ``reload()`` to pick up a fresh
+Files are treated as read-only. Call ``reload()`` to pick up a fresh
 scrape without restarting the server.
+
+Two files are read:
+  - the main data file (upcoming events, produced by a plain
+    ``scraper.py`` run) - required, resolved from DATA_FILE.
+  - the archive file (past events, produced by ``scraper.py --archive``)
+    - optional, resolved from ARCHIVE_DATA_FILE. Missing archive file
+    just means no archived events are available yet.
 """
 
 import json
@@ -13,28 +20,54 @@ from typing import List, Optional
 
 from models import Event
 
-# Default: the fundi-scraped-output.json sitting in the repo root, one
+# Defaults: the fundi-*-output.json files sitting in the repo root, one
-# directory above this file. Override with the DATA_FILE env var.
+# directory above this file. Override with the DATA_FILE / ARCHIVE_DATA_FILE
+# env vars.
 _DEFAULT_DATA_FILE = Path(__file__).resolve().parent.parent / "fundi-scraped-output.json"
+_DEFAULT_ARCHIVE_DATA_FILE = Path(__file__).resolve().parent.parent / "fundi-archive-output.json"
 
 
 class EventStore:
-    def __init__(self, path: Optional[os.PathLike] = None):
+    def __init__(self, path: Optional[os.PathLike] = None, archive_path: Optional[os.PathLike] = None):
         self.path = Path(path or os.environ.get("DATA_FILE") or _DEFAULT_DATA_FILE)
+        self.archive_path = Path(archive_path or os.environ.get("ARCHIVE_DATA_FILE") or _DEFAULT_ARCHIVE_DATA_FILE)
         self._lock = Lock()
         self._events: List[Event] = []
+        self._current_count = 0
+        self._archive_count = 0
         self.reload()
 
+    @staticmethod
+    def _load_file(path: Path) -> List[Event]:
+        if not path.exists():
+            return []
+        with path.open(encoding="utf-8") as fh:
+            raw = json.load(fh)
+        return [Event.model_validate(item) for item in raw]
+
     def reload(self) -> int:
-        """Re-read the JSON file from disk. Returns the number of events loaded."""
+        """Re-read the data file(s) from disk. Returns the number of events loaded."""
         if not self.path.exists():
             raise FileNotFoundError(f"Data file not found: {self.path}")
-        with self.path.open(encoding="utf-8") as fh:
+        current = self._load_file(self.path)
-            raw = json.load(fh)
+        archive = self._load_file(self.archive_path)
-        events = [Event.model_validate(item) for item in raw]
+
+        # Merge, de-duplicating on (date, name) in case an event that was
+        # scraped while upcoming was later also picked up by an archive
+        # scrape - the current-file copy wins.
+        seen = {(event.event_date, event.event_name) for event in current}
+        merged = list(current)
+        for event in archive:
+            key = (event.event_date, event.event_name)
+            if key not in seen:
+                merged.append(event)
+                seen.add(key)
+
         with self._lock:
-            self._events = events
+            self._events = merged
-        return len(events)
+            self._current_count = len(current)
+            self._archive_count = len(merged) - len(current)
+        return len(merged)
 
     def all(self) -> List[Event]:
         with self._lock:
@@ -46,6 +79,16 @@ class EventStore:
                 return self._events[index]
         return None
 
+    @property
+    def current_count(self) -> int:
+        with self._lock:
+            return self._current_count
+
+    @property
+    def archive_count(self) -> int:
+        with self._lock:
+            return self._archive_count
+
 
 def parse_event_date(value: str) -> Optional[date]:
     """Parse the scraper's ``dd.mm.yy`` date strings. Returns None if unparseable."""

api/main.py

@@ -8,12 +8,16 @@ Run with:
 """
 
 import datetime
+import os
 from typing import List, Optional
 
 from fastapi import FastAPI, HTTPException, Query
+from fastapi.responses import PlainTextResponse
 
+from calendar_feed import build_ics
 from datasource import EventStore, iter_artist_names, parse_event_date
 from models import Event, EventList
+from scrape_schedule import get_schedule_info
 
 app = FastAPI(
     title="Fundi Scraper API",
@@ -55,17 +59,34 @@ def _matches(
 
 @app.get("/health")
 def health():
-    return {"status": "ok", "events_loaded": len(store.all()), "data_file": str(store.path)}
+    scrape_cron = os.environ.get("SCRAPE_CRON")
+    archive_scrape_cron = os.environ.get("ARCHIVE_SCRAPE_CRON") or scrape_cron
+    return {
+        "status": "ok",
+        "events_loaded": len(store.all()),
+        "data_file": str(store.path),
+        "current_events": store.current_count,
+        "archive_file": str(store.archive_path),
+        "archive_file_found": store.archive_path.exists(),
+        "archive_events": store.archive_count,
+        "scrape": get_schedule_info(store.path, scrape_cron),
+        "archive_scrape": get_schedule_info(store.archive_path, archive_scrape_cron),
+    }
 
 
 @app.post("/reload")
 def reload():
-    """Re-read the JSON file from disk (e.g. after a fresh scrape)."""
+    """Re-read the data file(s) from disk (e.g. after a fresh scrape)."""
     try:
         count = store.reload()
     except FileNotFoundError as exc:
         raise HTTPException(status_code=500, detail=str(exc))
-    return {"reloaded": True, "events_loaded": count}
+    return {
+        "reloaded": True,
+        "events_loaded": count,
+        "current_events": store.current_count,
+        "archive_events": store.archive_count,
+    }
 
 
 @app.get("/events", response_model=EventList)
@@ -94,6 +115,38 @@ def list_artists():
     return sorted(names, key=str.lower)
 
 
+@app.get("/events/archive", response_model=EventList)
+def list_archive_events(
+    free: Optional[bool] = Query(None, description="Filter by free admission"),
+    name: Optional[str] = Query(None, description="Case-insensitive substring match on event name"),
+    artist: Optional[str] = Query(None, description="Case-insensitive substring match on any artist name"),
+    date: Optional[str] = Query(None, description="Exact match on the raw event date (dd.mm.yy)"),
+):
+    """Past events only - shorthand for ``/events?upcoming=false``."""
+    today = datetime.date.today()
+    events = [
+        e
+        for e in store.all()
+        if _matches(e, free, name, artist, date, upcoming=False, today=today)
+    ]
+    return EventList(count=len(events), events=events)
+
+
+@app.get("/calendar.ics")
+def calendar_ics():
+    """iCalendar feed of all known events (upcoming + archived).
+
+    Subscribe from a calendar app with ``webcal://<host>/calendar.ics``
+    (or the plain https URL) to get an auto-refreshing calendar.
+    """
+    ics = build_ics(store.all())
+    return PlainTextResponse(
+        content=ics,
+        media_type="text/calendar",
+        headers={"Content-Disposition": "inline; filename=fundbureau-events.ics"},
+    )
+
+
 @app.get("/events/{index}", response_model=Event)
 def get_event(index: int):
     event = store.get(index)

api/requirements.txt

@@ -1,2 +1,3 @@
 fastapi>=0.103
 uvicorn[standard]>=0.23
+croniter>=2.0

api/scrape_schedule.py

@@ -0,0 +1,58 @@
+"""Reports when a scrape last ran and when cron will next run it.
+
+"Last ran" is read from the mtime of the scraper's output file - true
+regardless of how/when the scrape was actually invoked. "Next run" requires
+telling this module the cron schedule via an env var (there's no reliable,
+portable way to introspect another process's crontab from here), given as
+a standard 5-field cron expression, e.g. "0 * * * *".
+"""
+
+from datetime import datetime
+from pathlib import Path
+from typing import Optional
+
+from croniter import croniter
+
+
+def _now_local() -> datetime:
+    return datetime.now().astimezone()
+
+
+def _file_mtime_local(path: Path) -> Optional[datetime]:
+    try:
+        return datetime.fromtimestamp(path.stat().st_mtime).astimezone()
+    except FileNotFoundError:
+        return None
+
+
+def _next_cron_run(cron_expr: str, after: datetime) -> Optional[datetime]:
+    try:
+        naive_after = after.replace(tzinfo=None)
+        return croniter(cron_expr, naive_after).get_next(datetime).astimezone()
+    except (ValueError, KeyError):
+        return None
+
+
+def get_schedule_info(path: Path, cron_expr: Optional[str]) -> dict:
+    """Build the last/next-scrape block for one scraper output file.
+
+    ``cron_expr`` is that scrape's cron expression (e.g. "0 * * * *");
+    None/invalid -> next-run fields are null.
+    """
+    now = _now_local()
+    cron_expr = (cron_expr or "").strip() or None
+
+    last_scrape_at = _file_mtime_local(path)
+    next_scrape_at = _next_cron_run(cron_expr, now) if cron_expr else None
+
+    return {
+        "cron_schedule": cron_expr,
+        "last_scrape_at": last_scrape_at.isoformat() if last_scrape_at else None,
+        "seconds_since_last_scrape": (
+            round((now - last_scrape_at).total_seconds()) if last_scrape_at else None
+        ),
+        "next_scrape_at": next_scrape_at.isoformat() if next_scrape_at else None,
+        "seconds_until_next_scrape": (
+            round((next_scrape_at - now).total_seconds()) if next_scrape_at else None
+        ),
+    }

scraper/scraper.py

@@ -8,6 +8,8 @@ load_local_file = False
 output_file = "fundi-scraped-output.json"
 output_path = "./"
 ignore_ticket_link = False
+ARCHIVE_URL = "https://fundbureau.de/archiv.html"
+ARCHIVE_OUTPUT_FILE = "fundi-archive-output.json"
 
 argp = argparse.ArgumentParser()
 argp.add_argument("--url", type=str, help="URL to scrape (default: https://fundbureau.de/)")
@@ -15,8 +17,12 @@ argp.add_argument("--output-file", type=str, help="Output JSON file name (defaul
 argp.add_argument("--output-path", type=str, help="Output path for the JSON file (default: current directory)")
 argp.add_argument("--load-local-file", action="store_true", help="Load local HTML file instead of scraping the website (default: False)")
 argp.add_argument("--ignore-ticket-link", action="store_true", help="Ignore ticket link extraction (will leave it empty in the output JSON)")
+argp.add_argument("--archive", action="store_true", help=f"Scrape the past-events archive ({ARCHIVE_URL}) instead of upcoming events. Changes the default url/output-file, both can still be overridden with --url/--output-file.")
 args = argp.parse_args()
 
+if vars(args)["archive"]:
+    scrape_url = ARCHIVE_URL
+    output_file = ARCHIVE_OUTPUT_FILE
 if vars(args)["url"]:
     scrape_url = vars(args)["url"]
 if vars(args)["output_file"]:
@@ -42,8 +48,13 @@ def scrape_website(url, load_local_file):
             page = browser.new_page()
             page.goto(url, wait_until="networkidle")
             try:
-                # Wait until at least one .event div actually renders in the DOM
+                # Wait until at least one .event div actually renders in the DOM.
-                page.wait_for_selector("#upcoming-events-container .event", timeout=15000)
+                # Upcoming events live in #upcoming-events-container, past/archive
+                # events in #past-events-container - match either.
+                page.wait_for_selector(
+                    "#upcoming-events-container .event, #past-events-container .event",
+                    timeout=15000,
+                )
             except Exception:
                 print("Warning: no .event elements appeared before timeout.")
             html = page.content()
@@ -86,7 +97,8 @@ def extract_events(soup, ignore_ticket_link):
                 })
 
 
-        event_starttime = event.find("div", class_="event-starttime").text.strip()
+        event_starttime_tag = event.find("div", class_="event-starttime")
+        event_starttime = event_starttime_tag.text.strip() if event_starttime_tag else ""
         if ignore_ticket_link:
             event_ticket_link = ""
             event_free = None