diff --git a/README.md b/README.md index baf3e46..02e210b 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ fundbureau.de ──scraper──▶ fundi-scraped-output.json ──api─ - Python 3.11+ - For the scraper: `playwright`, `beautifulsoup4` (plus `playwright install chromium`) -- For the API: `fastapi`, `uvicorn` (see [`api/requirements.txt`](api/requirements.txt)) +- For the API: `fastapi`, `uvicorn`, `croniter` (see [`api/requirements.txt`](api/requirements.txt)) ```bash pip install beautifulsoup4 playwright @@ -45,6 +45,21 @@ This writes `fundi-scraped-output.json` to the current directory. | `--output-path PATH` | Directory for the output file | `./` | | `--load-local-file` | Parse a local HTML file instead of scraping the live site (pass the file path as `--url`) | off | | `--ignore-ticket-link` | Don't extract ticket links; leaves `event_ticket_link` empty and `event_free` null | off | +| `--archive` | Scrape the [archive page](https://fundbureau.de/archiv.html) (past events) instead of the landing page. Changes the defaults to `--url https://fundbureau.de/archiv.html` and `--output-file fundi-archive-output.json`; both can still be overridden explicitly | off | + +### Archive (past events) + +The Fundbureau site keeps every past event on a separate page, +[archiv.html](https://fundbureau.de/archiv.html), using the same `.event` +markup as the landing page. Scrape it with: + +```bash +python scraper/scraper.py --archive +``` + +This writes `fundi-archive-output.json`, in the same schema as the regular +output. Run both commands (regularly, e.g. via a cron job) to keep an +up-to-date pair of files — the API merges them (see below). ### Output format @@ -79,8 +94,8 @@ The output is a JSON array of event objects: ## 2. REST API -Located in [`api/`](api/). It loads a scraped JSON file into memory and exposes it -over HTTP. +Located in [`api/`](api/). It loads the scraped JSON file(s) into memory and +exposes them over HTTP. ### Run @@ -93,26 +108,32 @@ uvicorn main:app --reload - Interactive docs (Swagger UI): http://127.0.0.1:8000/docs - OpenAPI schema: http://127.0.0.1:8000/openapi.json -By default the API reads `fundi-scraped-output.json` from the repo root. Override -with the `DATA_FILE` environment variable: +By default the API reads `fundi-scraped-output.json` (current/upcoming events) +and `fundi-archive-output.json` (past events, from `--archive`) from the repo +root, and merges them into one in-memory list. The archive file is optional — +if it doesn't exist yet, the API just serves the current events. Override +either path with an environment variable: ```bash -DATA_FILE=/path/to/events.json uvicorn main:app +DATA_FILE=/path/to/events.json ARCHIVE_DATA_FILE=/path/to/archive.json uvicorn main:app ``` ### Endpoints | Method | Path | Description | |--------|------|-------------| -| `GET` | `/health` | Service status, number of events loaded, resolved data file path | +| `GET` | `/health` | Service status, event counts, resolved data file paths, and last/next scrape timing | | `GET` | `/events` | List events, with optional filters (below) | -| `GET` | `/events/{index}` | A single event by its 0-based position in the file; `404` if out of range | +| `GET` | `/events/archive` | Past events only — shorthand for `/events?upcoming=false` | +| `GET` | `/events/{index}` | A single event by its 0-based position in the merged list; `404` if out of range | | `GET` | `/artists` | Deduplicated, case-insensitively sorted list of all artist names | -| `POST` | `/reload` | Re-read the JSON file from disk (call after re-running the scraper); `500` if the file is missing | +| `GET` | `/calendar.ics` | iCalendar/webcal feed of all events — subscribe from any calendar app | +| `POST` | `/reload` | Re-read the data file(s) from disk (call after re-running the scraper); `500` if the main file is missing | ### `/events` query parameters -All filters are optional and combine with AND. +All filters are optional and combine with AND. `/events/archive` accepts the +same filters except `upcoming` (it's always `false`). | Param | Type | Meaning | |-------|------|---------| @@ -124,7 +145,7 @@ All filters are optional and combine with AND. ### Response shapes -`GET /events` returns: +`GET /events` and `GET /events/archive` return: ```json { @@ -136,6 +157,51 @@ All filters are optional and combine with AND. `GET /events/{index}` returns a single event object (same schema as the scraper output). `GET /artists` returns a plain JSON array of strings. +### Scrape schedule reporting + +The scraper is meant to be run periodically via cron on the box hosting the +API. `GET /health` reports, for both the regular and archive scrape: + +- `last_scrape_at` / `seconds_since_last_scrape` — from the output file's + mtime, so this works no matter how the scrape was triggered. +- `next_scrape_at` / `seconds_until_next_scrape` — computed from a cron + expression *you provide*, since the API can't reliably read another + process's crontab. Set it to match what's actually in cron: + +```bash +SCRAPE_CRON="0 * * * *" ARCHIVE_SCRAPE_CRON="0 4 * * *" uvicorn main:app +``` + +`ARCHIVE_SCRAPE_CRON` falls back to `SCRAPE_CRON` if unset (handy if both +scrapes run off the same cron line). Leaving both unset just omits the +`next_scrape_at` fields (`null`) — `last_scrape_at` still works. + +```json +"scrape": { + "cron_schedule": "0 * * * *", + "last_scrape_at": "2026-09-05T13:00:00+02:00", + "seconds_since_last_scrape": 42, + "next_scrape_at": "2026-09-05T14:00:00+02:00", + "seconds_until_next_scrape": 3558 +} +``` + +### Calendar / webcal feed + +`GET /calendar.ics` renders every loaded event (upcoming + archived) as an +RFC 5545 `VCALENDAR`. Point a calendar app at it to get an +auto-refreshing subscription, using the `webcal://` scheme so the app treats +it as a subscription instead of a one-off download: + +``` +webcal://127.0.0.1:8000/calendar.ics +``` + +(swap in your deployed host; use `https://` instead of `webcal://` for tools +that don't understand the `webcal:` scheme, such as `curl`). Events without a +parseable door time (`event_starttime`) are rendered as all-day entries +instead of a guessed time. + ### Examples ```bash @@ -151,21 +217,29 @@ curl 'http://127.0.0.1:8000/events?artist=randali&upcoming=true' # Everything on a given night curl 'http://127.0.0.1:8000/events?date=05.09.26' -# First event in the file +# Past events (archive) +curl http://127.0.0.1:8000/events/archive + +# First event in the merged list curl http://127.0.0.1:8000/events/0 # All known artists curl http://127.0.0.1:8000/artists +# Calendar feed +curl http://127.0.0.1:8000/calendar.ics + # Refresh after re-scraping -python scraper/scraper.py && curl -X POST http://127.0.0.1:8000/reload +python scraper/scraper.py && python scraper/scraper.py --archive +curl -X POST http://127.0.0.1:8000/reload ``` ## Typical workflow ```bash -# 1. Scrape +# 1. Scrape upcoming events, and (occasionally) the archive python scraper/scraper.py +python scraper/scraper.py --archive # 2. Serve cd api && uvicorn main:app --reload @@ -180,12 +254,14 @@ curl -X POST http://127.0.0.1:8000/reload ``` . ├── scraper/ -│ └── scraper.py # scrapes fundbureau.de -> JSON +│ └── scraper.py # scrapes fundbureau.de (or --archive: archiv.html) -> JSON ├── api/ -│ ├── main.py # FastAPI app + routes -│ ├── datasource.py # loads/reloads the JSON, date & artist helpers -│ ├── models.py # Pydantic models for the event schema +│ ├── main.py # FastAPI app + routes +│ ├── datasource.py # loads/merges/reloads the JSON files, date & artist helpers +│ ├── calendar_feed.py # builds the /calendar.ics webcal feed +│ ├── models.py # Pydantic models for the event schema │ ├── requirements.txt │ └── README.md -└── fundi-scraped-output.json # example scraper output / default API datasource +├── fundi-scraped-output.json # scraper output: upcoming events / default API datasource +└── fundi-archive-output.json # scraper --archive output: past events / archive API datasource ``` diff --git a/api/README.md b/api/README.md index 3e29ecb..8e4da39 100644 --- a/api/README.md +++ b/api/README.md @@ -19,22 +19,43 @@ uvicorn main:app --reload Interactive docs: http://127.0.0.1:8000/docs -By default the API reads `../fundi-scraped-output.json` (the repo root). -Point it elsewhere with an env var: +By default the API reads `../fundi-scraped-output.json` (upcoming events) and +`../fundi-archive-output.json` (past events, optional — produced by +`scraper.py --archive`), merging both into one in-memory list. Point either +elsewhere with an env var: ```bash -DATA_FILE=/path/to/events.json uvicorn main:app +DATA_FILE=/path/to/events.json ARCHIVE_DATA_FILE=/path/to/archive.json uvicorn main:app ``` +### Scrape schedule reporting + +`/health` reports when each scraper output file was last written (its mtime) +and, if you tell it the cron schedule, when it's next due. This doesn't read +your crontab — set the same expression(s) you put there as env vars: + +```bash +SCRAPE_CRON="0 * * * *" ARCHIVE_SCRAPE_CRON="0 4 * * *" uvicorn main:app +``` + +`SCRAPE_CRON` covers the plain scrape (`fundi-scraped-output.json`); +`ARCHIVE_SCRAPE_CRON` covers `--archive` (`fundi-archive-output.json`) and +falls back to `SCRAPE_CRON` if unset — set it separately only if the archive +scrape runs on its own cron line. Leave both unset to just get the +last-scrape info with `next_scrape_at` / `seconds_until_next_scrape` as +`null`. + ## Endpoints -| Method | Path | Description | -|--------|------------------|-------------| -| GET | `/health` | Status + number of events loaded | -| GET | `/events` | List events, with optional filters (see below) | -| GET | `/events/{index}`| Single event by its position in the file (0-based) | -| GET | `/artists` | Deduplicated, sorted list of all artist names | -| POST | `/reload` | Re-read the JSON file from disk (after a fresh scrape) | +| Method | Path | Description | +|--------|-------------------|-------------| +| GET | `/health` | Status, event counts, and last/next scrape timing (see below) | +| GET | `/events` | List events, with optional filters (see below) | +| GET | `/events/archive` | Past events only — shorthand for `/events?upcoming=false` | +| GET | `/events/{index}` | Single event by its position in the merged list (0-based) | +| GET | `/artists` | Deduplicated, sorted list of all artist names | +| GET | `/calendar.ics` | iCalendar/webcal feed of all events — subscribe with `webcal:///calendar.ics` | +| POST | `/reload` | Re-read the data file(s) from disk (after a fresh scrape) | ### `/events` query parameters diff --git a/api/calendar_feed.py b/api/calendar_feed.py new file mode 100644 index 0000000..cc07d96 --- /dev/null +++ b/api/calendar_feed.py @@ -0,0 +1,126 @@ +"""Builds an RFC 5545 (iCalendar) feed from events, for webcal subscriptions. + +Point a calendar app at ``webcal:///calendar.ics`` (or the plain +``https://`` URL) to subscribe - the app will periodically re-fetch it. +""" + +import hashlib +import re +from datetime import date, datetime, timedelta, timezone +from typing import List, Optional + +from datasource import iter_artist_names, parse_event_date +from models import Event + +_STARTTIME_RE = re.compile(r"(\d{1,2}):(\d{2})") +_DEFAULT_DURATION = timedelta(hours=5) +_UID_DOMAIN = "fundbureau-scraper" + + +def _escape_text(value: str) -> str: + return ( + value.replace("\\", "\\\\") + .replace(";", "\\;") + .replace(",", "\\,") + .replace("\n", "\\n") + ) + + +def _fold_line(line: str) -> str: + """Fold a logical line to <=75 octets per line, as RFC 5545 requires.""" + data = line.encode("utf-8") + if len(data) <= 75: + return line + chunks = [] + start = 0 + limit = 75 + while start < len(data): + end = min(start + limit, len(data)) + # Don't split a multi-byte UTF-8 sequence in half. + while end < len(data) and (data[end] & 0xC0) == 0x80: + end -= 1 + chunks.append(data[start:end].decode("utf-8")) + start = end + limit = 74 # continuation lines are prefixed with a single space + return "\r\n ".join(chunks) + + +def _parse_starttime(raw: str) -> Optional[tuple]: + match = _STARTTIME_RE.search(raw or "") + if not match: + return None + hour, minute = int(match.group(1)), int(match.group(2)) + if 0 <= hour < 24 and 0 <= minute < 60: + return hour, minute + return None + + +def _event_uid(event: Event) -> str: + digest = hashlib.sha1(f"{event.event_date}|{event.event_name}".encode("utf-8")).hexdigest() + return f"{digest}@{_UID_DOMAIN}" + + +def _build_vevent(event: Event, dtstamp: str) -> List[str]: + event_date = parse_event_date(event.event_date) + lines = ["BEGIN:VEVENT", f"UID:{_event_uid(event)}", f"DTSTAMP:{dtstamp}"] + + if event_date is None: + # Can't place it on the calendar without a parseable date. + return [] + + starttime = _parse_starttime(event.event_starttime) + if starttime is not None: + hour, minute = starttime + dtstart = datetime(event_date.year, event_date.month, event_date.day, hour, minute) + dtend = dtstart + _DEFAULT_DURATION + lines.append(f"DTSTART:{dtstart.strftime('%Y%m%dT%H%M%S')}") + lines.append(f"DTEND:{dtend.strftime('%Y%m%dT%H%M%S')}") + else: + # No parseable door time - render as an all-day event instead of + # guessing a time. + next_day = event_date + timedelta(days=1) + lines.append(f"DTSTART;VALUE=DATE:{event_date.strftime('%Y%m%d')}") + lines.append(f"DTEND;VALUE=DATE:{next_day.strftime('%Y%m%d')}") + + lines.append(f"SUMMARY:{_escape_text(event.event_name)}") + lines.append("LOCATION:" + _escape_text("Fundbureau, Hamburg")) + + description_parts = [] + lineup = ", ".join(iter_artist_names(event)) + if lineup: + description_parts.append(f"Line-up: {lineup}") + if event.event_starttime: + description_parts.append(event.event_starttime) + if event.event_free: + description_parts.append("Eintritt frei") + elif event.event_ticket_link and event.event_ticket_link != "n/a": + description_parts.append(f"Tickets: {event.event_ticket_link}") + if description_parts: + lines.append(f"DESCRIPTION:{_escape_text(chr(10).join(description_parts))}") + + if event.event_ticket_link and event.event_ticket_link not in ("", "n/a"): + lines.append(f"URL:{event.event_ticket_link}") + + lines.append("END:VEVENT") + return lines + + +def build_ics(events: List[Event]) -> str: + """Render a list of events as a VCALENDAR document (CRLF-terminated).""" + dtstamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + lines = [ + "BEGIN:VCALENDAR", + "VERSION:2.0", + "PRODID:-//fundi-scraper-api//Fundbureau Events//DE", + "CALSCALE:GREGORIAN", + "METHOD:PUBLISH", + "X-WR-CALNAME:Fundbureau Events", + "X-WR-CALDESC:Upcoming and past events at the Fundbureau, Hamburg", + "X-WR-TIMEZONE:Europe/Berlin", + "REFRESH-INTERVAL;VALUE=DURATION:PT12H", + "X-PUBLISHED-TTL:PT12H", + ] + for event in events: + lines.extend(_build_vevent(event, dtstamp)) + lines.append("END:VCALENDAR") + return "\r\n".join(_fold_line(line) for line in lines) + "\r\n" diff --git a/api/datasource.py b/api/datasource.py index c8204e8..e439dcb 100644 --- a/api/datasource.py +++ b/api/datasource.py @@ -1,7 +1,14 @@ -"""Loads the scraped events JSON file and keeps it in memory. +"""Loads the scraped events JSON file(s) and keeps them in memory. -The file is treated as read-only. Call ``reload()`` to pick up a fresh +Files are treated as read-only. Call ``reload()`` to pick up a fresh scrape without restarting the server. + +Two files are read: + - the main data file (upcoming events, produced by a plain + ``scraper.py`` run) - required, resolved from DATA_FILE. + - the archive file (past events, produced by ``scraper.py --archive``) + - optional, resolved from ARCHIVE_DATA_FILE. Missing archive file + just means no archived events are available yet. """ import json @@ -13,28 +20,54 @@ from typing import List, Optional from models import Event -# Default: the fundi-scraped-output.json sitting in the repo root, one -# directory above this file. Override with the DATA_FILE env var. +# Defaults: the fundi-*-output.json files sitting in the repo root, one +# directory above this file. Override with the DATA_FILE / ARCHIVE_DATA_FILE +# env vars. _DEFAULT_DATA_FILE = Path(__file__).resolve().parent.parent / "fundi-scraped-output.json" +_DEFAULT_ARCHIVE_DATA_FILE = Path(__file__).resolve().parent.parent / "fundi-archive-output.json" class EventStore: - def __init__(self, path: Optional[os.PathLike] = None): + def __init__(self, path: Optional[os.PathLike] = None, archive_path: Optional[os.PathLike] = None): self.path = Path(path or os.environ.get("DATA_FILE") or _DEFAULT_DATA_FILE) + self.archive_path = Path(archive_path or os.environ.get("ARCHIVE_DATA_FILE") or _DEFAULT_ARCHIVE_DATA_FILE) self._lock = Lock() self._events: List[Event] = [] + self._current_count = 0 + self._archive_count = 0 self.reload() + @staticmethod + def _load_file(path: Path) -> List[Event]: + if not path.exists(): + return [] + with path.open(encoding="utf-8") as fh: + raw = json.load(fh) + return [Event.model_validate(item) for item in raw] + def reload(self) -> int: - """Re-read the JSON file from disk. Returns the number of events loaded.""" + """Re-read the data file(s) from disk. Returns the number of events loaded.""" if not self.path.exists(): raise FileNotFoundError(f"Data file not found: {self.path}") - with self.path.open(encoding="utf-8") as fh: - raw = json.load(fh) - events = [Event.model_validate(item) for item in raw] + current = self._load_file(self.path) + archive = self._load_file(self.archive_path) + + # Merge, de-duplicating on (date, name) in case an event that was + # scraped while upcoming was later also picked up by an archive + # scrape - the current-file copy wins. + seen = {(event.event_date, event.event_name) for event in current} + merged = list(current) + for event in archive: + key = (event.event_date, event.event_name) + if key not in seen: + merged.append(event) + seen.add(key) + with self._lock: - self._events = events - return len(events) + self._events = merged + self._current_count = len(current) + self._archive_count = len(merged) - len(current) + return len(merged) def all(self) -> List[Event]: with self._lock: @@ -46,6 +79,16 @@ class EventStore: return self._events[index] return None + @property + def current_count(self) -> int: + with self._lock: + return self._current_count + + @property + def archive_count(self) -> int: + with self._lock: + return self._archive_count + def parse_event_date(value: str) -> Optional[date]: """Parse the scraper's ``dd.mm.yy`` date strings. Returns None if unparseable.""" diff --git a/api/main.py b/api/main.py index b1ac055..4cb50fa 100644 --- a/api/main.py +++ b/api/main.py @@ -8,12 +8,16 @@ Run with: """ import datetime +import os from typing import List, Optional from fastapi import FastAPI, HTTPException, Query +from fastapi.responses import PlainTextResponse +from calendar_feed import build_ics from datasource import EventStore, iter_artist_names, parse_event_date from models import Event, EventList +from scrape_schedule import get_schedule_info app = FastAPI( title="Fundi Scraper API", @@ -55,17 +59,34 @@ def _matches( @app.get("/health") def health(): - return {"status": "ok", "events_loaded": len(store.all()), "data_file": str(store.path)} + scrape_cron = os.environ.get("SCRAPE_CRON") + archive_scrape_cron = os.environ.get("ARCHIVE_SCRAPE_CRON") or scrape_cron + return { + "status": "ok", + "events_loaded": len(store.all()), + "data_file": str(store.path), + "current_events": store.current_count, + "archive_file": str(store.archive_path), + "archive_file_found": store.archive_path.exists(), + "archive_events": store.archive_count, + "scrape": get_schedule_info(store.path, scrape_cron), + "archive_scrape": get_schedule_info(store.archive_path, archive_scrape_cron), + } @app.post("/reload") def reload(): - """Re-read the JSON file from disk (e.g. after a fresh scrape).""" + """Re-read the data file(s) from disk (e.g. after a fresh scrape).""" try: count = store.reload() except FileNotFoundError as exc: raise HTTPException(status_code=500, detail=str(exc)) - return {"reloaded": True, "events_loaded": count} + return { + "reloaded": True, + "events_loaded": count, + "current_events": store.current_count, + "archive_events": store.archive_count, + } @app.get("/events", response_model=EventList) @@ -94,6 +115,38 @@ def list_artists(): return sorted(names, key=str.lower) +@app.get("/events/archive", response_model=EventList) +def list_archive_events( + free: Optional[bool] = Query(None, description="Filter by free admission"), + name: Optional[str] = Query(None, description="Case-insensitive substring match on event name"), + artist: Optional[str] = Query(None, description="Case-insensitive substring match on any artist name"), + date: Optional[str] = Query(None, description="Exact match on the raw event date (dd.mm.yy)"), +): + """Past events only - shorthand for ``/events?upcoming=false``.""" + today = datetime.date.today() + events = [ + e + for e in store.all() + if _matches(e, free, name, artist, date, upcoming=False, today=today) + ] + return EventList(count=len(events), events=events) + + +@app.get("/calendar.ics") +def calendar_ics(): + """iCalendar feed of all known events (upcoming + archived). + + Subscribe from a calendar app with ``webcal:///calendar.ics`` + (or the plain https URL) to get an auto-refreshing calendar. + """ + ics = build_ics(store.all()) + return PlainTextResponse( + content=ics, + media_type="text/calendar", + headers={"Content-Disposition": "inline; filename=fundbureau-events.ics"}, + ) + + @app.get("/events/{index}", response_model=Event) def get_event(index: int): event = store.get(index) diff --git a/api/requirements.txt b/api/requirements.txt index f53a12a..1480ba7 100644 --- a/api/requirements.txt +++ b/api/requirements.txt @@ -1,2 +1,3 @@ fastapi>=0.103 uvicorn[standard]>=0.23 +croniter>=2.0 diff --git a/api/scrape_schedule.py b/api/scrape_schedule.py new file mode 100644 index 0000000..d2e876d --- /dev/null +++ b/api/scrape_schedule.py @@ -0,0 +1,58 @@ +"""Reports when a scrape last ran and when cron will next run it. + +"Last ran" is read from the mtime of the scraper's output file - true +regardless of how/when the scrape was actually invoked. "Next run" requires +telling this module the cron schedule via an env var (there's no reliable, +portable way to introspect another process's crontab from here), given as +a standard 5-field cron expression, e.g. "0 * * * *". +""" + +from datetime import datetime +from pathlib import Path +from typing import Optional + +from croniter import croniter + + +def _now_local() -> datetime: + return datetime.now().astimezone() + + +def _file_mtime_local(path: Path) -> Optional[datetime]: + try: + return datetime.fromtimestamp(path.stat().st_mtime).astimezone() + except FileNotFoundError: + return None + + +def _next_cron_run(cron_expr: str, after: datetime) -> Optional[datetime]: + try: + naive_after = after.replace(tzinfo=None) + return croniter(cron_expr, naive_after).get_next(datetime).astimezone() + except (ValueError, KeyError): + return None + + +def get_schedule_info(path: Path, cron_expr: Optional[str]) -> dict: + """Build the last/next-scrape block for one scraper output file. + + ``cron_expr`` is that scrape's cron expression (e.g. "0 * * * *"); + None/invalid -> next-run fields are null. + """ + now = _now_local() + cron_expr = (cron_expr or "").strip() or None + + last_scrape_at = _file_mtime_local(path) + next_scrape_at = _next_cron_run(cron_expr, now) if cron_expr else None + + return { + "cron_schedule": cron_expr, + "last_scrape_at": last_scrape_at.isoformat() if last_scrape_at else None, + "seconds_since_last_scrape": ( + round((now - last_scrape_at).total_seconds()) if last_scrape_at else None + ), + "next_scrape_at": next_scrape_at.isoformat() if next_scrape_at else None, + "seconds_until_next_scrape": ( + round((next_scrape_at - now).total_seconds()) if next_scrape_at else None + ), + } diff --git a/scraper/scraper.py b/scraper/scraper.py index bd1dc01..dde539b 100644 --- a/scraper/scraper.py +++ b/scraper/scraper.py @@ -8,6 +8,8 @@ load_local_file = False output_file = "fundi-scraped-output.json" output_path = "./" ignore_ticket_link = False +ARCHIVE_URL = "https://fundbureau.de/archiv.html" +ARCHIVE_OUTPUT_FILE = "fundi-archive-output.json" argp = argparse.ArgumentParser() argp.add_argument("--url", type=str, help="URL to scrape (default: https://fundbureau.de/)") @@ -15,8 +17,12 @@ argp.add_argument("--output-file", type=str, help="Output JSON file name (defaul argp.add_argument("--output-path", type=str, help="Output path for the JSON file (default: current directory)") argp.add_argument("--load-local-file", action="store_true", help="Load local HTML file instead of scraping the website (default: False)") argp.add_argument("--ignore-ticket-link", action="store_true", help="Ignore ticket link extraction (will leave it empty in the output JSON)") +argp.add_argument("--archive", action="store_true", help=f"Scrape the past-events archive ({ARCHIVE_URL}) instead of upcoming events. Changes the default url/output-file, both can still be overridden with --url/--output-file.") args = argp.parse_args() +if vars(args)["archive"]: + scrape_url = ARCHIVE_URL + output_file = ARCHIVE_OUTPUT_FILE if vars(args)["url"]: scrape_url = vars(args)["url"] if vars(args)["output_file"]: @@ -42,8 +48,13 @@ def scrape_website(url, load_local_file): page = browser.new_page() page.goto(url, wait_until="networkidle") try: - # Wait until at least one .event div actually renders in the DOM - page.wait_for_selector("#upcoming-events-container .event", timeout=15000) + # Wait until at least one .event div actually renders in the DOM. + # Upcoming events live in #upcoming-events-container, past/archive + # events in #past-events-container - match either. + page.wait_for_selector( + "#upcoming-events-container .event, #past-events-container .event", + timeout=15000, + ) except Exception: print("Warning: no .event elements appeared before timeout.") html = page.content() @@ -86,7 +97,8 @@ def extract_events(soup, ignore_ticket_link): }) - event_starttime = event.find("div", class_="event-starttime").text.strip() + event_starttime_tag = event.find("div", class_="event-starttime") + event_starttime = event_starttime_tag.text.strip() if event_starttime_tag else "" if ignore_ticket_link: event_ticket_link = "" event_free = None