Files

T

Ebbe Baß b2468cc083 changed out example address to example domain.

2026-09-09 13:59:19 +02:00

11 KiB

Raw Blame History

fundi-scraper-api

Simple scraper that turns all upcoming and past events at the Fundbureau (Hamburg) into a REST API. This project came up through the idea of having a way to ask my voice assistant what the upcomming events at the Fundi are.

Two parts:

scraper/ — a Playwright + BeautifulSoup script that scrapes the Fundbureau site and writes the events to a JSON file.
api/ — a FastAPI app that serves that JSON file as a REST API.

fundbureau.de  ──scraper──▶  fundi-scraped-output.json  ──api──▶  REST endpoints

Requirements

Python 3.11+
For the scraper: playwright, beautifulsoup4 (plus playwright install chromium)
For the API: fastapi, uvicorn, croniter (see api/requirements.txt)

pip install beautifulsoup4 playwright
playwright install chromium
pip install -r api/requirements.txt

1. Scraper

Located in scraper/scraper.py. Run it from the repo root:

python scraper/scraper.py

This writes fundi-scraped-output.json to the current directory.

Options

Flag	Description	Default
`--url URL`	URL to scrape	`https://fundbureau.de/`
`--output-file NAME`	Output JSON file name	`fundi-scraped-output.json`
`--output-path PATH`	Directory for the output file	`./`
`--load-local-file`	Parse a local HTML file instead of scraping the live site (pass the file path as `--url`)	off
`--ignore-ticket-link`	Don't extract ticket links; leaves `event_ticket_link` empty and `event_free` null	off
`--archive`	Scrape the archive page (past events) instead of the landing page. Changes the defaults to `--url https://fundbureau.de/archiv.html` and `--output-file fundi-archive-output.json`; both can still be overridden explicitly	off

Archive (past events)

The Fundbureau site keeps every past event on a separate page, archiv.html, using the same .event markup as the landing page. Scrape it with:

python scraper/scraper.py --archive

This writes fundi-archive-output.json, in the same schema as the regular output. Run both commands (regularly, e.g. via a cron job) to keep an up-to-date pair of files — the API merges them (see below).

Output format

The output is a JSON array of event objects:

[
  {
    "event_date": "04.09.26",
    "event_name": "Trance",
    "event_artists": [
      [
        { "artist_name": "Skkin Velvet", "artist_play_time_start": "0",  "artist_play_time_end": "Open end" },
        { "artist_name": "Bizarre",      "artist_play_time_start": "23", "artist_play_time_end": "3" }
      ]
    ],
    "event_starttime": "EINLASS 23:00",
    "event_ticket_link": "https://www.ticketmaster.de/venue/fundbureau-hamburg-tickets/hamfundb/701",
    "event_free": false
  }
]

Field	Type	Notes
`event_date`	string	`dd.mm.yy`
`event_name`	string
`event_artists`	array of arrays of objects	outer array is the line-up grouping; each artist has `artist_name`, `artist_play_time_start`, `artist_play_time_end` (times are hours as strings, `"n/a"`, or `"Open end"`)
`event_starttime`	string	raw door-time text, e.g. `"EINLASS 23:00"`
`event_ticket_link`	string	URL, or `"n/a"` for free events, or `""` if unknown
`event_free`	bool \| null	`true` if admission is free, `null` when `--ignore-ticket-link` is used

2. REST API

Located in api/. It loads the scraped JSON file(s) into memory and exposes them over HTTP.

Run

cd api
uvicorn main:app --reload

API root: https://fundi.api.example.com
Interactive docs (Swagger UI): https://fundi.api.example.com/docs
OpenAPI schema: https://fundi.api.example.com/openapi.json

By default the API reads fundi-scraped-output.json (current/upcoming events) and fundi-archive-output.json (past events, from --archive) from the repo root, and merges them into one in-memory list. The archive file is optional — if it doesn't exist yet, the API just serves the current events. Override either path with an environment variable:

DATA_FILE=/path/to/events.json ARCHIVE_DATA_FILE=/path/to/archive.json uvicorn main:app

Endpoints

Method	Path	Description
`GET`	`/health`	Service status, event counts, resolved data file paths, and last/next scrape timing
`GET`	`/events`	List events, with optional filters (below)
`GET`	`/events/archive`	Past events only — shorthand for `/events?upcoming=false`
`GET`	`/events/{index}`	A single event by its 0-based position in the merged list; `404` if out of range
`GET`	`/artists`	Deduplicated, case-insensitively sorted list of all artist names
`GET`	`/calendar.ics`	iCalendar/webcal feed of all events — subscribe from any calendar app
`POST`	`/reload`	Re-read the data file(s) from disk (call after re-running the scraper); rate-limited (below); `500` if the main file is missing

`/events` query parameters

All filters are optional and combine with AND. /events/archive accepts the same filters except upcoming (it's always false).

Param	Type	Meaning
`free`	bool	`true` = only free events, `false` = only paid events
`name`	string	Case-insensitive substring match on `event_name`
`artist`	string	Case-insensitive substring match on any artist name in the line-up
`date`	string	Exact match on the raw `event_date` string (`dd.mm.yy`)
`upcoming`	bool	`true` = event date is today or later, `false` = past events (events with an unparseable date are excluded)

Response shapes

GET /events and GET /events/archive return:

{
  "count": 2,
  "events": [ { "event_date": "…", "event_name": "…", "...": "…" } ]
}

GET /events/{index} returns a single event object (same schema as the scraper output). GET /artists returns a plain JSON array of strings.

Scrape schedule reporting

The scraper is meant to be run periodically via cron on the box hosting the API. GET /health reports, for both the regular and archive scrape:

last_scrape_at / seconds_since_last_scrape — from the output file's mtime, so this works no matter how the scrape was triggered.
next_scrape_at / seconds_until_next_scrape — computed from a cron expression, sourced in order:
1. SCRAPE_CRON / ARCHIVE_SCRAPE_CRON env vars, if set — always wins, and the only option that works when the API doesn't run as the same user/host as the cron job.
2. Otherwise, the API's own OS user's crontab (crontab -l), looked up for a line invoking scraper.py (with vs. without --archive picks archive vs. plain; an @reboot line for the same script is skipped since it isn't a recurring schedule). Only works when the API process runs as the same user whose personal crontab holds the scrape job — not a system crontab/cron.d entry, not a job under a different user or host.

SCRAPE_CRON="0 * * * *" ARCHIVE_SCRAPE_CRON="0 4 * * *" uvicorn main:app

ARCHIVE_SCRAPE_CRON falls back to SCRAPE_CRON (env or crontab-discovered) if unset (handy if both scrapes run off the same cron line). If neither an env var nor a matching crontab line is found, you just get the last-scrape info with next_scrape_at / seconds_until_next_scrape as null.

"scrape": {
  "cron_schedule": "0 * * * *",
  "last_scrape_at": "2026-09-05T13:00:00+02:00",
  "seconds_since_last_scrape": 42,
  "next_scrape_at": "2026-09-05T14:00:00+02:00",
  "seconds_until_next_scrape": 3558
}

`/reload` rate limiting

POST /reload is limited to one call per RELOAD_MIN_INTERVAL_SECONDS (default 10) — a call within that window returns 429 with a Retry-After header instead of re-reading the file(s). It's a single shared cooldown, not per-caller, so it also protects the server if several callers hit it at once.

RELOAD_MIN_INTERVAL_SECONDS=30 uvicorn main:app

Calendar / webcal feed

GET /calendar.ics renders every loaded event (upcoming + archived) as an RFC 5545 VCALENDAR. Point a calendar app at it to get an auto-refreshing subscription, using the webcal:// scheme so the app treats it as a subscription instead of a one-off download:

webcal://127.0.0.1:8000/calendar.ics

(swap in your deployed host; use https:// instead of webcal:// for tools that don't understand the webcal: scheme, such as curl). Events without a parseable door time (event_starttime) are rendered as all-day entries instead of a guessed time.

Examples

# All events
curl https://fundi.api.example.com/events

# Only free events
curl 'https://fundi.api.example.com/events?free=true'

# Upcoming events featuring an artist whose name contains "randali"
curl 'https://fundi.api.example.com/events?artist=randali&upcoming=true'

# Everything on a given night
curl 'https://fundi.api.example.com/events?date=05.09.26'

# Past events (archive)
curl https://fundi.api.example.com/events/archive

# First event in the merged list
curl https://fundi.api.example.com/events/0

# All known artists
curl https://fundi.api.example.com/artists

# Calendar feed
curl https://fundi.api.example.com/calendar.ics

# Refresh after re-scraping
python scraper/scraper.py && python scraper/scraper.py --archive
curl -X POST https://fundi.api.example.com/reload

Typical workflow

# 1. Scrape upcoming events, and (occasionally) the archive
python scraper/scraper.py
python scraper/scraper.py --archive

# 2. Serve
cd api && uvicorn main:app --reload

# 3. (later) re-scrape and hot-reload the API without restarting it
python scraper/scraper.py
curl -X POST https://fundi.api.example.com/reload

Project layout

.
├── scraper/
│   └── scraper.py            # scrapes fundbureau.de (or --archive: archiv.html) -> JSON
├── api/
│   ├── main.py                # FastAPI app + routes
│   ├── datasource.py           # loads/merges/reloads the JSON files, date & artist helpers
│   ├── calendar_feed.py         # builds the /calendar.ics webcal feed
│   ├── scrape_schedule.py       # last/next-scrape info for /health (env var or crontab)
│   ├── rate_limit.py            # shared cooldown gate used by /reload
│   ├── models.py               # Pydantic models for the event schema
│   ├── requirements.txt
│   └── README.md
├── fundi-scraped-output.json  # scraper output: upcoming events / default API datasource
└── fundi-archive-output.json  # scraper --archive output: past events / archive API datasource

11 KiB Raw Blame History