fundi-scraper-api/README.md

# fundi-scraper-api

Simple scraper that turns all upcoming and past events at the
[Fundbureau](https://fundbureau.de/) (Hamburg) into a REST API.
This project came up through the idea of having a way to ask my voice assistant what the upcomming events at the Fundi are.

Two parts:

1. **`scraper/`** — a Playwright + BeautifulSoup script that scrapes the
   Fundbureau site and writes the events to a JSON file.
2. **`api/`** — a FastAPI app that serves that JSON file as a REST API.

```
fundbureau.de  ──scraper──▶  fundi-scraped-output.json  ──api──▶  REST endpoints
```

## Requirements

- Python 3.11+
- For the scraper: `playwright`, `beautifulsoup4` (plus `playwright install chromium`)
- For the API: `fastapi`, `uvicorn`, `croniter` (see [`api/requirements.txt`](api/requirements.txt))

```bash
pip install beautifulsoup4 playwright
playwright install chromium
pip install -r api/requirements.txt
```

## 1. Scraper

Located in [`scraper/scraper.py`](scraper/scraper.py). Run it from the repo root:

```bash
python scraper/scraper.py
```

This writes `fundi-scraped-output.json` to the current directory.

### Options

| Flag | Description | Default |
|------|-------------|---------|
| `--url URL` | URL to scrape | `https://fundbureau.de/` |
| `--output-file NAME` | Output JSON file name | `fundi-scraped-output.json` |
| `--output-path PATH` | Directory for the output file | `./` |
| `--load-local-file` | Parse a local HTML file instead of scraping the live site (pass the file path as `--url`) | off |
| `--ignore-ticket-link` | Don't extract ticket links; leaves `event_ticket_link` empty and `event_free` null | off |
| `--archive` | Scrape the [archive page](https://fundbureau.de/archiv.html) (past events) instead of the landing page. Changes the defaults to `--url https://fundbureau.de/archiv.html` and `--output-file fundi-archive-output.json`; both can still be overridden explicitly | off |

### Archive (past events)

The Fundbureau site keeps every past event on a separate page,
[archiv.html](https://fundbureau.de/archiv.html), using the same `.event`
markup as the landing page. Scrape it with:

```bash
python scraper/scraper.py --archive
```

This writes `fundi-archive-output.json`, in the same schema as the regular
output. Run both commands (regularly, e.g. via a cron job) to keep an
up-to-date pair of files — the API merges them (see below).

### Output format

The output is a JSON array of event objects:

```json
[
  {
    "event_date": "04.09.26",
    "event_name": "Trance",
    "event_artists": [
      [
        { "artist_name": "Skkin Velvet", "artist_play_time_start": "0",  "artist_play_time_end": "Open end" },
        { "artist_name": "Bizarre",      "artist_play_time_start": "23", "artist_play_time_end": "3" }
      ]
    ],
    "event_starttime": "EINLASS 23:00",
    "event_ticket_link": "https://www.ticketmaster.de/venue/fundbureau-hamburg-tickets/hamfundb/701",
    "event_free": false
  }
]
```

| Field | Type | Notes |
|-------|------|-------|
| `event_date` | string | `dd.mm.yy` |
| `event_name` | string | |
| `event_artists` | array of arrays of objects | outer array is the line-up grouping; each artist has `artist_name`, `artist_play_time_start`, `artist_play_time_end` (times are hours as strings, `"n/a"`, or `"Open end"`) |
| `event_starttime` | string | raw door-time text, e.g. `"EINLASS 23:00"` |
| `event_ticket_link` | string | URL, or `"n/a"` for free events, or `""` if unknown |
| `event_free` | bool \| null | `true` if admission is free, `null` when `--ignore-ticket-link` is used |

## 2. REST API

Located in [`api/`](api/). It loads the scraped JSON file(s) into memory and
exposes them over HTTP.

### Run

```bash
cd api
uvicorn main:app --reload
```

- API root: https://fundi.api.example.com
- Interactive docs (Swagger UI): https://fundi.api.example.com/docs
- OpenAPI schema: https://fundi.api.example.com/openapi.json

By default the API reads `fundi-scraped-output.json` (current/upcoming events)
and `fundi-archive-output.json` (past events, from `--archive`) from the repo
root, and merges them into one in-memory list. The archive file is optional —
if it doesn't exist yet, the API just serves the current events. Override
either path with an environment variable:

```bash
DATA_FILE=/path/to/events.json ARCHIVE_DATA_FILE=/path/to/archive.json uvicorn main:app
```

### Endpoints

| Method | Path | Description |
|--------|------|-------------|
| `GET`  | `/health` | Service status, event counts, resolved data file paths, and last/next scrape timing |
| `GET`  | `/events` | List events, with optional filters (below) |
| `GET`  | `/events/archive` | Past events only — shorthand for `/events?upcoming=false` |
| `GET`  | `/events/{index}` | A single event by its 0-based position in the merged list; `404` if out of range |
| `GET`  | `/artists` | Deduplicated, case-insensitively sorted list of all artists, each with every date they appear on |
| `GET`  | `/calendar.ics` | iCalendar/webcal feed of all events — subscribe from any calendar app |
| `POST` | `/reload` | Re-read the data file(s) from disk (call after re-running the scraper); rate-limited (below); `500` if the main file is missing |

### `/events` query parameters

All filters are optional and combine with AND. `/events/archive` accepts the
same filters except `upcoming` (it's always `false`).

| Param | Type | Meaning |
|-------|------|---------|
| `free` | bool | `true` = only free events, `false` = only paid events |
| `name` | string | Case-insensitive substring match on `event_name` |
| `artist` | string | Case-insensitive substring match on any artist name in the line-up |
| `date` | string | Exact match on the raw `event_date` string (`dd.mm.yy`) |
| `upcoming` | bool | `true` = event date is today or later, `false` = past events (events with an unparseable date are excluded) |
| `sort` | string | `asc`/`desc` = chronological by `event_date` (events with an unparseable date always sort last, regardless of direction); `az`/`za` = alphabetical by `event_name` (case-insensitive) |
| `count` | int ≥ 1 | Limit the number of results returned, applied after filtering and sorting |

### `/artists` query parameters

| Param | Type | Meaning |
|-------|------|---------|
| `sort` | string | `az`/`za` = alphabetical by artist name (case-insensitive, default `az`); `asc`/`desc` = chronological by each artist's *latest* date (artists with no parseable date always sort last, regardless of direction) |
| `count` | int ≥ 1 | Limit the number of artists returned |

An artist's own `dates` list is always chronological (earliest first) —
`sort` only controls the order artists appear in, not the order of dates
within one artist.

### Response shapes

`GET /events` and `GET /events/archive` return:

```json
{
  "count": 2,
  "events": [ { "event_date": "…", "event_name": "…", "...": "…" } ]
}
```

`GET /events/{index}` returns a single event object (same schema as the scraper
output). `GET /artists` returns each artist once with every date (past or
upcoming) they're in the line-up for, sorted chronologically:

```json
[
  { "artist_name": "Bizarre", "dates": ["05.09.26", "12.10.26"] },
  { "artist_name": "Skkin Velvet", "dates": ["04.09.26"] }
]
```

### Scrape schedule reporting

The scraper is meant to be run periodically via cron on the box hosting the
API. `GET /health` reports, for both the regular and archive scrape:

- `last_scrape_at` / `seconds_since_last_scrape` — from the output file's
  mtime, so this works no matter how the scrape was triggered.
- `next_scrape_at` / `seconds_until_next_scrape` — computed from a cron
  expression, sourced in order:
  1. `SCRAPE_CRON` / `ARCHIVE_SCRAPE_CRON` env vars, if set — always wins,
     and the only option that works when the API doesn't run as the same
     user/host as the cron job.
  2. Otherwise, the API's own OS user's crontab (`crontab -l`), looked up
     for a line invoking `scraper.py` (with vs. without `--archive` picks
     archive vs. plain; an `@reboot` line for the same script is skipped
     since it isn't a recurring schedule). Only works when the API process
     runs as the same user whose personal crontab holds the scrape job —
     not a system crontab/cron.d entry, not a job under a different user
     or host.

```bash
SCRAPE_CRON="0 * * * *" ARCHIVE_SCRAPE_CRON="0 4 * * *" uvicorn main:app
```

`ARCHIVE_SCRAPE_CRON` falls back to `SCRAPE_CRON` (env or crontab-discovered)
if unset (handy if both scrapes run off the same cron line). If neither an
env var nor a matching crontab line is found, you just get the last-scrape
info with `next_scrape_at` / `seconds_until_next_scrape` as `null`.

```json
"scrape": {
  "cron_schedule": "0 * * * *",
  "last_scrape_at": "2026-09-05T13:00:00+02:00",
  "seconds_since_last_scrape": 42,
  "next_scrape_at": "2026-09-05T14:00:00+02:00",
  "seconds_until_next_scrape": 3558
}
```

### `/reload` rate limiting

`POST /reload` is limited to one call per `RELOAD_MIN_INTERVAL_SECONDS`
(default 10) — a call within that window returns `429` with a `Retry-After`
header instead of re-reading the file(s). It's a single shared cooldown, not
per-caller, so it also protects the server if several callers hit it at once.

```bash
RELOAD_MIN_INTERVAL_SECONDS=30 uvicorn main:app
```

### Calendar / webcal feed

`GET /calendar.ics` renders every loaded event (upcoming + archived) as an
RFC 5545 `VCALENDAR`. Point a calendar app at it to get an
auto-refreshing subscription, using the `webcal://` scheme so the app treats
it as a subscription instead of a one-off download:

```
webcal://127.0.0.1:8000/calendar.ics
```

(swap in your deployed host; use `https://` instead of `webcal://` for tools
that don't understand the `webcal:` scheme, such as `curl`). Events without a
parseable door time (`event_starttime`) are rendered as all-day entries
instead of a guessed time.

### Examples

```bash
# All events
curl https://fundi.api.example.com/events

# Only free events
curl 'https://fundi.api.example.com/events?free=true'

# Upcoming events featuring an artist whose name contains "randali"
curl 'https://fundi.api.example.com/events?artist=randali&upcoming=true'

# Everything on a given night
curl 'https://fundi.api.example.com/events?date=05.09.26'

# Soonest events first
curl 'https://fundi.api.example.com/events?sort=asc'

# Next 5 upcoming events
curl 'https://fundi.api.example.com/events?upcoming=true&sort=asc&count=5'

# Past events (archive)
curl https://fundi.api.example.com/events/archive

# First event in the merged list
curl https://fundi.api.example.com/events/0

# All known artists
curl https://fundi.api.example.com/artists

# 10 most recently active artists
curl 'https://fundi.api.example.com/artists?sort=desc&count=10'

# Calendar feed
curl https://fundi.api.example.com/calendar.ics

# Refresh after re-scraping
python scraper/scraper.py && python scraper/scraper.py --archive
curl -X POST https://fundi.api.example.com/reload
```

## Typical workflow

```bash
# 1. Scrape upcoming events, and (occasionally) the archive
python scraper/scraper.py
python scraper/scraper.py --archive

# 2. Serve
cd api && uvicorn main:app --reload

# 3. (later) re-scrape and hot-reload the API without restarting it
python scraper/scraper.py
curl -X POST https://fundi.api.example.com/reload
```

## Project layout

```
.
├── scraper/
│   └── scraper.py            # scrapes fundbureau.de (or --archive: archiv.html) -> JSON
├── api/
│   ├── main.py                # FastAPI app + routes
│   ├── datasource.py           # loads/merges/reloads the JSON files, date & artist helpers
│   ├── calendar_feed.py         # builds the /calendar.ics webcal feed
│   ├── scrape_schedule.py       # last/next-scrape info for /health (env var or crontab)
│   ├── rate_limit.py            # shared cooldown gate used by /reload
│   ├── models.py               # Pydantic models for the event schema
│   ├── requirements.txt
│   └── README.md
├── fundi-scraped-output.json  # scraper output: upcoming events / default API datasource
└── fundi-archive-output.json  # scraper --archive output: past events / archive API datasource
```