fundi-scraper-api/README.md

# fundi-scraper-api

Simple scraper that turns all upcoming and past events at the
[Fundbureau](https://fundbureau.de/) (Hamburg) into a REST API.
This project came up through the idea of having a way to ask my voice assistant what the upcomming events at the Fundi are.

Two parts:

1. **`scraper/`** — a Playwright + BeautifulSoup script that scrapes the
   Fundbureau site and writes the events to a JSON file.
2. **`api/`** — a FastAPI app that serves that JSON file as a REST API.

```
fundbureau.de  ──scraper──▶  fundi-scraped-output.json  ──api──▶  REST endpoints
```

## Requirements

- Python 3.11+
- For the scraper: `playwright`, `beautifulsoup4` (plus `playwright install chromium`)
- For the API: `fastapi`, `uvicorn`, `croniter` (see [`api/requirements.txt`](api/requirements.txt))

```bash
pip install beautifulsoup4 playwright
playwright install chromium
pip install -r api/requirements.txt
```

## 1. Scraper

Located in [`scraper/scraper.py`](scraper/scraper.py). Run it from the repo root:

```bash
python scraper/scraper.py
```

This writes `fundi-scraped-output.json` to the current directory.

### Options

| Flag | Description | Default |
|------|-------------|---------|
| `--url URL` | URL to scrape | `https://fundbureau.de/` |
| `--output-file NAME` | Output JSON file name | `fundi-scraped-output.json` |
| `--output-path PATH` | Directory for the output file | `./` |
| `--load-local-file` | Parse a local HTML file instead of scraping the live site (pass the file path as `--url`) | off |
| `--ignore-ticket-link` | Don't extract ticket links; leaves `event_ticket_link` empty and `event_free` null | off |
| `--archive` | Scrape the [archive page](https://fundbureau.de/archiv.html) (past events) instead of the landing page. Changes the defaults to `--url https://fundbureau.de/archiv.html` and `--output-file fundi-archive-output.json`; both can still be overridden explicitly | off |

### Archive (past events)

The Fundbureau site keeps every past event on a separate page,
[archiv.html](https://fundbureau.de/archiv.html), using the same `.event`
markup as the landing page. Scrape it with:

```bash
python scraper/scraper.py --archive
```

This writes `fundi-archive-output.json`, in the same schema as the regular
output. Run both commands (regularly, e.g. via a cron job) to keep an
up-to-date pair of files — the API merges them (see below).

### Output format

The output is a JSON array of event objects:

```json
[
  {
    "event_date": "04.09.26",
    "event_name": "Trance",
    "event_artists": [
      [
        { "artist_name": "Skkin Velvet", "artist_play_time_start": "0",  "artist_play_time_end": "Open end" },
        { "artist_name": "Bizarre",      "artist_play_time_start": "23", "artist_play_time_end": "3" }
      ]
    ],
    "event_starttime": "EINLASS 23:00",
    "event_ticket_link": "https://www.ticketmaster.de/venue/fundbureau-hamburg-tickets/hamfundb/701",
    "event_free": false
  }
]
```

| Field | Type | Notes |
|-------|------|-------|
| `event_date` | string | `dd.mm.yy` |
| `event_name` | string | |
| `event_artists` | array of arrays of objects | outer array is the line-up grouping; each artist has `artist_name`, `artist_play_time_start`, `artist_play_time_end` (times are hours as strings, `"n/a"`, or `"Open end"`) |
| `event_starttime` | string | raw door-time text, e.g. `"EINLASS 23:00"` |
| `event_ticket_link` | string | URL, or `"n/a"` for free events, or `""` if unknown |
| `event_free` | bool \| null | `true` if admission is free, `null` when `--ignore-ticket-link` is used |

## 2. REST API

Located in [`api/`](api/). It loads the scraped JSON file(s) into memory and
exposes them over HTTP.

### Run

```bash
cd api
uvicorn main:app --reload
```

- API root: http://127.0.0.1:8000
- Interactive docs (Swagger UI): http://127.0.0.1:8000/docs
- OpenAPI schema: http://127.0.0.1:8000/openapi.json

By default the API reads `fundi-scraped-output.json` (current/upcoming events)
and `fundi-archive-output.json` (past events, from `--archive`) from the repo
root, and merges them into one in-memory list. The archive file is optional —
if it doesn't exist yet, the API just serves the current events. Override
either path with an environment variable:

```bash
DATA_FILE=/path/to/events.json ARCHIVE_DATA_FILE=/path/to/archive.json uvicorn main:app
```

### Endpoints

| Method | Path | Description |
|--------|------|-------------|
| `GET`  | `/health` | Service status, event counts, resolved data file paths, and last/next scrape timing |
| `GET`  | `/events` | List events, with optional filters (below) |
| `GET`  | `/events/archive` | Past events only — shorthand for `/events?upcoming=false` |
| `GET`  | `/events/{index}` | A single event by its 0-based position in the merged list; `404` if out of range |
| `GET`  | `/artists` | Deduplicated, case-insensitively sorted list of all artist names |
| `GET`  | `/calendar.ics` | iCalendar/webcal feed of all events — subscribe from any calendar app |
| `POST` | `/reload` | Re-read the data file(s) from disk (call after re-running the scraper); `500` if the main file is missing |

### `/events` query parameters

All filters are optional and combine with AND. `/events/archive` accepts the
same filters except `upcoming` (it's always `false`).

| Param | Type | Meaning |
|-------|------|---------|
| `free` | bool | `true` = only free events, `false` = only paid events |
| `name` | string | Case-insensitive substring match on `event_name` |
| `artist` | string | Case-insensitive substring match on any artist name in the line-up |
| `date` | string | Exact match on the raw `event_date` string (`dd.mm.yy`) |
| `upcoming` | bool | `true` = event date is today or later, `false` = past events (events with an unparseable date are excluded) |

### Response shapes

`GET /events` and `GET /events/archive` return:

```json
{
  "count": 2,
  "events": [ { "event_date": "…", "event_name": "…", "...": "…" } ]
}
```

`GET /events/{index}` returns a single event object (same schema as the scraper
output). `GET /artists` returns a plain JSON array of strings.

### Scrape schedule reporting

The scraper is meant to be run periodically via cron on the box hosting the
API. `GET /health` reports, for both the regular and archive scrape:

- `last_scrape_at` / `seconds_since_last_scrape` — from the output file's
  mtime, so this works no matter how the scrape was triggered.
- `next_scrape_at` / `seconds_until_next_scrape` — computed from a cron
  expression *you provide*, since the API can't reliably read another
  process's crontab. Set it to match what's actually in cron:

```bash
SCRAPE_CRON="0 * * * *" ARCHIVE_SCRAPE_CRON="0 4 * * *" uvicorn main:app
```

`ARCHIVE_SCRAPE_CRON` falls back to `SCRAPE_CRON` if unset (handy if both
scrapes run off the same cron line). Leaving both unset just omits the
`next_scrape_at` fields (`null`) — `last_scrape_at` still works.

```json
"scrape": {
  "cron_schedule": "0 * * * *",
  "last_scrape_at": "2026-09-05T13:00:00+02:00",
  "seconds_since_last_scrape": 42,
  "next_scrape_at": "2026-09-05T14:00:00+02:00",
  "seconds_until_next_scrape": 3558
}
```

### Calendar / webcal feed

`GET /calendar.ics` renders every loaded event (upcoming + archived) as an
RFC 5545 `VCALENDAR`. Point a calendar app at it to get an
auto-refreshing subscription, using the `webcal://` scheme so the app treats
it as a subscription instead of a one-off download:

```
webcal://127.0.0.1:8000/calendar.ics
```

(swap in your deployed host; use `https://` instead of `webcal://` for tools
that don't understand the `webcal:` scheme, such as `curl`). Events without a
parseable door time (`event_starttime`) are rendered as all-day entries
instead of a guessed time.

### Examples

```bash
# All events
curl http://127.0.0.1:8000/events

# Only free events
curl 'http://127.0.0.1:8000/events?free=true'

# Upcoming events featuring an artist whose name contains "randali"
curl 'http://127.0.0.1:8000/events?artist=randali&upcoming=true'

# Everything on a given night
curl 'http://127.0.0.1:8000/events?date=05.09.26'

# Past events (archive)
curl http://127.0.0.1:8000/events/archive

# First event in the merged list
curl http://127.0.0.1:8000/events/0

# All known artists
curl http://127.0.0.1:8000/artists

# Calendar feed
curl http://127.0.0.1:8000/calendar.ics

# Refresh after re-scraping
python scraper/scraper.py && python scraper/scraper.py --archive
curl -X POST http://127.0.0.1:8000/reload
```

## Typical workflow

```bash
# 1. Scrape upcoming events, and (occasionally) the archive
python scraper/scraper.py
python scraper/scraper.py --archive

# 2. Serve
cd api && uvicorn main:app --reload

# 3. (later) re-scrape and hot-reload the API without restarting it
python scraper/scraper.py
curl -X POST http://127.0.0.1:8000/reload
```

## Project layout

```
.
├── scraper/
│   └── scraper.py            # scrapes fundbureau.de (or --archive: archiv.html) -> JSON
├── api/
│   ├── main.py                # FastAPI app + routes
│   ├── datasource.py           # loads/merges/reloads the JSON files, date & artist helpers
│   ├── calendar_feed.py         # builds the /calendar.ics webcal feed
│   ├── models.py               # Pydantic models for the event schema
│   ├── requirements.txt
│   └── README.md
├── fundi-scraped-output.json  # scraper output: upcoming events / default API datasource
└── fundi-archive-output.json  # scraper --archive output: past events / archive API datasource
```