ebbe.bass/fundi-scraper-api
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add archive scraping, webcal feed, and scrape schedule reporting

Browse Source 
- scraper.py: --archive flag to scrape https://fundbureau.de/archiv.html
  (past events), fixing a container-specific wait selector and a crash
  on events with no door time that this surfaced.
- api: EventStore now merges an optional archive JSON file into the
  main event list (deduped by date+name); adds GET /events/archive.
- api: GET /calendar.ics serves an RFC 5545 feed of all events for
  webcal subscriptions, with an all-day fallback when no door time is
  parseable.
- api: GET /health reports last-scrape time (from file mtime) and, via
  new SCRAPE_CRON/ARCHIVE_SCRAPE_CRON env vars, the next scheduled run
  and seconds until it, for both the regular and archive scrape.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
ebbe.bass
2026-09-05 13:42:51 +02:00
co-authored by Claude Sonnet 5
parent e4ffa61988
commit eaed7d4ce6
8 changed files with 436 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+95 -19
View File
@@ -18,7 +18,7 @@ fundbureau.de ──scraper──▶ fundi-scraped-output.json ──api─
- Python 3.11+
- For the scraper: `playwright`, `beautifulsoup4` (plus `playwright install chromium`)
- For the API: `fastapi`, `uvicorn` (see [`api/requirements.txt`](api/requirements.txt))
- For the API: `fastapi`, `uvicorn`, `croniter` (see [`api/requirements.txt`](api/requirements.txt))
```bash
pip install beautifulsoup4 playwright
@@ -45,6 +45,21 @@ This writes `fundi-scraped-output.json` to the current directory.
| `--output-path PATH` | Directory for the output file | `./` |
| `--load-local-file` | Parse a local HTML file instead of scraping the live site (pass the file path as `--url`) | off |
| `--ignore-ticket-link` | Don't extract ticket links; leaves `event_ticket_link` empty and `event_free` null | off |
| `--archive` | Scrape the [archive page](https://fundbureau.de/archiv.html) (past events) instead of the landing page. Changes the defaults to `--url https://fundbureau.de/archiv.html` and `--output-file fundi-archive-output.json`; both can still be overridden explicitly | off |
### Archive (past events)
The Fundbureau site keeps every past event on a separate page,
[archiv.html](https://fundbureau.de/archiv.html), using the same `.event`
markup as the landing page. Scrape it with:
```bash
python scraper/scraper.py --archive
```
This writes `fundi-archive-output.json`, in the same schema as the regular
output. Run both commands (regularly, e.g. via a cron job) to keep an
up-to-date pair of files — the API merges them (see below).
### Output format
@@ -79,8 +94,8 @@ The output is a JSON array of event objects:
## 2. REST API
Located in [`api/`](api/). It loads a scraped JSON file into memory and exposes it
over HTTP.
Located in [`api/`](api/). It loads the scraped JSON file(s) into memory and
exposes them over HTTP.
### Run
@@ -93,26 +108,32 @@ uvicorn main:app --reload
- Interactive docs (Swagger UI): http://127.0.0.1:8000/docs
- OpenAPI schema: http://127.0.0.1:8000/openapi.json
By default the API reads `fundi-scraped-output.json` from the repo root. Override
with the `DATA_FILE` environment variable:
By default the API reads `fundi-scraped-output.json` (current/upcoming events)
and `fundi-archive-output.json` (past events, from `--archive`) from the repo
root, and merges them into one in-memory list. The archive file is optional —
if it doesn't exist yet, the API just serves the current events. Override
either path with an environment variable:
```bash
DATA_FILE=/path/to/events.json uvicorn main:app
DATA_FILE=/path/to/events.json ARCHIVE_DATA_FILE=/path/to/archive.json uvicorn main:app
```
### Endpoints
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/health` | Service status, number of events loaded, resolved data file path |
| `GET` | `/health` | Service status, event counts, resolved data file paths, and last/next scrape timing |
| `GET` | `/events` | List events, with optional filters (below) |
| `GET` | `/events/{index}` | A single event by its 0-based position in the file; `404` if out of range |
| `GET` | `/events/archive` | Past events only — shorthand for `/events?upcoming=false` |
| `GET` | `/events/{index}` | A single event by its 0-based position in the merged list; `404` if out of range |
| `GET` | `/artists` | Deduplicated, case-insensitively sorted list of all artist names |
| `POST` | `/reload` | Re-read the JSON file from disk (call after re-running the scraper); `500` if the file is missing |
| `GET` | `/calendar.ics` | iCalendar/webcal feed of all events — subscribe from any calendar app |
| `POST` | `/reload` | Re-read the data file(s) from disk (call after re-running the scraper); `500` if the main file is missing |
### `/events` query parameters
All filters are optional and combine with AND.
All filters are optional and combine with AND. `/events/archive` accepts the
same filters except `upcoming` (it's always `false`).
| Param | Type | Meaning |
|-------|------|---------|
@@ -124,7 +145,7 @@ All filters are optional and combine with AND.
### Response shapes
`GET /events` returns:
`GET /events` and `GET /events/archive` return:
```json
{
@@ -136,6 +157,51 @@ All filters are optional and combine with AND.
`GET /events/{index}` returns a single event object (same schema as the scraper
output). `GET /artists` returns a plain JSON array of strings.
### Scrape schedule reporting
The scraper is meant to be run periodically via cron on the box hosting the
API. `GET /health` reports, for both the regular and archive scrape:
- `last_scrape_at` / `seconds_since_last_scrape` — from the output file's
mtime, so this works no matter how the scrape was triggered.
- `next_scrape_at` / `seconds_until_next_scrape` — computed from a cron
expression *you provide*, since the API can't reliably read another
process's crontab. Set it to match what's actually in cron:
```bash
SCRAPE_CRON="0 * * * *" ARCHIVE_SCRAPE_CRON="0 4 * * *" uvicorn main:app
```
`ARCHIVE_SCRAPE_CRON` falls back to `SCRAPE_CRON` if unset (handy if both
scrapes run off the same cron line). Leaving both unset just omits the
`next_scrape_at` fields (`null`) — `last_scrape_at` still works.
```json
"scrape": {
"cron_schedule": "0 * * * *",
"last_scrape_at": "2026-09-05T13:00:00+02:00",
"seconds_since_last_scrape": 42,
"next_scrape_at": "2026-09-05T14:00:00+02:00",
"seconds_until_next_scrape": 3558
}
```
### Calendar / webcal feed
`GET /calendar.ics` renders every loaded event (upcoming + archived) as an
RFC 5545 `VCALENDAR`. Point a calendar app at it to get an
auto-refreshing subscription, using the `webcal://` scheme so the app treats
it as a subscription instead of a one-off download:
```
webcal://127.0.0.1:8000/calendar.ics
```
(swap in your deployed host; use `https://` instead of `webcal://` for tools
that don't understand the `webcal:` scheme, such as `curl`). Events without a
parseable door time (`event_starttime`) are rendered as all-day entries
instead of a guessed time.
### Examples
```bash
@@ -151,21 +217,29 @@ curl 'http://127.0.0.1:8000/events?artist=randali&upcoming=true'
# Everything on a given night
curl 'http://127.0.0.1:8000/events?date=05.09.26'
# First event in the file
# Past events (archive)
curl http://127.0.0.1:8000/events/archive
# First event in the merged list
curl http://127.0.0.1:8000/events/0
# All known artists
curl http://127.0.0.1:8000/artists
# Calendar feed
curl http://127.0.0.1:8000/calendar.ics
# Refresh after re-scraping
python scraper/scraper.py && curl -X POST http://127.0.0.1:8000/reload
python scraper/scraper.py && python scraper/scraper.py --archive
curl -X POST http://127.0.0.1:8000/reload
```
## Typical workflow
```bash
# 1. Scrape
# 1. Scrape upcoming events, and (occasionally) the archive
python scraper/scraper.py
python scraper/scraper.py --archive
# 2. Serve
cd api && uvicorn main:app --reload
@@ -180,12 +254,14 @@ curl -X POST http://127.0.0.1:8000/reload
```
.
├── scraper/
│ └── scraper.py # scrapes fundbureau.de -> JSON
│ └── scraper.py # scrapes fundbureau.de (or --archive: archiv.html) -> JSON
├── api/
│ ├── main.py # FastAPI app + routes
│ ├── datasource.py # loads/reloads the JSON, date & artist helpers
│ ├── models.py # Pydantic models for the event schema
│ ├── main.py # FastAPI app + routes
│ ├── datasource.py # loads/merges/reloads the JSON files, date & artist helpers
│ ├── calendar_feed.py # builds the /calendar.ics webcal feed
│ ├── models.py # Pydantic models for the event schema
│ ├── requirements.txt
│ └── README.md
└── fundi-scraped-output.json # example scraper output / default API datasource
├── fundi-scraped-output.json # scraper output: upcoming events / default API datasource
└── fundi-archive-output.json # scraper --archive output: past events / archive API datasource
```