ebbe.bass/fundi-scraper-api
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add archive scraping, webcal feed, and scrape schedule reporting

Browse Source 
- scraper.py: --archive flag to scrape https://fundbureau.de/archiv.html
  (past events), fixing a container-specific wait selector and a crash
  on events with no door time that this surfaced.
- api: EventStore now merges an optional archive JSON file into the
  main event list (deduped by date+name); adds GET /events/archive.
- api: GET /calendar.ics serves an RFC 5545 feed of all events for
  webcal subscriptions, with an all-day fallback when no door time is
  parseable.
- api: GET /health reports last-scrape time (from file mtime) and, via
  new SCRAPE_CRON/ARCHIVE_SCRAPE_CRON env vars, the next scheduled run
  and seconds until it, for both the regular and archive scrape.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
ebbe.bass
2026-09-05 13:42:51 +02:00
co-authored by Claude Sonnet 5
parent e4ffa61988
commit eaed7d4ce6
8 changed files with 436 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
api/README.md
+31 -10
View File
@@ -19,22 +19,43 @@ uvicorn main:app --reload
Interactive docs: http://127.0.0.1:8000/docs
By default the API reads `../fundi-scraped-output.json` (the repo root).
Point it elsewhere with an env var:
By default the API reads `../fundi-scraped-output.json` (upcoming events) and
`../fundi-archive-output.json` (past events, optional — produced by
`scraper.py --archive`), merging both into one in-memory list. Point either
elsewhere with an env var:
```bash
DATA_FILE=/path/to/events.json uvicorn main:app
DATA_FILE=/path/to/events.json ARCHIVE_DATA_FILE=/path/to/archive.json uvicorn main:app
```
### Scrape schedule reporting
`/health` reports when each scraper output file was last written (its mtime)
and, if you tell it the cron schedule, when it's next due. This doesn't read
your crontab — set the same expression(s) you put there as env vars:
```bash
SCRAPE_CRON="0 * * * *" ARCHIVE_SCRAPE_CRON="0 4 * * *" uvicorn main:app
```
`SCRAPE_CRON` covers the plain scrape (`fundi-scraped-output.json`);
`ARCHIVE_SCRAPE_CRON` covers `--archive` (`fundi-archive-output.json`) and
falls back to `SCRAPE_CRON` if unset — set it separately only if the archive
scrape runs on its own cron line. Leave both unset to just get the
last-scrape info with `next_scrape_at` / `seconds_until_next_scrape` as
`null`.
## Endpoints
| Method | Path | Description |
|--------|------------------|-------------|
| GET | `/health` | Status + number of events loaded |
| GET | `/events` | List events, with optional filters (see below) |
| GET | `/events/{index}`| Single event by its position in the file (0-based) |
| GET | `/artists` | Deduplicated, sorted list of all artist names |
| POST | `/reload` | Re-read the JSON file from disk (after a fresh scrape) |
| Method | Path | Description |
|--------|-------------------|-------------|
| GET | `/health` | Status, event counts, and last/next scrape timing (see below) |
| GET | `/events` | List events, with optional filters (see below) |
| GET | `/events/archive` | Past events only — shorthand for `/events?upcoming=false` |
| GET | `/events/{index}` | Single event by its position in the merged list (0-based) |
| GET | `/artists` | Deduplicated, sorted list of all artist names |
| GET | `/calendar.ics` | iCalendar/webcal feed of all events — subscribe with `webcal://<host>/calendar.ics` |
| POST | `/reload` | Re-read the data file(s) from disk (after a fresh scrape) |
### `/events` query parameters