Refactor event handling and caching mechanism

- Removed unused dependencies from requirements.txt.
- Updated event data processing in attending.py to include new fields: Category and Tags.
- Replaced CacheStore with direct file handling for event data in attending.py.
- Introduced hf_sync.py for handling Hugging Face dataset interactions.
- Removed cache_store.py and fetcher.py as they are no longer needed.
- Simplified org_colors.py to improve color assignment logic.
- Updated models.py to streamline event data loading and writing.
- Enhanced tests to cover new event structure and functionality.

ARCHITECTURE.md

@@ -1,264 +0,0 @@
-# Architecture / Code Flow (with ASCII maps)
-
-This repo is intentionally small:
-
-- `app.py` = UI + HTTP API (Gradio mounted into FastAPI)
-- `src/fetcher.py` = crawling + LLM extraction + validation + caching
-- `sources.json` = list of org sources to crawl
-- `./data/events.json` (golden format also stored in `.data/events.json`) = cache / golden output format
-
----
-
-## 1) High-level module map
-
-```
-+--------------------+        imports / calls        +---------------------+
-|       app.py       | --------------------------->  |   src/fetcher.py     |
-|                    |                               |                     |
-|  SeminarsWebApp    |                               |  SeminarFetcher      |
-|  - Gradio UI       |                               |  + refresh_all*       |
-|  - /refresh API    |                               |  + models + cache     |
-+--------------------+                               +---------------------+
-          |
-          | reads
-          v
-+--------------------+
-|    sources.json    |
-|  list[OrgSource]   |
-+--------------------+
-
-Cache file (written/read by fetcher):
-
-+--------------------+
-| ./data/events.json  |  (golden snapshot at .data/events.json)
-| meta + results     |
-+--------------------+
-```
-
----
-
-## 2) Runtime flow (what happens when you open the UI)
-
-### 2.1 Gradio initial load
-
-```
-Browser
-  |
-  v
-Gradio page load
-  |
-  v
-app.py: demo.load(load_initial)
-  |
-  v
-SeminarsWebApp._stream_refresh(force=False)  [generator]
-  |
-  v
-src/fetcher.py: refresh_all_stream(... force=False)
-  |
-  +--> if cache usable -> yield logs + cached results -> done
-  |
-  \--> else -> crawl + LLM -> write cache -> done
-```
-
-### 2.2 Manual refresh button
-
-```
-User clicks "Refresh now"
-  |
-  v
-app.py: refresh_btn.click(refresh_click)
-  |
-  v
-SeminarsWebApp._stream_refresh(force=True)
-  |
-  v
-src/fetcher.py: refresh_all_stream(... force=True)
-  |
-  v
-Always crawls + LLM, then writes cache
-```
-
----
-
-## 3) Cache flow (explicit in the app)
-
-The app explicitly checks cache before crawling.
-
-```
-SeminarsWebApp.stream_refresh(force=False)
-  |
-  v
-cache = CacheStore(config=..., ttl_hours=...)
-  |
-  +--> if cache.is_usable(): cache.load() -> UI updates
-  |
-  \--> else: crawl + LLM -> cache.write(results)
-```
-
----
-
-## 4) Detailed fetcher flow (one org)
-
-The fetcher is designed around a *stream* of events:
-
-- log event: `("log", level, message)`
-- result event: `("result", EventResult)`
-
-### 4.1 One-org pipeline
-
-```
-SeminarFetcher.fetch_next_event_for_org_stream(org)
-  |
-  v
-for hop in 1..max_hops:
-  |
-  +--> fetch_html(url)
-  |       - httpx GET
-  |       - if 403: optional curl fallback
-  |
-  +--> llm_extract(...)
-  |       - text_and_links(html)
-  |       - LiteLLM completion(...)
-  |       - safe_json() + normalize_llm_payload()
-  |       - Pydantic validation -> LlmHopResult
-  |
-  +--> validate_events(hop, now)
-  |       - parse_dt(start_time)
-  |       - filter to future events only
-  |       - ensure evidence + http(s) URL
-  |
-  +--> yield ("result", EventResult)
-  |
-  \--> (optional) follow hop.next_url_to_check if provided
-```
-
-### 4.2 Key idea: strictness retry
-
-If the LLM returns something that is not valid JSON or doesn’t validate, the code retries once in “strict” mode.
-
-```
-llm_extract(strict=False)
-  |
-  +--> (fails JSON / schema) => retry
-  v
-llm_extract(strict=True)
-```
-
----
-
-## 5) Data model / JSON shapes
-
-### 5.1 Source input (`sources.json`)
-
-```
-[
-  {
-    "id": "utoronto",
-    "name": "UofT Robotics",
-    "url": "https://...",
-    "tags": ["canada", "university"]
-  },
-  ...
-]
-```
-
-Validated into `OrgSource`.
-
-### 5.2 Per-org output (`EventResult`)
-
-```
-EventResult:
-  org_id, org_name, source_url
-  status: "ok" | "no_upcoming" | "error"
-  events: [LlmEvent, ...]
-  checked_at
-  hops
-  visited_urls
-  error (optional)
-```
-
-### 5.3 Cache file (`./data/events.json`)
-
-```
-{
-  "meta": {
-    "model": "...",
-    "schema_version": 3,
-    "cached_at": "...",
-    "ttl_hours": 12
-  },
-  "results": [ EventResult, ... ]
-}
-```
-
-Cache is considered usable when:
-- file exists
-- file age < ttl
-- `meta.schema_version == 3`
-- `meta.model == current LLM model`
-
----
-
-## 6) Where logs come from
-
-Logs are generated in two layers:
-
-1) Fetcher (per hop / per org)
-
-```
-"{org}: hop i/j — HTTP GET start: ..."
-"{org}: hop i/j — HTTP GET done (...)"
-"{org}: hop i/j — LLM call start (model=...)"
-"{org}: hop i/j — LLM call done (...)"
-"{org}: hop i/j — validating extracted event(s)"
-"{org}: success (...)" OR "no upcoming events" OR "error (...)"
-```
-
-2) App wrapper (per org result summary)
-
-```
-"{org}: ok (k event(s))"
-"{org}: no upcoming events found"
-"{org}: <error message>"
-```
-
----
-
-## 7) Environment variables (practical cheat-sheet)
-
-### App / paths
-
-- `SOURCES_PATH` (default `sources.json`)
-- `DATA_DIR` (default `./data`)
-- `CACHE_TTL_HOURS` (default `12`)
-- `PORT` (default `7860`)
-
-### `/refresh` auth
-
-- `REFRESH_TOKEN` (required to use `/refresh`)
-
-### LLM (LiteLLM)
-
-- `LITELLM_MODEL` (or `GEMINI_MODEL` fallback)
-- `LITELLM_API_KEY` (or `GEMINI_API_KEY` fallback)
-- `LITELLM_API_BASE` (optional)
-
-Optional knobs:
-- `LLM_TEMPERATURE` (default `0`)
-- `LLM_SEED` (optional)
-- `LLM_MIN_INTERVAL_SECONDS` (optional throttling)
-- `NOW_ISO` (optional override of “current time” for deterministic runs)
-
----
-
-## 8) Quick “read order” (if you’re new)
-
-1) `app.py`:
-   - `SeminarsWebApp._stream_refresh()` to see end-to-end UI flow
-   - `build_fastapi()` for `/refresh`
-
-2) `src/fetcher.py`:
-   - `refresh_all_stream()` to see caching vs crawling
-   - `SeminarFetcher.fetch_next_event_for_org_stream()` for the main pipeline
-   - `llm_extract()` + `validate_events()` for correctness guarantees

README.md

@@ -9,58 +9,32 @@ app_file: app.py
 pinned: false
 ---
 
-# Robotic seminars (HF Space)
-
-This Hugging Face Space aggregates the **next upcoming event** per organization using an LLM via **LiteLLM**. It fetches each page as HTML, extracts a compact text context + candidate links, and asks the model to return structured JSON (up to 3 hops).
+# Robotic seminars
 
+This app reads `assets/events.json` and shows the upcoming seminar list.
 ## Files
 
-- [sources.json](sources.json): list of orgs + starting URLs
-- [app.py](app.py): Gradio UI
-- [src/fetcher.py](src/fetcher.py): LiteLLM hop loop, validation, caching
-
-## Environment variables (HF Space “Secrets”)
-
-- `LITELLM_MODEL` (recommended): LiteLLM model string, e.g. `gemini/gemini-2.0-flash`, `openai/gpt-4o-mini`, `anthropic/claude-3-5-sonnet-20241022`
-- `LITELLM_FALLBACK_MODELS` (optional): comma-separated fallback models to try if the primary hits a rate limit
-- `LLM_MIN_INTERVAL_SECONDS` (optional): minimum delay between LLM calls (useful for very low RPM limits)
-- `LITELLM_API_KEY` (optional): explicit API key to pass to LiteLLM
-- `LITELLM_API_BASE` (optional): custom base URL (useful for proxies/self-hosted endpoints)
-
-Backwards-compatible (still accepted):
-- `GEMINI_API_KEY` (optional): used as a fallback for `LITELLM_API_KEY`
-- `GEMINI_MODEL` (optional): used as a fallback for `LITELLM_MODEL`
-
-Provider-specific env vars also work (recommended):
-- `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`, etc.
-
-- `CACHE_TTL_HOURS` (optional): defaults to `12`
+- `app.py`: Gradio UI
+- `assets/events.json`: event data read by the app and updated when an attending count is clicked
+- `assets/sources.json`: source metadata used for tags
+- `assets/prompt.txt`: prompt for your manual event-generation workflow
 
-## Cache
+## HF dataset sync
 
-The app writes its cache to `./data/events.json`.
+The app pulls the latest remote events file before rendering and pushes updates back after an attending click.
 
-- Locally: create `./data` and ensure it’s writable by your user.
-
-If you don't want to pay for persistent storage, you can sync the cache to a dataset repo instead:
-
-- `HF_CACHE_REPO`: dataset repo id like `username/robotic-seminars-cache`
-- `HUGGINGFACE_HUB_TOKEN` (or `HF_TOKEN`): token with write access to the dataset repo
-
-On startup and refresh, the app pulls the cache from the dataset first. If a refresh runs and writes a new cache, it pushes the updated file back to the dataset repo.
+- Dataset repo is fixed to `ar0s/robotic-seminars-cache`
+- `HF_TOKEN` or `HUGGINGFACE_HUB_TOKEN`: token with dataset write access
+- Remote file name is fixed to `events.json`
+- Manual sync commands:
+  `python -m src.hf_sync pull`
+  `python -m src.hf_sync push`
 
 ## Local run
 
 ```bash
-python -m venv .venv
-source .venv/bin/activate
 pip install -r requirements.txt
-
-# Create a .env from the template and set your keys/model:
-cp .env.example .env
-
-# Or export env vars manually if you prefer.
 python app.py
 ```
 
-Open http://localhost:7860
+Open `http://localhost:7860`.

app.py

@@ -1,173 +1,72 @@
 from __future__ import annotations
 
-import json
 import os
-import threading
-import time
-import traceback
-from datetime import datetime, timezone
 from pathlib import Path
 
 import gradio as gr
 from dotenv import load_dotenv
-from fastapi import FastAPI
 
-from src.cache_store import CacheStore
-from src.fetcher import SeminarFetcher
 from src.attending import COLS, inc_attending, results_table
-from src.models import EventResult, LlmConfig, OrgSource, parse_dt_utc
+from src.hf_sync import hf_enabled, pull_events
+from src.models import infer_category, load_payload
 from src.org_colors import org_colors
-from src.ui_log import clear, error, info, lines
-
-# import debugpy
-# print("Waiting for debugger attach...")
-# debugpy.listen(5678)
-# debugpy.wait_for_client()
-# print("Debugger attached.")
 
 load_dotenv(Path(__file__).with_name(".env"))
 
+ASSETS_DIR = Path(os.environ.get("ASSETS_DIR", "assets"))
+EVENTS_PATH = Path(os.environ.get("EVENTS_PATH", ASSETS_DIR / "events.json"))
+
+
+def prepare_results(results: list[dict]) -> list[dict]:
+    for result in results:
+        for event in result["events"]:
+            event.setdefault(
+                "category",
+                infer_category(
+                    title=event["title"],
+                    org_name=result["org_name"],
+                    event_url=event["event_url"] or "",
+                ),
+            )
+            event.setdefault("tags", [])
+    return results
+
 
-def ts() -> str:
-    return datetime.now(timezone.utc).isoformat(timespec="seconds")
-
-def env_strip(name: str, default: str = "") -> str:
-    return (os.environ.get(name, default) or "").strip()
-
-SOURCES_PATH = os.environ.get("SOURCES_PATH", "sources.json")
-TTL_HOURS = float(os.environ.get("CACHE_TTL_HOURS", "12"))
-LLM = LlmConfig(
-    model=env_strip("LITELLM_MODEL", "gemini/gemini-2.0-flash"),
-    api_key=env_strip("LITELLM_API_KEY") or None,
-    api_base=env_strip("LITELLM_API_BASE") or None,
-)
-HF_CACHE_REPO = env_strip("HF_CACHE_REPO")
-HF_TOKEN = env_strip("HF_TOKEN") or None
-REFRESH_INTERVAL_HOURS = float(os.environ.get("REFRESH_INTERVAL_HOURS", "48"))
-
-SOURCES_RAW = json.loads(Path(SOURCES_PATH).read_text(encoding="utf-8"))
-COLORS = org_colors(SOURCES_RAW)
-
-SOURCES_MD = "\n".join(
-    f"- [{s['name']}]({s['url']})"
-    for s in SOURCES_RAW
-)
-
-
-def stream_refresh(force: bool):
-    logs = lines()
-    clear()
-    results: list[dict] = []
-    colors = COLORS
-
-    def emit(status: str):
-        df, row_map = results_table(results, colors)
-        return status, df, "<br>\n".join(logs), results, row_map
-
-    try:
-        info(f"refresh(force={force}, ttl_hours={TTL_HOURS})")
-        info(f"model={LLM.model}")
-        yield emit("Starting refresh…")
-
-        done = 0
-        sources = [OrgSource.model_validate(s) for s in SOURCES_RAW]
-
-        started = ts()
-        cache = CacheStore(config=LLM, ttl_hours=TTL_HOURS, repo_id=HF_CACHE_REPO, hf_token=HF_TOKEN)
-        cache.pull_from_hf()
-        if not force and cache.is_usable():
-            results.extend(json.loads(cache.path.read_text(encoding="utf-8"))["results"])
-            info(f"used cache: {cache.path}")
-            yield emit(f"Refreshed: {ts()} (started {started})")
-            return
-        info("cache miss, crawling")
-
-        old_results = json.loads(cache.path.read_text(encoding="utf-8"))["results"] if cache.path.exists() else []
-        prev_by_org = {r["org_id"]: r["events"] for r in old_results}
-
-        old_by_exact: dict[tuple[str, str, str], int] = {}
-        old_by_dt: dict[tuple[str, str], int] = {}
-        old_by_dt_n: dict[tuple[str, str], int] = {}
-        for r in old_results:
-            for ev in r["events"]:
-                dt = parse_dt_utc(ev["start_time"]).isoformat(timespec="minutes")
-                url = ev["event_url"] or ""
-                old_by_exact[(r["org_id"], dt, url)] = ev["attending_count"]
-                k2 = (r["org_id"], dt)
-                try:
-                    old_by_dt_n[k2] += 1
-                except KeyError:
-                    old_by_dt_n[k2] = 1
-                if k2 not in old_by_dt:
-                    old_by_dt[k2] = ev["attending_count"]
-
-        fetcher = SeminarFetcher(config=LLM, now=None, max_hops=3, max_events=3, previous_events_by_org=prev_by_org)
-        event_results = []
-        for org in sources:
-            try:
-                done += 1
-                r = fetcher.fetch_next_event_for_org(org)
-
-                for ev in r.events:
-                    dt = parse_dt_utc(ev.start_time).isoformat(timespec="minutes")
-                    url = ev.event_url or ""
-                    k1 = (r.org_id, dt, url)
-                    if k1 in old_by_exact:
-                        ev.attending_count = old_by_exact[k1]
-                        continue
-                    k2 = (r.org_id, dt)
-                    if k2 in old_by_dt and old_by_dt_n[k2] == 1:
-                        ev.attending_count = old_by_dt[k2]
-                event_results.append(r)
-                results.append(r.model_dump())
-                yield emit(f"Refreshed {done}/{len(sources)} (started {started})")
-            except Exception:
-                error(f"Inspecting {org.name}:\n{traceback.format_exc()}")
-                continue
-
-        cache.write(results=event_results)
-        info(f"wrote cache: {cache.path}")
-        yield emit(f"Refreshed: {ts()} (started {started})")
-    except Exception:
-        error(f"Unhandled exception:\n{traceback.format_exc()}")
-        yield emit("Refresh failed.")
-        return
+def render():
+    if hf_enabled():
+        pull_events(EVENTS_PATH)
+    payload = load_payload(EVENTS_PATH)
+    results = prepare_results(payload["results"])
+    colors = org_colors(result["org_name"] for result in results)
+    table, row_map = results_table(results, colors)
+    sources = "\n".join(f"- [{result['org_name']}]({result['source_url']})" for result in results)
+    status = f"Loaded {len(results)} organizations from `{EVENTS_PATH.name}`.\n\nUpdated: {payload['meta']['cached_at']}"
+    return status, table, results, row_map, sources
+
+
+def on_select(results: list[dict], row_map: list[tuple[int, int]], evt: gr.SelectData):
+    colors = org_colors(result["org_name"] for result in results)
+    return inc_attending(evt, results, row_map, colors=colors, events_path=EVENTS_PATH)
 
 
 with gr.Blocks(
     css=".gradio-container table thead th,.gradio-container table thead th *{white-space:nowrap;word-break:keep-all;}"
 ) as demo:
-    gr.Markdown("# Robotic seminars\nNext upcoming event per org.")
-    status = gr.Markdown("")
+    gr.Markdown("# Robotic seminars\nUpcoming events loaded directly from `assets/events.json`.")
+    status = gr.Markdown()
     table = gr.Dataframe(
         headers=COLS,
-        datatype=["str", "markdown", "str", "markdown", "str"],
-        column_widths=[140, 360, 200, 200, 120],
+        datatype=["str", "str", "markdown", "str", "str", "str", "markdown", "str"],
+        column_widths=[140, 140, 360, 200, 140, 160, 200, 120],
         interactive=True,
         wrap=True,
     )
     results_state = gr.State([])
     row_map_state = gr.State([])
-    refresh_btn = gr.Button("Refresh now")
     with gr.Accordion("Sources", open=False):
-        gr.Markdown(SOURCES_MD)
-    with gr.Accordion("Logs", open=False):
-        logs_box = gr.Markdown()
-
-    def on_select(results: list[dict], row_map: list[tuple[int, int]], evt: gr.SelectData):
-        return inc_attending(
-            evt,
-            results,
-            row_map,
-            colors=COLORS,
-            llm=LLM,
-            ttl_hours=TTL_HOURS,
-            cache_repo_id=HF_CACHE_REPO,
-            cache_hf_token=HF_TOKEN,
-        )
-
-    demo.load(stream_refresh, inputs=[gr.State(False)], outputs=[status, table, logs_box, results_state, row_map_state])
-    refresh_btn.click(stream_refresh, inputs=[gr.State(True)], outputs=[status, table, logs_box, results_state, row_map_state])
+        sources_box = gr.Markdown()
+
+    demo.load(render, outputs=[status, table, results_state, row_map_state, sources_box])
     table.select(
         on_select,
         inputs=[results_state, row_map_state],
@@ -176,16 +75,4 @@ with gr.Blocks(
 
 
 if __name__ == "__main__":
-    # import uvicorn
-    # HF Spaces runs Gradio apps itself; avoid mounting + running Uvicorn here.
-    # app = gr.mount_gradio_app(FastAPI(), demo, path="/")
-    # uvicorn.run(app, host="0.0.0.0", port=int(os.environ.get("PORT", "7860")))
-
-    def _periodic_refresh():
-        while True:
-            for _ in stream_refresh(True):
-                pass
-            time.sleep(REFRESH_INTERVAL_HOURS * 3600)
-
-    threading.Thread(target=_periodic_refresh, daemon=True).start()
     demo.launch(server_name="0.0.0.0", server_port=int(os.environ.get("PORT", "7860")))

assets/events.json

@@ -0,0 +1,9 @@
+{
+  "meta": {
+    "schema_version": 5,
+    "cached_at": "2026-04-12T22:28:10+00:00",
+    "ttl_hours": 12.0
+  },
+  "results": [
+  ]
+}

assets/prompt.txt

@@ -0,0 +1,87 @@
+Inspect the current working tree before editing anything. Treat the current repo files as the source of truth for behavior and schema, especially `assets/sources.json`, `assets/events.json`, `app.py`, `src/models.py`, `src/attending.py`, `src/hf_sync.py`, and `tests/test_golden.py`. If `README.md` or any older docs disagree with the code, trust the code.
+
+Your task is to crawl the live event sources and update only the events payload in `assets/events.json` in this repo. Do not edit docs, tests, source code, `.data`, or any other file unless I explicitly ask.
+
+Before you crawl, use `src/hf_sync.py` to fetch the latest remote events payload by calling `pull_events`, so your work starts from the current dataset-backed `assets/events.json`. After you finish creating the new `assets/events.json`, push it back through `push_events`.
+
+Execution requirements:
+1. Inspect the current contents of `assets/sources.json` and `assets/events.json` first.
+2. Call `src/hf_sync.py` `pull_events` before editing `assets/events.json`.
+3. For each source in `assets/sources.json`, browse the live `source_url` and follow up to 3 same-domain hops when needed to find upcoming events.
+4. Record the real `visited_urls` and actual `hops` used for each org.
+5. Build exactly one result entry per source id in `assets/sources.json`. Do not silently omit any org.
+6. If an org cannot be resolved cleanly, still include it with `status: "error"` and a short `error` message instead of skipping it.
+7. After writing the new `assets/events.json`, call `src/hf_sync.py` `push_events`.
+
+`assets/events.json` contract:
+
+- Write only `assets/events.json`.
+- Preserve the existing `meta.ttl_hours`.
+- Set `meta.schema_version` to `5`.
+- Refresh `meta.cached_at` to the current UTC ISO-8601 timestamp.
+- `results` must contain exactly one object per source with:
+  - `org_id`
+  - `org_name`
+  - `source_url`
+  - `status`
+  - `events`
+  - `checked_at`
+  - `hops`
+  - `visited_urls`
+  - `error`
+- `status` must be one of `ok`, `no_upcoming`, or `error`.
+- For `no_upcoming`, use `events: []` and `error: null`.
+- Put `tags` on each event object, not on the organization-level result object.
+- Infer `tags` from each event's actual content.
+- Do not just copy the source-level tags from `assets/sources.json`.
+- Use 2 to 6 concise lowercase tags when the content supports it.
+- If the tags cannot be inferred with reasonable confidence, it is acceptable to leave an event's `tags` empty.
+- Each event object must include:
+  - `title`
+  - `start_time`
+  - `event_url`
+  - `speaker`
+  - `affiliation`
+  - `category`
+  - `tags`
+  - `attending_count`
+
+Extraction rules:
+
+- Extract only events strictly after the actual current time at execution.
+- Keep all future events, sorted by `start_time` ascending.
+- If only a date is known, use `YYYY-MM-DD`.
+- If the page gives a timezone offset, preserve it.
+- If the page gives a local time without an offset, normalize it using that org's `timezone` from `assets/sources.json`.
+- Never invent timezone offsets.
+- Never invent missing dates, titles, speakers, affiliations, categories, tags, or event URLs.
+- Avoid using series names, page headings, site headings, navigation labels, or organization names as event titles.
+- Prefer the most specific event page as `event_url`; fall back to the source URL only when there is no dedicated event URL.
+- Fill `category` with the best specific event type supported by the source, for example `PhD defense`, `PhD proposal`, `Seminar`, `Talk`, `Lecture`, or `Podcast`.
+- Infer tags primarily from the abstract or description of each event when available, and use the title only as a secondary signal.
+- Broad umbrella tags are allowed when appropriate, and one or two more specific tags may be added when they are clearly supported by the event content.
+
+Attending-count preservation rules:
+
+- Preserve existing `attending_count` values from the current `assets/events.json` wherever possible.
+- Match first on `(org_id, normalized start_time to minute in UTC, event_url)`.
+- If there is no exact match, reuse the count only when there is exactly one prior event for the same `(org_id, normalized start_time to minute in UTC)`.
+- Otherwise leave `attending_count` at `0`.
+
+Validation requirements before finishing:
+
+1. Confirm every `assets/sources.json` id appears exactly once in `assets/events.json.results`.
+2. Confirm every `start_time` is ISO-parseable and sorts correctly in UTC after normalization.
+3. Confirm preserved `attending_count` values remain attached to matched events.
+4. Confirm every event has `category` and `tags`.
+5. Run `conda run -n robotic-seminars pytest -q`.
+6. Manually spot-check at least 3 orgs:
+   - one with explicit timezone offsets in the source page
+   - one with local times that need timezone normalization
+   - one with date-only entries
+
+Working style:
+
+- Use the web tool to browse live pages and the repo files together.
+- Do the work end-to-end without stopping for confirmation unless you hit a real blocker that prevents writing a complete `assets/events.json`.
+- In your final response, summarize what changed in `assets/events.json`, note any orgs that ended up with `status: "error"` or `status: "no_upcoming"`, and include the validation results plus whether `pull_events` and `push_events` succeeded.

sources.json → assets/sources.json

@@ -3,59 +3,69 @@
     "id": "cmu-ri-seminar",
     "name": "Carnegie Mellon University Robotics Institute Seminar Series",
     "url": "https://www.ri.cmu.edu/events/",
+    "timezone": "America/New_York",
     "tags": ["robotics"]
   },
   {
     "id": "rig",
     "name": "Robotics Institute Germany (RIG) Lecture Series",
     "url": "https://robotics-institute-germany.de/rig-lecture-series-weekly-online-lectures-on-robotics/",
+    "timezone": "Europe/Berlin",
     "tags": ["robotics"]
   },
   {
     "id": "stanford-engr319",
     "name": "Stanford Robotics & Autonomous Systems Seminar (ENGR319)",
     "url": "https://stanfordasl.github.io/robotics_seminar/",
+    "timezone": "America/Los_Angeles",
     "tags": ["robotics"]
   },
   {
     "id": "utoronto-ri",
     "name": "University of Toronto Robotics Institute Seminar Series",
     "url": "https://robotics.utoronto.ca/seminar-series/",
+    "timezone": "America/Toronto",
     "tags": ["robotics"]
   },
   {
     "id": "eth-rvc-talks",
     "name": "ETH Zürich Robotics, Vision, and Controls Talks",
     "url": "https://robotics-talks.com/",
+    "timezone": "Europe/Zurich",
     "tags": ["robotics"]
   },
   {
     "id": "umd-mrc-seminars",
     "name": "Maryland Robotics Center (UMD) Robotics Seminar Series",
     "url": "https://robotics.umd.edu/events/mrc-seminars",
+    "timezone": "America/New_York",
     "tags": ["robotics"]
   },
   {
     "id": "imperial-rl-seminar",
     "name": "Imperial College London Robot Learning Seminar Series",
     "url": "https://www.robot-learning.uk/seminar-series",
+    "timezone": "Europe/London",
     "tags": ["robotics"]
   },
   {
     "id": "gatech-irim-seminar",
     "name": "Georgia Tech IRIM Seminar Series",
     "url": "https://research.gatech.edu/robotics/irim-seminar-series",
+    "timezone": "America/New_York",
     "tags": ["robotics"]
   },
     { "id": "robot-talk", 
     "name": "Robot Talk", 
     "url": "https://www.robottalk.org/latest-episodes/",
+    "timezone": "UTC",
     "tags": ["robotics"] 
   },
   {
     "id": "montreal-robotics",
     "name": "Montréal Robotics / Mila Robot Learning Seminar",
     "url": "https://montrealrobotics.ca/robotlearningseries/",
+    "timezone": "America/Toronto",
     "tags": ["robotics"]
   }
-]
+]

events.json

@@ -1,270 +0,0 @@
-{
-  "meta": {
-    "schema_version": 4,
-    "cached_at": "2026-01-14T11:51:02.611033+00:00",
-    "ttl_hours": 12.0
-  },
-  "results": [
-    {
-      "org_id": "cmu-ri-seminar",
-      "org_name": "Carnegie Mellon University Robotics Institute Seminar Series",
-      "source_url": "https://www.ri.cmu.edu/events/",
-      "status": "ok",
-      "events": [
-        {
-          "title": "PhD Thesis Proposal",
-          "start_time": "2026-01-15T08:30:00-05:00",
-          "event_url": "https://www.ri.cmu.edu/events/",
-          "speaker": "Hsu-kuang Chiu",
-          "affiliation": "PhD Student, Robotics Institute, Carnegie Mellon University",
-          "attending_count": 0
-        },
-        {
-          "title": "Faculty Events",
-          "start_time": "2026-01-16T12:00:00-05:00",
-          "event_url": "https://www.ri.cmu.edu/events/",
-          "speaker": "Jean Hyaejin Oh",
-          "affiliation": "Associate Research Professor, Robotics Institute, Carnegie Mellon University",
-          "attending_count": 0
-        },
-        {
-          "title": "PhD Thesis Defense",
-          "start_time": "2026-01-20T12:00:00-05:00",
-          "event_url": "https://www.ri.cmu.edu/events/",
-          "speaker": "Muhammad Suhail Saleem",
-          "affiliation": "PhD Student, Robotics Institute, Carnegie Mellon University",
-          "attending_count": 0
-        }
-      ],
-      "checked_at": "2026-01-14T11:50:06.854624+00:00",
-      "hops": 1,
-      "visited_urls": [],
-      "error": null
-    },
-    {
-      "org_id": "rig",
-      "org_name": "Robotics Institute Germany (RIG) Lecture Series",
-      "source_url": "https://robotics-institute-germany.de/rig-lecture-series-weekly-online-lectures-on-robotics/",
-      "status": "ok",
-      "events": [
-        {
-          "title": "Distributed Petri Nets for Cobotics",
-          "start_time": "2026-01-15T14:15+00:00",
-          "event_url": "https://tu-berlin.zoom-x.de/j/63606623807?pwd=01u0brYFzcrZVRM60LOoAXW1GxAQXg.1",
-          "speaker": "Uwe Aßmann, Sebastian Ebert",
-          "affiliation": "TU Dresden",
-          "attending_count": 0
-        },
-        {
-          "title": "Robotics and Society: Lessons Learned from the Responsible Robotics Project",
-          "start_time": "2026-01-22T14:15+00:00",
-          "event_url": "https://tu-berlin.zoom-x.de/j/63606623807?pwd=01u0brYFzcrZVRM60LOoAXW1GxAQXg.1",
-          "speaker": "Svenja Breuer, Ruth Müller",
-          "affiliation": "TU München",
-          "attending_count": 0
-        },
-        {
-          "title": "Commercializing Robotic Innovations: Insights and Lessons from Three Startup Frontlines",
-          "start_time": "2026-01-29T14:15+00:00",
-          "event_url": "https://tu-berlin.zoom-x.de/j/63606623807?pwd=01u0brYFzcrZVRM60LOoAXW1GxAQXg.1",
-          "speaker": "Oskar von Stryk",
-          "affiliation": "TU Darmstadt",
-          "attending_count": 0
-        }
-      ],
-      "checked_at": "2026-01-14T11:50:25.769949+00:00",
-      "hops": 1,
-      "visited_urls": [],
-      "error": null
-    },
-    {
-      "org_id": "stanford-engr319",
-      "org_name": "Stanford Robotics & Autonomous Systems Seminar (ENGR319)",
-      "source_url": "https://stanfordasl.github.io/robotics_seminar/",
-      "status": "ok",
-      "events": [
-        {
-          "title": "Resilient Autonomy for Extreme and Uncertain Environments",
-          "start_time": "2026-01-16T15:00:00",
-          "event_url": "https://stanfordasl.github.io/robotics_seminar/",
-          "speaker": "Sebastian Scherer",
-          "affiliation": "CMU",
-          "attending_count": 0
-        },
-        {
-          "title": "TBD",
-          "start_time": "2026-01-30T15:00:00",
-          "event_url": "https://stanfordasl.github.io/robotics_seminar/",
-          "speaker": "Madhur Behl",
-          "affiliation": "UVirginia",
-          "attending_count": 0
-        },
-        {
-          "title": "TBD",
-          "start_time": "2026-02-06T15:00:00",
-          "event_url": "https://stanfordasl.github.io/robotics_seminar/",
-          "speaker": "Koushil Sreenath",
-          "affiliation": "UC Berkeley",
-          "attending_count": 0
-        }
-      ],
-      "checked_at": "2026-01-14T11:50:32.288175+00:00",
-      "hops": 1,
-      "visited_urls": [],
-      "error": null
-    },
-    {
-      "org_id": "utoronto-ri",
-      "org_name": "University of Toronto Robotics Institute Seminar Series",
-      "source_url": "https://robotics.utoronto.ca/seminar-series/",
-      "status": "ok",
-      "events": [
-        {
-          "title": "Robotics Institute Seminar: Mahdi Tavakoli (University of Alberta)",
-          "start_time": "2026-01-23T15:00:00-05:00",
-          "event_url": "https://robotics.utoronto.ca/event/robotics-institute-seminar-series-mahdi-tavakoli-university-of-alberta/",
-          "speaker": "Mahdi Tavakoli",
-          "affiliation": "University of Alberta",
-          "attending_count": 0
-        },
-        {
-          "title": "Robotics Institute Seminar: Sergey Stavisky (UC Davis)",
-          "start_time": "2026-03-20T15:00:00-05:00",
-          "event_url": "https://robotics.utoronto.ca/event/robotics-institute-seminar-series-sergey-stavisky-uc-davis/",
-          "speaker": "Sergey Stavisky",
-          "affiliation": "UC Davis",
-          "attending_count": 0
-        }
-      ],
-      "checked_at": "2026-01-14T11:50:37.362828+00:00",
-      "hops": 1,
-      "visited_urls": [],
-      "error": null
-    },
-    {
-      "org_id": "eth-rvc-talks",
-      "org_name": "ETH Zürich Robotics, Vision, and Controls Talks",
-      "source_url": "https://robotics-talks.com/",
-      "status": "ok",
-      "events": [
-        {
-          "title": "TBD",
-          "start_time": "2026-02-20T16:00:00+01:00",
-          "event_url": "https://robotics-talks.com/",
-          "speaker": "TBD",
-          "affiliation": null,
-          "attending_count": 0
-        },
-        {
-          "title": "TBD",
-          "start_time": "2026-03-06T16:00:00+01:00",
-          "event_url": "https://robotics-talks.com/",
-          "speaker": "TBD",
-          "affiliation": null,
-          "attending_count": 0
-        },
-        {
-          "title": "TBD",
-          "start_time": "2026-03-20T16:00:00+01:00",
-          "event_url": "https://robotics-talks.com/",
-          "speaker": "TBD",
-          "affiliation": null,
-          "attending_count": 0
-        }
-      ],
-      "checked_at": "2026-01-14T11:50:41.039165+00:00",
-      "hops": 1,
-      "visited_urls": [],
-      "error": null
-    },
-    {
-      "org_id": "umd-mrc-seminars",
-      "org_name": "Maryland Robotics Center (UMD) Robotics Seminar Series",
-      "source_url": "https://robotics.umd.edu/events/mrc-seminars",
-      "status": "ok",
-      "events": [
-        {
-          "title": "Robot Motion Learning with Physics-Based PDE Priors",
-          "start_time": "2026-02-02",
-          "event_url": "https://robotics.umd.edu/events/mrc-seminars#main-content",
-          "speaker": "Dr. Ahmed Qureshi",
-          "affiliation": "Purdue University",
-          "attending_count": 0
-        }
-      ],
-      "checked_at": "2026-01-14T11:50:44.115684+00:00",
-      "hops": 1,
-      "visited_urls": [],
-      "error": null
-    },
-    {
-      "org_id": "imperial-rl-seminar",
-      "org_name": "Imperial College London Robot Learning Seminar Series",
-      "source_url": "https://www.robot-learning.uk/seminar-series",
-      "status": "no_upcoming",
-      "events": [],
-      "checked_at": "2026-01-14T11:50:47.251095+00:00",
-      "hops": 1,
-      "visited_urls": [],
-      "error": null
-    },
-    {
-      "org_id": "gatech-irim-seminar",
-      "org_name": "Georgia Tech IRIM Seminar Series",
-      "source_url": "https://research.gatech.edu/robotics/irim-seminar-series",
-      "status": "ok",
-      "events": [
-        {
-          "title": "Beyond Scaling: Exploration, Guidance, and Symmetry in Robot Perception",
-          "start_time": "2026-01-14T12:15:00+00:00",
-          "event_url": "https://research.gatech.edu/robotics/irim-seminar-series",
-          "speaker": "Kostas Daniilidis",
-          "affiliation": "UPenn",
-          "attending_count": 0
-        },
-        {
-          "title": "TBA",
-          "start_time": "2026-01-28T12:15:00+00:00",
-          "event_url": "https://research.gatech.edu/robotics/irim-seminar-series",
-          "speaker": "Alexander Leonessa",
-          "affiliation": "Clemson",
-          "attending_count": 0
-        },
-        {
-          "title": "TBA",
-          "start_time": "2026-02-11T12:15:00+00:00",
-          "event_url": "https://research.gatech.edu/robotics/irim-seminar-series",
-          "speaker": "Robert Stuart-Smith",
-          "affiliation": "UPenn",
-          "attending_count": 0
-        }
-      ],
-      "checked_at": "2026-01-14T11:50:48.269849+00:00",
-      "hops": 1,
-      "visited_urls": [],
-      "error": null
-    },
-    {
-      "org_id": "robot-talk",
-      "org_name": "Robot Talk",
-      "source_url": "https://www.robottalk.org/latest-episodes/",
-      "status": "no_upcoming",
-      "events": [],
-      "checked_at": "2026-01-14T11:50:53.174374+00:00",
-      "hops": 1,
-      "visited_urls": [],
-      "error": null
-    },
-    {
-      "org_id": "montreal-robotics",
-      "org_name": "Montréal Robotics / Mila Robot Learning Seminar",
-      "source_url": "https://montrealrobotics.ca/robotlearningseries/",
-      "status": "no_upcoming",
-      "events": [],
-      "checked_at": "2026-01-14T11:50:58.296781+00:00",
-      "hops": 1,
-      "visited_urls": [],
-      "error": null
-    }
-  ]
-}

requirements.txt

@@ -1,14 +1,5 @@
 gradio==5.12.0
-fastapi==0.115.6
-uvicorn==0.34.0
-pydantic==2.10.5
-python-dateutil==2.9.0.post0
 pandas==2.2.3
-httpx==0.28.1
-h2>=4.1.0
-litellm>=1.0.0
+huggingface_hub>=0.22.0
 python-dotenv>=1.0.0
-beautifulsoup4>=4.12.2
 pytest>=8.0.0
-curl_cffi>=0.14.0
-huggingface_hub>=0.22.0

src/attending.py

@@ -1,84 +1,88 @@
 from __future__ import annotations
 
 import os
+from pathlib import Path
 
-import gradio as gr
 import pandas as pd
 
-from .cache_store import CacheStore
-from .models import EventResult, LlmConfig
+from .hf_sync import hf_enabled, push_events
+from .models import load_payload, parse_dt_utc, write_payload
 from .org_colors import org_tag
 
-COLS = ["Date/Time (UTC)", "Title", "Speaker", "Organization", "Attending"]
+LOCAL_DISPLAY_TZ = os.environ.get("LOCAL_DISPLAY_TZ", "Europe/Rome")
+LOCAL_TIME_COL = f"Local Time ({LOCAL_DISPLAY_TZ})"
+COLS = ["Source Time", LOCAL_TIME_COL, "Title", "Speaker", "Category", "Tags", "Organization", "Attending"]
 
 
-# (Previously had helper functions here; removed to keep file minimal.)
+def format_event_time(value: str, *, tz_name: str | None = None) -> str:
+    if "T" not in value:
+        return pd.Timestamp(value).strftime("%b %d").replace(" 0", " ")
+    dt = pd.Timestamp(value)
+    if tz_name and dt.tz is not None:
+        dt = dt.tz_convert(tz_name)
+    return dt.strftime("%b %d at %H:%M").replace(" 0", " ")
 
 
 def results_table(results: list[dict], colors: dict[str, str]) -> tuple[pd.DataFrame, list[tuple[int, int]]]:
     rows: list[dict[str, object]] = []
-    offset_h = int(os.environ.get("DISPLAY_UTC_OFFSET_HOURS", "0"))
-    tz_label = f"UTC{offset_h:+d}" if offset_h else "UTC"
-    for r_i, r in enumerate(results):
-        if r["status"] != "ok":
+    for result_index, result in enumerate(results):
+        if result["status"] != "ok":
             continue
-        for ev_i, ev in enumerate(r["events"]):
-            sp = ev["speaker"] or ""
-            aff = ev["affiliation"] or ""
-            speaker = f"{sp} ({aff})" if sp and aff else sp or aff
+        for event_index, event in enumerate(result["events"]):
+            speaker = event["speaker"] or ""
+            affiliation = event["affiliation"] or ""
+            event_tags = event.get("tags") or []
             rows.append(
                 {
-                    "Date/Time (UTC)": ev["start_time"],
-                    "_iso": ev["start_time"],
-                    "Title": f"[{ev['title']}]({ev['event_url']})" if ev["event_url"] else ev["title"],
-                    "Speaker": speaker,
-                    "Organization": org_tag(r["org_name"], colors),
-                    "Attending": f"{ev['attending_count']} (+)",
-                    "_org": r["org_name"],
-                    "_r": r_i,
-                    "_e": ev_i,
+                    "Source Time": format_event_time(event["start_time"]),
+                    LOCAL_TIME_COL: format_event_time(event["start_time"], tz_name=LOCAL_DISPLAY_TZ),
+                    "Title": f"[{event['title']}]({event['event_url']})" if event["event_url"] else event["title"],
+                    "Speaker": f"{speaker} ({affiliation})" if speaker and affiliation else speaker or affiliation,
+                    "Category": event["category"],
+                    "Tags": ", ".join(event_tags),
+                    "Organization": org_tag(result["org_name"], colors),
+                    "Attending": f"{event['attending_count']} (+)",
+                    "_sort": parse_dt_utc(event["start_time"]),
+                    "_org": result["org_name"],
+                    "_result": result_index,
+                    "_event": event_index,
                 }
             )
+
     if not rows:
         return pd.DataFrame(columns=COLS), []
-    df = pd.DataFrame(rows)
-    df["_sort"] = pd.to_datetime(df["_iso"], utc=True, format="ISO8601", errors="raise")
 
-    disp = df["_sort"] + pd.Timedelta(hours=offset_h)
-    date_only = ~df["_iso"].astype(str).str.contains("T")
-    df.loc[date_only, "Date/Time (UTC)"] = disp[date_only].dt.strftime("%b %d").str.replace(" 0", " ")
-    df.loc[~date_only, "Date/Time (UTC)"] = disp[~date_only].dt.strftime("%b %d at %H:%M").str.replace(" 0", " ")
-    df = df.sort_values(by=["_sort", "_org"], na_position="last").reset_index(drop=True)
-    row_map = list(zip(df["_r"].astype(int).tolist(), df["_e"].astype(int).tolist()))
-    return df.drop(columns=["_iso", "_sort", "_org", "_r", "_e"])[COLS], row_map
+    df = pd.DataFrame(rows).sort_values(by=["_sort", "_org"]).reset_index(drop=True)
+    row_map = list(zip(df["_result"].astype(int).tolist(), df["_event"].astype(int).tolist()))
+    return df.drop(columns=["_sort", "_org", "_result", "_event"])[COLS], row_map
 
 
 def inc_attending(
-    evt: gr.SelectData | None,
+    evt,
     results: list[dict],
     row_map: list[tuple[int, int]],
     *,
     colors: dict[str, str],
-    llm: LlmConfig,
-    ttl_hours: float,
-    cache_repo_id: str,
-    cache_hf_token: str | None,
+    events_path: str | Path = Path(os.environ.get("EVENTS_PATH", Path(os.environ.get("ASSETS_DIR", "assets")) / "events.json")),
+    **_ignored,
 ):
     if evt is None or evt.index is None:
-        df, row_map = results_table(results, colors)
-        return df, results, row_map
-    r = int(evt.index[0])
-    c = int(evt.index[1])
-    if c != COLS.index("Attending"):
-        df, row_map = results_table(results, colors)
-        return df, results, row_map
+        table, row_map = results_table(results, colors)
+        return table, results, row_map
 
-    r_i, ev_i = row_map[r]
-    results[r_i]["events"][ev_i]["attending_count"] += 1
+    row_index, column_index = map(int, evt.index)
+    if column_index != COLS.index("Attending"):
+        table, row_map = results_table(results, colors)
+        return table, results, row_map
 
-    cache = CacheStore(config=llm, ttl_hours=ttl_hours, repo_id=cache_repo_id, hf_token=cache_hf_token)
+    result_index, event_index = row_map[row_index]
+    results[result_index]["events"][event_index]["attending_count"] += 1
 
-    cache.write(results=[EventResult.model_validate(x) for x in results])
+    payload = load_payload(events_path)
+    payload["results"] = results
+    write_payload(payload, events_path)
+    if hf_enabled():
+        push_events(events_path)
 
-    df, row_map = results_table(results, colors)
-    return df, results, row_map
+    table, row_map = results_table(results, colors)
+    return table, results, row_map

src/cache_store.py

@@ -1,80 +0,0 @@
-from __future__ import annotations
-
-import json
-import shutil
-from datetime import datetime, timezone
-from pathlib import Path
-
-from huggingface_hub import hf_hub_download, HfApi  # type: ignore
-
-from .models import EventResult, LlmConfig
-from .ui_log import info
-
-HF_CACHE_PATH = "events.json"
-PATH = HF_CACHE_PATH
-CACHE_SCHEMA_VERSION = 4
-
-
-class CacheStore:
-    def __init__(
-        self,
-        config: LlmConfig,
-        ttl_hours: float,
-        repo_id: str,
-        hf_token: str | None = None,
-    ):
-        self.config = config
-        self.ttl_hours = ttl_hours
-        self.path = Path(PATH)
-        self.hf_repo_id = repo_id
-        self.hf_path = HF_CACHE_PATH
-        self.hf_token = hf_token
-
-    def pull_from_hf(self) -> None:
-
-        info(f"Pulling cache from HF dataset: {self.hf_repo_id}/{self.hf_path}")
-        downloaded = hf_hub_download(
-            repo_id=self.hf_repo_id,
-            filename=self.hf_path,
-            repo_type="dataset",
-            token=self.hf_token or None,
-        )
-        self.path.parent.mkdir(parents=True, exist_ok=True)
-        shutil.copy2(downloaded, self.path)
-
-    def _push_to_hf(self) -> None:
-
-        info(f"Pushing cache to HF dataset: {self.hf_repo_id}/{self.hf_path}")
-        api = HfApi(token=self.hf_token or None)
-        api.upload_file(
-            path_or_fileobj=str(self.path),
-            path_in_repo=self.hf_path,
-            repo_id=self.hf_repo_id,
-            repo_type="dataset",
-            commit_message="Update robotic_seminars cache",
-        )
-
-    def is_fresh(self) -> bool:
-        if not self.path.exists():
-            return False
-        cached_at = datetime.fromtimestamp(self.path.stat().st_mtime, tz=timezone.utc)
-        return (datetime.now(timezone.utc) - cached_at).total_seconds() < self.ttl_hours * 3600
-
-    def is_usable(self) -> bool:
-        if not self.is_fresh():
-            return False
-        meta = json.loads(self.path.read_text(encoding="utf-8"))["meta"]
-        return meta["schema_version"] == CACHE_SCHEMA_VERSION
-
-    def write(self, *, results: list[EventResult]) -> None:
-        self.path.parent.mkdir(parents=True, exist_ok=True)
-        payload = {
-            "meta": {
-                "schema_version": CACHE_SCHEMA_VERSION,
-                "cached_at": datetime.now(timezone.utc).isoformat(),
-                "ttl_hours": self.ttl_hours,
-            },
-            "results": [r.model_dump() for r in results],
-        }
-        self.path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
-        self._push_to_hf()

src/fetcher.py

@@ -1,115 +0,0 @@
-from __future__ import annotations
-
-import os
-import time
-from datetime import datetime, timezone
-from typing import Iterator
-
-from urllib.parse import urlparse
-from curl_cffi import requests as r  # type: ignore
-
-from dateutil import parser as dtparser
-
-from .models import (
-	EventResult,
-	LlmConfig,
-	LlmEvent,
-	OrgSource,
-	USER_AGENT,
-	llm_extract,
-	parse_dt_utc,
-)
-from .ui_log import error, info, warn
-
-
-class SeminarFetcher:
-    def __init__(
-        self,
-        config: LlmConfig,
-        now: datetime | None = None,
-        max_hops: int = 3,
-        max_events: int = 3,
-        previous_events_by_org: dict[str, list[dict]] | None = None,
-    ):
-        self.config = config
-        self.max_hops = max_hops
-        self.max_events = max_events
-        self.previous_events_by_org = previous_events_by_org or {}
-
-        if now is None:
-            raw = (os.environ.get("NOW_ISO") or "").strip()
-            now = dtparser.isoparse(raw) if raw else datetime.now(timezone.utc)
-        if now.tzinfo is None:
-            now = now.replace(tzinfo=timezone.utc)
-        self.now = now.astimezone(timezone.utc)
-
-    def fetch_html(self, url: str) -> str:
-        p = urlparse(url)
-        headers={"User-Agent": USER_AGENT, "Accept": "text/html", "Referer": f"{p.scheme}://{p.netloc}/"}
-        for i in range(3):
-            try:
-                resp = r.get(url, timeout=20, allow_redirects=True, impersonate="chrome120", headers=headers)
-                resp.raise_for_status()
-                return resp.text
-            except r.exceptions.RequestException as e:
-                if i == 2:
-                    raise
-                warn(f"HTTP fetch failed: {url}: {e} (retrying)")
-                time.sleep(2 ** i)
-
-    def fetch_next_event_for_org_stream(self, org: OrgSource) -> Iterator[EventResult]:
-        checked_at = datetime.now(timezone.utc).isoformat()
-        now_iso = self.now.isoformat()
-        url = str(org.url)
-        visited: list[str] = []
-        info(f"Inspecting {org.name}…")
-        result = EventResult(org_id=org.id, org_name=org.name, source_url=str(org.url), status="no_upcoming",
-            events=[], checked_at=checked_at, hops=self.max_hops, visited_urls=visited, error="Max hops reached.",
-        )
-
-        for hop_i in range(1, self.max_hops + 1):
-            visited.append(url)
-            info(f"Inspecting {org.name}: fetching {url}")
-
-            previous_events = self.previous_events_by_org[org.id] if org.id in self.previous_events_by_org else []
-            page_html = self.fetch_html(url)
-            hop = llm_extract(config=self.config, org=org, url=url, page_html=page_html, now_iso=now_iso, previous_events=previous_events)
-
-            if hop.status != "ok":
-                next_url = hop.next_url_to_check
-                if next_url and next_url not in visited:
-                    info(f"Inspecting {org.name}: following {next_url}")
-                    url = next_url
-                    continue
-                (error if hop.status == "error" else warn)(f"Inspecting {org.name}: {hop.status}: {hop.error}")
-                result.status = hop.status
-                result.hops = hop_i
-                result.error = hop.error
-                break
-
-            events: list[LlmEvent] = []
-            parsed = [(parse_dt_utc(e.start_time), e) for e in hop.events]
-            parsed.sort(key=lambda x: x[0])
-            for dt, e in parsed:
-                if dt <= self.now:
-                    continue
-                if e.event_url is None:
-                    e = e.model_copy(update={"event_url": url})
-                events.append(e)
-                if len(events) >= self.max_events:
-                    break
-
-            result.status = "ok" if events else "no_upcoming"
-            result.events = events
-            result.hops = hop_i
-            result.error = hop.error
-            if events:
-                info(f"Found {len(events)} upcoming event(s) for {org.name}")
-            else:
-                warn(f"No upcoming events found for {org.name}")
-            break
-
-        yield result
-
-    def fetch_next_event_for_org(self, org: OrgSource) -> EventResult:
-        return next(self.fetch_next_event_for_org_stream(org))

src/hf_sync.py

@@ -0,0 +1,59 @@
+from __future__ import annotations
+
+import os
+import shutil
+import sys
+from pathlib import Path
+
+from dotenv import load_dotenv
+
+ROOT_DIR = Path(__file__).resolve().parent.parent
+ASSETS_EVENTS_PATH = ROOT_DIR / "assets" / "events.json"
+HF_REPO_ID = "ar0s/robotic-seminars-cache"
+HF_EVENTS_FILE = "events.json"
+
+load_dotenv(ROOT_DIR / ".env")
+
+
+def hf_repo_id() -> str:
+    return HF_REPO_ID
+
+
+def hf_token() -> str | None:
+    return (os.environ.get("HF_TOKEN") or os.environ.get("HUGGINGFACE_HUB_TOKEN") or "").strip() or None
+
+def hf_enabled() -> bool:
+    return bool(hf_token())
+
+
+def pull_events(path: str | Path = ASSETS_EVENTS_PATH) -> None:
+    from huggingface_hub import hf_hub_download
+
+    downloaded = hf_hub_download(
+        repo_id=hf_repo_id(),
+        filename=HF_EVENTS_FILE,
+        repo_type="dataset",
+        token=hf_token(),
+    )
+    Path(path).parent.mkdir(parents=True, exist_ok=True)
+    shutil.copy2(downloaded, path)
+
+
+def push_events(path: str | Path = ASSETS_EVENTS_PATH) -> None:
+    from huggingface_hub import HfApi
+
+    HfApi(token=hf_token()).upload_file(
+        path_or_fileobj=str(path),
+        path_in_repo=HF_EVENTS_FILE,
+        repo_id=hf_repo_id(),
+        repo_type="dataset",
+        commit_message="Update robotic seminars events",
+    )
+
+
+if __name__ == "__main__":
+    command = sys.argv[1] if len(sys.argv) > 1 else ""
+    if command == "pull":
+        pull_events()
+    if command == "push":
+        push_events()

src/models.py

@@ -2,214 +2,39 @@ from __future__ import annotations
 
 import json
 import os
-from dataclasses import dataclass
 from datetime import datetime, timezone
-from typing import Literal
+from pathlib import Path
 
-import html
-from urllib.parse import urljoin, urlparse
+DEFAULT_EVENTS_PATH = Path(os.environ.get("EVENTS_PATH", Path(os.environ.get("ASSETS_DIR", "assets")) / "events.json"))
 
-from pydantic import BaseModel, Field, HttpUrl
 
-Status = Literal["ok", "no_upcoming", "error"]
+def load_payload(path: str | Path = DEFAULT_EVENTS_PATH) -> dict:
+    return json.loads(Path(path).read_text(encoding="utf-8"))
 
-USER_AGENT = (
-    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
-    "(KHTML, like Gecko) Chrome/120 Safari/537.36"
-)
 
-# SYSTEM_PROMPT = (
-#             "You are given the page text content and a list of links from the page. "
-#             "Find up to the NEXT 3 upcoming talks/events AFTER now, sorted by time ascending. "
-#             "If the page lists only a date (no time), output start_time as an ISO-8601 DATE like '2026-01-15'. "
-#             "If events are listed as lines with a date (e.g. 'Jan 15, 2026 – ...'), treat each such line as an event. "
-#             "If there is no dedicated per-event URL, set event_url to the Source URL. "
-#             "If the schedule is not on this page, set next_url_to_check to the single best link to follow (one of LINKS). "
-#             "Do NOT invent placeholder titles like 'TBA' unless the page text explicitly contains 'TBA'. "
-#             "IMPORTANT: Keep fields clean and separated: "
-#             "- events[].title MUST be the talk/event title ONLY (no speaker name, no affiliation, no date/time). "
-#             "- Put the speaker/person name in events[].speaker when present in the text (e.g. 'Speaker: …', 'by …', 'Presenter: …'). "
-#             "- If the speaker affiliation/institution is present (e.g. 'UC Davis', 'MIT', 'Google DeepMind'), put it in events[].affiliation (do not mix it into title). "
-#             "- If a line contains both speaker and title (e.g. 'Jane Doe — Learning Robots' or 'Learning Robots — Jane Doe'), split them correctly. "
-#             "Choose the talk title that is best-supported by the page text. Give priority to explicit cues like 'Title:', 'Talk title:', 'Topic:', and text near 'Abstract:'/'Summary:'. "
-#             "If a header includes a series label plus a person name/affiliation (e.g. 'Seminar: Jane Doe (MIT)'), treat that as speaker/affiliation (not title) and keep searching the body for the real title. "
-#             "e.g. this Robotics Institute Seminar: Mahdi Tavakoli (University of Alberta) is not a title"
-#             "Keep person name and affiliation seprate. Each should be put in its own field. "
-#             "Never guess; quote evidence from the provided text."
-#             "Never reply with any text other than the JSON object. If you don't find any events, still reply with a JSON object containing the no_upcoming status. "
-#             "REMEMBER: START TIME IS ALWAYS REQUIRED FOR EACH EVENT. IF THERE IS NO START TIME, DO NOT INCLUDE THE EVENT. "
+def write_payload(payload: dict, path: str | Path = DEFAULT_EVENTS_PATH) -> None:
+    Path(path).write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
 
-#             "Respond in JSON with the following schema: "
-
-#             "status: ok | no_upcoming | error"
-#             "events: array of up to 3 objects (required if ok)"
-#             "events[].title: string"
-#             "events[].start_time: ISO-8601 string; if only date is known, use YYYY-MM-DD (always required)" 
-#             "events[].event_url: string URL"
-#             "events[].speaker: string "
-#             "events[].affiliation: string (if prersent)"
-#             "events[].evidence: short snippet from provided text (always required)" 
-#             "error: short error string (required if error)"
-#             "next_url_to_check: string URL (optional - must be one of LINKS if provided)"
-# )
-
-SYSTEM_PROMPT = """
-You are a JSON extraction engine. You do NOT write code.
-
-CRITICAL OUTPUT CONSTRAINTS (HARD):
-- Your entire reply MUST be valid JSON (RFC 8259).
-- Reply with exactly ONE JSON object.
-- The first non-whitespace character MUST be "{" and the last MUST be "}".
-- Use double quotes for all JSON strings. Never use single quotes.
-- Do NOT include markdown fences (```), explanations, pseudocode, or Python.
-
-If you cannot follow these constraints, reply exactly:
-{"status":"error","error":"non_json_or_invalid_schema"}
-
-Event extraction rules (HARD):
-- Return up to 3 upcoming events after "now", sorted by start_time ascending.
-- Every event MUST include start_time and it MUST be ISO-8601 (e.g. "2026-01-15" or "2026-01-15T14:00+00:00").
-- Do NOT use month names or informal dates like "Jan 15, 2026".
-- If you cannot find a date/time in PAGE_TEXT for an event, DO NOT include that event.
-- If you find zero events with a date/time, return {"status":"no_upcoming"}.
-
-Title rule (HARD):
-- title MUST be copied verbatim from PAGE_TEXT (no paraphrasing).
-- title MUST come from the same local event block as the date/time:
-  - it must appear within 300 characters of the date/time text you used for start_time.
-- Do NOT use site/series/page headings or navigation as title.
-  Examples of INVALID titles: "Seminar Series", "Events", "Robotics", "University of Toronto", page header text.
-- If you cannot find a specific talk/topic title near the date/time, DO NOT include the event.
-
-If PREVIOUS_EVENTS is provided:
-- Use PREVIOUS_EVENTS as a strict copy source.
-- If PAGE_TEXT contains an event that matches one in PREVIOUS_EVENTS, you MUST include that event in your output and you MUST copy the entire event object exactly from PREVIOUS_EVENTS.
-- Do NOT omit a matched event. Do NOT say it is "already known". Do NOT reduce the number of returned events because PREVIOUS_EVENTS were provided.
-- If PREVIOUS_EVENTS contains events that are NOT present in PAGE_TEXT, ignore them.
-
-Final self-check (HARD, perform before replying):
-- Your reply must be valid JSON only.
-- For each event: verify start_time exists and is a non-empty string.
-- If an event has missing/empty start_time, REMOVE that event.
-- If no events remain, output {"status":"no_upcoming"}.
-
-Schema:
-{ "status": "ok"|"no_upcoming"|"error",
-    "events": [{"title": "...", "start_time": "...", "event_url": "...", "speaker": "", "affiliation": null, "attending_count": 0}],
-  "error": "...",
-  "next_url_to_check": "..." }
-
-"""
-
-
-class OrgSource(BaseModel):
-    id: str
-    name: str
-    url: HttpUrl
-    tags: list[str] = Field(default_factory=list)
-
-
-class LlmEvent(BaseModel):
-    title: str
-    start_time: str
-    event_url: str | None = None
-    speaker: str
-    affiliation: str | None = None
-    attending_count: int = 0
-
-
-class LlmHopResult(BaseModel):
-    status: Status
-    events: list[LlmEvent] = Field(default_factory=list)
-    error: str | None = None
-    next_url_to_check: str | None = None
-
-
-class EventResult(BaseModel):
-    org_id: str
-    org_name: str
-    source_url: str
-    status: Status
-    events: list[LlmEvent] = Field(default_factory=list)
-    checked_at: str
-    hops: int = 0
-    visited_urls: list[str] = Field(default_factory=list)
-    error: str | None = None
-
-
-@dataclass
-class LlmConfig:
-    model: str
-    api_key: str | None = None
-    api_base: str | None = None
 
 def parse_dt_utc(value: str) -> datetime:
-    from dateutil import parser as dtparser
-
-    dt = dtparser.isoparse(value)
+    dt = datetime.fromisoformat(value)
     if dt.tzinfo is None:
         dt = dt.replace(tzinfo=timezone.utc)
     return dt.astimezone(timezone.utc)
 
 
-def text_and_links(page_html: str, *, base_url: str, limit: int = 40) -> tuple[str, list[str]]:
-    from bs4 import BeautifulSoup
-
-    soup = BeautifulSoup(html.unescape(page_html), "html.parser")
-    for tag in soup(["script", "style", "noscript"]):
-        tag.decompose()
-    text = "\n".join(ln.strip() for ln in soup.get_text("\n").splitlines() if ln.strip())[:24000]
-
-    base_dom = urlparse(base_url).netloc.lower()
-    links: list[str] = []
-    for a in soup.find_all("a", href=True):
-        u = urljoin(base_url, str(a["href"]).strip())
-        p = urlparse(u)
-        if p.scheme in {"http", "https"} and p.netloc.lower() == base_dom:
-            links.append(u)
-        if len(links) >= limit:
-            break
-
-    return text, links
-
-
-def llm_extract(*, config: LlmConfig, org: OrgSource, url: str, page_html: str, now_iso: str, previous_events: list[dict]) -> LlmHopResult:
-    from litellm import completion  # type: ignore
-
-    page_text, links = text_and_links(page_html, base_url=url)
-    messages = [
-        {"role": "system", "content": SYSTEM_PROMPT},
-        {"role": "user", "content": json.dumps({"org": org.name, "now": now_iso, "source_url": url, "PREVIOUS_EVENTS": previous_events})},
-        {"role": "user", "content": "LINKS:\n" + "\n".join(links)},
-        {"role": "user", "content": "PAGE_TEXT_BEGIN\n" + page_text + "\nPAGE_TEXT_END"},
-        {"role": "user", "content": "Return ONLY one JSON object (no markdown, no code). "
-                        "Must start with '{' and end with '}'. Use double quotes only. "
-                        "start_time MUST be ISO-8601 only (e.g. 2026-01-15 or 2026-01-15T14:00+00:00), never 'Jan 15, 2026'. "
-                        "Before returning, delete any event missing/empty start_time. "
-                        "Title must be copied verbatim from PAGE_TEXT near the date/time. "
-                        "IMPORTANT: PREVIOUS_EVENTS are NOT a reason to omit events. If PAGE_TEXT contains an event that matches PREVIOUS_EVENTS, you MUST re-output it by copying the entire event object exactly from PREVIOUS_EVENTS. "
-                        "If none remain, return {\"status\":\"no_upcoming\"}."}
-    ]
-
-    timeout = float(os.environ.get("LLM_TIMEOUT_SECS", "60"))
-    kwargs: dict[str, object] = {"model": config.model, "temperature": 0, "timeout": timeout}
-    if config.api_key:
-        kwargs["api_key"] = config.api_key
-    if config.api_base:
-        kwargs["api_base"] = config.api_base
-
-    content = completion(messages=messages, **kwargs)["choices"][0]["message"]["content"]
-    content = content.replace('`', '').replace('json', '')
-    content = content[content.find("{") : content.rfind("}") + 1]
-
-    try:
-        content = json.loads(content)
-    except json.JSONDecodeError:
-        raise ValueError(f"LLM did not return valid JSON:\n{content}")
-    
-    try:
-        LlmHopResult.model_validate(content)
-    except Exception as e:
-        raise ValueError(f"LLM returned JSON that does not match schema: {e}\n{content}")
-
-    return LlmHopResult.model_validate(content)
+def infer_category(*, title: str, org_name: str, event_url: str) -> str:
+    text = " ".join((title, org_name, event_url)).lower().replace("-", " ").replace("_", " ").replace("/", " ")
+    if "thesis defense" in text or "phd defense" in text:
+        return "PhD defense"
+    if "thesis proposal" in text or "phd proposal" in text:
+        return "PhD proposal"
+    if "lecture" in text:
+        return "Lecture"
+    if "talk" in text:
+        return "Talk"
+    if "podcast" in text or "episode" in text:
+        return "Podcast"
+    if "seminar" in text:
+        return "Seminar"
+    return ""

src/org_colors.py

@@ -2,24 +2,12 @@ from __future__ import annotations
 
 from html import escape
 
-PALETTE = [
-    "#e6194b",
-    "#3cb44b",
-    "#4363d8",
-    "#f58231",
-    "#911eb4",
-    "#46f0f0",
-    "#f032e6",
-    "#bcf60c",
-    "#fabebe",
-    "#008080",
-    "#e6beff",
-    "#9a6324",
-]
+PALETTE = ["#e6194b", "#3cb44b", "#4363d8", "#f58231", "#911eb4", "#008080"]
 
 
-def org_colors(sources: list[dict]) -> dict[str, str]:
-    return {s["name"]: PALETTE[i % len(PALETTE)] for i, s in enumerate(sources)}
+def org_colors(names) -> dict[str, str]:
+    unique_names = list(dict.fromkeys(names))
+    return {name: PALETTE[index % len(PALETTE)] for index, name in enumerate(unique_names)}
 
 
 def org_tag(org: str, colors: dict[str, str]) -> str:

src/ui_log.py

@@ -1,39 +0,0 @@
-from __future__ import annotations
-
-from datetime import datetime, timezone
-
-LINES: list[str] = []
-
-
-def ts() -> str:
-    return datetime.now(timezone.utc).isoformat(timespec="seconds")
-
-
-def log(level: str, msg: str) -> None:
-    html = {
-        "INFO": "dodgerblue",
-        "WARN": "goldenrod",
-        "ERR": "crimson",
-    }
-    c = html.get(level)
-    tag = f"<span style=\"color:{c}\">{level}</span>" if c else level
-    LINES.append(f"[{ts()}] {tag}: {msg}")
-
-
-def info(msg: str) -> None:
-    log("INFO", msg)
-
-
-def warn(msg: str) -> None:
-    log("WARN", msg)
-
-
-def error(msg: str) -> None:
-    log("ERR", msg)
-
-def clear() -> None:
-    LINES.clear()
-
-
-def lines() -> list[str]:
-    return LINES

tests/test_golden.py

@@ -1,54 +1,84 @@
-import json, os, shutil, tempfile
-from pathlib import Path
-
-ROOT = Path(__file__).resolve().parents[1]
-GOLDEN = ROOT / ".data" / "events.json"
-
-
-def _load(p: Path) -> dict: return json.loads(p.read_text(encoding="utf-8"))
-
-
-def _norm(d: dict) -> dict:
-    d = json.loads(json.dumps(d)); (d.get("meta") or {}).pop("cached_at", None)
-    for r in d.get("results") or []:
-        if isinstance(r, dict): r.pop("checked_at", None)
-    return d
-
-
-def test_cached_results_match_golden():
-    golden = _load(GOLDEN); model = (golden.get("meta") or {}).get("model"); assert isinstance(model, str)
-    with tempfile.TemporaryDirectory() as td:
-        data_dir = Path(td); shutil.copy2(GOLDEN, data_dir / "events.json")
-        from src.cache_store import CacheStore
-        from src.models import LlmConfig
-        cache = CacheStore(config=LlmConfig(model=model), ttl_hours=9999.0, repo_id="", path=str(data_dir / "events.json"))
-        assert cache.is_usable()
-        got = _load(cache.path)
-        assert got["results"] == golden["results"]
-
-
-def test_live_crawl_matches_golden_snapshot():
-    if os.environ.get("RUN_LIVE_TESTS") != "1": return
-    golden = _load(GOLDEN); model = (golden.get("meta") or {}).get("model"); assert isinstance(model, str)
-    with tempfile.TemporaryDirectory() as td:
-        data_dir = Path(td)
-        os.environ.update({"LLM_TEMPERATURE": "0", "LLM_SEED": "1", "LITELLM_FALLBACK_MODELS": ""})
-        cached_at = (golden.get("meta") or {}).get("cached_at")
-        if isinstance(cached_at, str) and cached_at.strip(): os.environ["NOW_ISO"] = cached_at.strip()
-        from src.cache_store import CacheStore
-        from src.fetcher import SeminarFetcher
-        from src.models import LlmConfig, OrgSource
-        cfg = LlmConfig(model=model)
-        raw_sources = json.loads((ROOT / "sources.json").read_text(encoding="utf-8"))
-        sources = [OrgSource.model_validate(s) for s in raw_sources]
-        fetcher = SeminarFetcher(config=cfg, now=None, max_hops=3, max_events=3)
-        results = [fetcher.fetch_next_event_for_org(o) for o in sources]
-        cache = CacheStore(
-            config=cfg,
-            ttl_hours=float((golden.get("meta") or {}).get("ttl_hours") or 12.0),
-            repo_id="",
-            path=str(data_dir / "events.json"),
-        )
-        cache.write(results=results)
-        produced = _load(cache.path)
-    assert _norm(produced) == _norm(golden)
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+from src.attending import COLS, LOCAL_TIME_COL, inc_attending, results_table
+from src.models import infer_category, load_payload, parse_dt_utc, write_payload
+
+
+def sample_results() -> list[dict]:
+    return [
+        {
+            "org_id": "cmu",
+            "org_name": "CMU",
+            "source_url": "https://example.com",
+            "status": "ok",
+            "events": [
+                {
+                    "title": "Event",
+                    "start_time": "2026-04-13T09:00:00-04:00",
+                    "event_url": "https://example.com/event",
+                    "speaker": "Speaker",
+                    "affiliation": None,
+                    "category": "Seminar",
+                    "tags": ["robotics", "university"],
+                    "attending_count": 0,
+                }
+            ],
+            "checked_at": "2026-04-12T20:22:12+00:00",
+            "hops": 1,
+            "visited_urls": [],
+            "error": None,
+        }
+    ]
+
+
+def test_results_table_shows_source_and_local_time():
+    table, _ = results_table(sample_results(), {"CMU": "#000000"})
+    assert list(table.columns) == COLS
+    assert table.loc[0, "Source Time"] == "Apr 13 at 9:00"
+    assert table.loc[0, LOCAL_TIME_COL] == "Apr 13 at 15:00"
+    assert table.loc[0, "Category"] == "Seminar"
+    assert table.loc[0, "Tags"] == "robotics, university"
+
+
+def test_inc_attending_updates_results_and_events_file(tmp_path):
+    path = tmp_path / "events.json"
+    payload = {
+        "meta": {
+            "schema_version": 5,
+            "cached_at": "2026-04-12T20:22:12+00:00",
+            "ttl_hours": 12.0,
+        },
+        "results": sample_results(),
+    }
+    write_payload(payload, path)
+
+    results = load_payload(path)["results"]
+    table, row_map = results_table(results, {"CMU": "#000000"})
+
+    updated_table, updated_results, updated_row_map = inc_attending(
+        SimpleNamespace(index=(0, COLS.index("Attending"))),
+        results,
+        row_map,
+        colors={"CMU": "#000000"},
+        events_path=path,
+    )
+
+    assert updated_results[0]["events"][0]["attending_count"] == 1
+    assert updated_table.loc[0, "Attending"] == "1 (+)"
+    assert updated_row_map == row_map
+    assert load_payload(path)["results"][0]["events"][0]["attending_count"] == 1
+
+
+def test_parse_dt_utc_normalizes_naive_and_offset_datetimes():
+    assert parse_dt_utc("2026-04-13T09:00:00").isoformat() == "2026-04-13T09:00:00+00:00"
+    assert parse_dt_utc("2026-04-13T09:00:00-04:00").isoformat() == "2026-04-13T13:00:00+00:00"
+
+
+def test_infer_category_prefers_defense_over_seminar():
+    assert infer_category(
+        title="Efficient and Controllable Diffusion Models",
+        org_name="Robotics Institute Seminar Series",
+        event_url="https://example.com/ri-phd-thesis-defense/",
+    ) == "PhD defense"