Compare commits

...

82 Commits

Author SHA1 Message Date
Ryan Walters
966732a6d2 feat: modernize build tooling and add CI/CD workflow
Switch to Bun for 2-5x faster frontend builds, implement cargo-chef for
reliable Rust dependency caching, and add Biome for fast code
formatting.

Build system improvements:
- Replace pnpm with Bun for frontend package management
- Add cargo-chef to Dockerfile for better Rust build layer caching
- Update all commands to use bun instead of pnpm

Developer experience:
- Add comprehensive Justfile commands (format, format-check, db)
- Implement automated PostgreSQL Docker setup with random port
allocation
- Add stricter checks (--deny warnings on clippy, --all-features flag)

Code quality:
- Add Biome formatter for 10-100x faster TypeScript/JavaScript
formatting
- Add GitHub Actions CI/CD workflow for automated checks
- Update .dockerignore with comprehensive exclusions
- Format all code with cargo fmt (Rust) and Biome (TypeScript)

All changes maintain backward compatibility and can be tested
incrementally.
2025-11-18 18:59:03 -06:00
Ryan Walters
3292d35521 build(docker): copy migrations directory to build context
Ensures database migration files are available during the Docker build process.
2025-11-03 12:07:27 -06:00
Ryan Walters
71ac0782d0 feat(json): enhance error context with debug mode detailed reporting
Improve JSON parsing error messages with build-specific behavior:
- Debug builds: Show full parent object context and type mismatch details
- Release builds: Keep minimal snippets to avoid log spam

Add comprehensive test coverage for error handling and path parsing.
2025-11-03 12:04:20 -06:00
Ryan Walters
1c6d2d4b6e perf: implement batch operations and optimize database indexes
Add batch upsert functionality to reduce database round-trips from N to 1 when inserting courses. Create comprehensive database indexes for common query patterns including term/subject lookups, time-series metrics, and job scheduling. Remove redundant indexes and add monitoring guidance for BRIN index effectiveness.
2025-11-03 11:18:42 -06:00
Ryan Walters
51f8256e61 feat: implement comprehensive retry mechanism and improve observability
Add retry tracking to scrape jobs with configurable max retries (default 5), implement
automatic database migrations on startup, and significantly reduce logging noise from
infrastructure layers. Enhanced tracing with structured spans for better debugging while
keeping output readable by suppressing verbose trace logs from rate limiters and session
management. Improved error handling with detailed retry context and proper session cookie
validation.
2025-11-03 10:18:07 -06:00
Ryan Walters
b1ed2434f8 feat: add ESLint configuration and testing infrastructure
Add comprehensive ESLint setup with React and TypeScript support, create basic integration tests for the shutdown utilities, and enhance the Justfile with a new check command that runs all validation steps (cargo check, clippy, tests, and linting).
2025-11-03 02:21:35 -06:00
Ryan Walters
47c23459f1 refactor: implement comprehensive graceful shutdown across all services
Implements graceful shutdown with broadcast channels and proper timeout handling
for scraper workers, scheduler, bot service, and status update tasks. Introduces
centralized shutdown utilities and improves service manager to handle parallel
shutdown with per-service timeouts instead of shared timeout budgets.

Key changes:
- Add utils module with shutdown helper functions
- Update ScraperService to return errors on shutdown failures
- Refactor scheduler with cancellable work tasks and 5s grace period
- Extract worker shutdown logic into helper methods for clarity
- Add broadcast channel shutdown support to BotService and status task
- Improve ServiceManager to shutdown services in parallel with individual timeouts
2025-11-03 02:10:01 -06:00
Ryan Walters
8af9b0a1a2 refactor(scraper): implement graceful shutdown with broadcast channels
Replace task abortion with broadcast-based graceful shutdown for scheduler and workers. Implement cancellation tokens for in-progress work with 5s timeout. Add tokio-util dependency for CancellationToken support. Update ServiceManager to use completion channels and abort handles for better service lifecycle control.
2025-11-03 01:22:12 -06:00
020a00254f chore: improve database pool connection options, tighter thresholds & limits 2025-09-14 12:18:39 -05:00
45de5be60d refactor: redistribute main.rs into new modules for app & service initialization 2025-09-14 12:18:15 -05:00
8384f418c8 refactor: remove unused/dead code, apply allowances to the rest 2025-09-14 01:57:30 -05:00
3dca896a35 feat(web): add 10 second timeout layer 2025-09-14 01:47:52 -05:00
1b7d2d2824 fix: make version retrieval search current dir, add basic logs, existence check 2025-09-13 22:08:48 -05:00
e370008d75 fix: pass RAILWAY_GIT_COMMIT_SHA through Docker, provide Cargo.toml for frontend (version retrieval) 2025-09-13 22:04:44 -05:00
176574343f fix: provide proper theme-based colors to all elements necessary 2025-09-13 21:57:56 -05:00
91899bb109 fix: limit devtools panel to dev mode 2025-09-13 21:52:14 -05:00
08ae54c093 fix: use wildcard COPY for .git directory, use RAILWAY_GIT_COMMIT_SHA as fallback 2025-09-13 21:20:16 -05:00
33b8681b19 chore: use locale-based number formatting 2025-09-13 21:12:13 -05:00
398a1b9474 feat: dark mode with theme toggle button 2025-09-13 21:11:16 -05:00
a732ff9a15 feat: better frontend state implementation, acquire version in frontend build time 2025-09-13 20:29:18 -05:00
bfcd868337 refactor: proper implementation of services status, better styling/appearance/logic 2025-09-13 19:34:34 -05:00
99f0d0bc49 fix: add build.rs and .git dir to Dockerfile COPY build step, add git dependency 2025-09-13 19:09:27 -05:00
8b7729788d chore: replace template properties 2025-09-13 19:02:01 -05:00
27b0cb877e feat: display project version on frontend 2025-09-13 18:58:35 -05:00
8ec2f7d36f chore: bump version to 0.3.2 2025-09-13 18:52:23 -05:00
28a8a15b6b feat: embed git commit into binary, provide link on frontend 2025-09-13 18:51:48 -05:00
19b3a98f66 feat: setup span recording for CustomJsonFormatter, use 'yansi' for better ANSI terminal colors in CustomPrettyFormatter 2025-09-13 18:40:55 -05:00
b64aa41b14 feat: better profile-based router assembly, tracing layer for responses with span-based request paths 2025-09-13 18:03:20 -05:00
64449e8976 feat: setup pretty frontend for system status 2025-09-13 17:49:35 -05:00
2e0fefa5ee feat: implement interval backoff for presence indicator 2025-09-13 16:15:33 -05:00
97488494fb chore: bump version to 0.3.0 2025-09-13 15:52:40 -05:00
b3322636a9 feat: setup frontend build code, tune .dockerignore patterns
also removed diesel.toml
2025-09-13 15:48:25 -05:00
878cc5f773 docs: setup proper documentation, organize & clean README 2025-09-13 15:27:32 -05:00
94fb6b4190 chore: set banner URL default in config, remove old mentions of redis 2025-09-13 14:48:49 -05:00
e3b638a7d8 feat: add ETag & Cache-Control headers, cached hexadecimal hashes via rapidhash 2025-09-13 13:24:54 -05:00
404a52e64c feat: cache mime types for valid assets, use octet-stream content type 2025-09-13 12:37:36 -05:00
a917315967 fix: simplify asset serving, use fallback primarily 2025-09-13 12:23:27 -05:00
9d51fde893 feat: add arguments for enabling/disabling srevices 2025-09-13 12:06:10 -05:00
79fc931077 refactor: remove 'auto' mode, just specify value via constant for better clap visibility 2025-09-13 11:38:43 -05:00
f3861a60c4 chore: add dev-release helper profile into Cargo.toml 2025-09-13 11:34:25 -05:00
26b1a88860 chore: use clippy by default for check command, fix lint 2025-09-13 11:31:09 -05:00
27ac9a7302 feat: add formatter CLI argument, setup asset embedding in release mode 2025-09-13 11:30:57 -05:00
1d345ed247 chore: customize bacon, add 'dev' job 2025-09-13 11:30:23 -05:00
6f831f5fa6 feat: setup web/ for tanstack router frontend 2025-09-13 11:30:11 -05:00
ac2638dd9a feat: implement proper SIGTERM handling for container shutdown 2025-09-13 09:43:47 -05:00
cfb847f2e5 feat: holiday exclusion logic for ICS command 2025-09-13 02:20:27 -05:00
e7d47f1f96 feat: implement ICS command 2025-09-13 01:50:18 -05:00
9a48587479 chore: drop redis 2025-09-13 01:49:47 -05:00
624247ee14 feat: basic activity status 2025-09-13 01:04:46 -05:00
430e2a255b fix: avoid crashing due to odd url parse 2025-09-13 01:01:49 -05:00
bbc78131ec feat: setup recoverable/unrecoverable job error distinction, delete unrecoverable jobs 2025-09-13 00:48:11 -05:00
77ab71d4d5 feat: map RAILWAY_DEPLOYMENT_DRAINING_SECONDS to SHUTDOWN_TIMEOUT 2025-09-13 00:36:11 -05:00
9d720bb0a7 feat: implement common job trait & better interface for scheduler & workers 2025-09-13 00:17:53 -05:00
dcc564dee6 fix: credit_hour_session is optional 2025-09-12 23:50:36 -05:00
4ca55a1fd4 feat: schedule & query jobs efficiently in batches 2025-09-12 23:41:27 -05:00
a6e7adcaef fix: improve json error handling, make email_address optional 2025-09-12 23:36:07 -05:00
752c855dec chore: drop env prefixed config vars 2025-09-12 22:39:32 -05:00
14b02df8f4 feat: much better JSON logging, project-wide logging improvements, better use of debug/trace levels, field attributes 2025-09-12 22:01:14 -05:00
00cb209052 fix: disable poor error snippet 2025-09-12 21:40:07 -05:00
dfc05a2789 feat: setup rate limiter middleware & config 2025-09-12 21:12:06 -05:00
fe798e1867 fix: avoid COPY of non existent dir, add .dockerignore 2025-09-12 20:57:33 -05:00
39688f800f chore: update Dockerfile rust to 1.89.0 2025-09-12 20:53:24 -05:00
b2b4bb67f0 chore: rustfmt 2025-09-12 20:52:07 -05:00
e5d8cec2d6 refactor: reorganize banner api files, fix clippy lints, reformat 2025-09-12 20:50:47 -05:00
e9a0558535 feat: asynchronous, rate limited term session acquisition 2025-09-12 20:35:12 -05:00
353c36bcf2 feat: 'search' example binary 2025-09-12 20:12:41 -05:00
2f853a7de9 feat: middleware headers, fix concurrent session cookies issue, middleware headers, invalid session details 2025-09-12 20:12:12 -05:00
dd212c3239 chore: update dependencies, add sqlx 'macros', add futures, add 'http' (explicit) 2025-09-12 20:11:13 -05:00
8ff3a18c3e feat: Dockerfile 2025-09-01 00:47:26 -05:00
43647096e9 feat: scraper system 2025-09-01 00:46:38 -05:00
1bdbd1d6d6 chore: remove unused dependencies 2025-09-01 00:26:20 -05:00
23be6035ed feat: much better, smarter session acquisition 2025-08-31 15:34:49 -05:00
139e4aa635 feat: translate over to sqlx, remove diesel 2025-08-31 15:34:49 -05:00
677bb05b87 chore: update & sort dependencies, add sqlx, remove 'migrations' 2025-08-29 12:52:46 -05:00
f2bd02c970 chore: add bacon config 2025-08-29 12:10:57 -05:00
8cdf969a53 feat: command logging, explicit builtin command error handler 2025-08-29 12:10:57 -05:00
4764d48ac9 feat: move scraper into separate module, begin building data models 2025-08-29 11:07:46 -05:00
e734e40347 feat: setup diesel & schema, course with metrics/audit tables 2025-08-27 18:57:43 -05:00
c7117f14a3 feat: smart day string, terse refactor and use types properly, work on unimplemented commands lightly, util modules, 2025-08-27 13:46:41 -05:00
cb8a595326 chore: solve lints, improve formatting 2025-08-27 12:43:43 -05:00
ac70306c04 feat: improve logging, solve lints, improve implementations, remove unused code, standardize things 2025-08-27 12:43:43 -05:00
9972357cf6 feat: implement simple web service, improve ServiceManager encapsulation 2025-08-27 11:58:57 -05:00
98 changed files with 9219 additions and 1592 deletions

51
.dockerignore Normal file
View File

@@ -0,0 +1,51 @@
# Build artifacts
target/
**/target/
# Documentation
README.md
docs/
*.md
# Old Go codebase
go/
# Development configuration
bacon.toml
.env
.env.*
!.env.example
# CI/CD
.github/
.git/
# Development tools
Justfile
rust-toolchain.toml
# Frontend build artifacts and cache
web/node_modules/
web/dist/
web/.vite/
web/.tanstack/
web/.vscode/
# IDE and editor files
.vscode/
.idea/
*.swp
*.swo
*~
# OS files
.DS_Store
Thumbs.db
# Test coverage
coverage/
*.profdata
*.profraw
# SQLx offline mode (include this in builds)
!.sqlx/

65
.github/workflows/ci.yml vendored Normal file
View File

@@ -0,0 +1,65 @@
name: CI
on:
push:
branches: [master]
pull_request:
branches: [master]
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
components: rustfmt, clippy
- name: Setup Bun
uses: oven-sh/setup-bun@v1
with:
bun-version: latest
- name: Cache Rust dependencies
uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Install frontend dependencies
working-directory: web
run: bun install --frozen-lockfile
- name: Check Rust formatting
run: cargo fmt --all -- --check
- name: Check TypeScript formatting
working-directory: web
run: bun run format:check
- name: TypeScript type check
working-directory: web
run: bun run typecheck
- name: ESLint
working-directory: web
run: bun run lint
- name: Clippy
run: cargo clippy --all-features -- --deny warnings
- name: Run tests
run: cargo test --all-features
- name: Build frontend
working-directory: web
run: bun run build
- name: Build backend
run: cargo build --release --bin banner

4
.gitignore vendored
View File

@@ -1,3 +1,5 @@
.env
/target
/go/
/go/
.cargo/config.toml
src/scraper/README.md

3
.vscode/settings.json vendored Normal file
View File

@@ -0,0 +1,3 @@
{
"rust-analyzer.check.command": "clippy"
}

1143
Cargo.lock generated
View File

File diff suppressed because it is too large Load Diff

View File

@@ -1,32 +1,60 @@
[package]
name = "banner"
version = "0.1.0"
version = "0.3.4"
edition = "2024"
default-run = "banner"
[dependencies]
tokio = { version = "1.47.1", features = ["full"] }
axum = "0.8.4"
serenity = { version = "0.12.4", features = ["rustls_backend"] }
reqwest = { version = "0.12.23", features = ["json", "cookies"] }
diesel = { version = "2.2.12", features = ["chrono", "postgres", "uuid"] }
redis = { version = "0.32.5", features = ["tokio-comp"] }
figment = { version = "0.10.19", features = ["toml", "env"] }
serde_json = "1.0.143"
serde = { version = "1.0.219", features = ["derive"] }
governor = "0.10.1"
tracing = "0.1.41"
tracing-subscriber = { version = "0.3.19", features = ["env-filter"] }
dotenvy = "0.15.7"
poise = "0.6.1"
async-trait = "0.1"
fundu = "2.0.1"
anyhow = "1.0.99"
thiserror = "2.0.16"
chrono = { version = "0.4", features = ["serde"] }
chrono-tz = "0.8"
rand = "0.8"
regex = "1.10"
url = "2.5"
async-trait = "0.1"
axum = "0.8.4"
bitflags = { version = "2.9.4", features = ["serde"] }
chrono = { version = "0.4.42", features = ["serde"] }
compile-time = "0.2.0"
time = "0.3.41"
bitflags = { version = "2.9.3", features = ["serde"] }
cookie = "0.18.1"
dashmap = "6.1.0"
dotenvy = "0.15.7"
figment = { version = "0.10.19", features = ["toml", "env"] }
fundu = "2.0.1"
futures = "0.3"
http = "1.3.1"
poise = "0.6.1"
rand = "0.9.2"
regex = "1.10"
reqwest = { version = "0.12.23", features = ["json", "cookies"] }
reqwest-middleware = { version = "0.4.2", features = ["json"] }
serde = { version = "1.0.219", features = ["derive"] }
serde_json = "1.0.143"
serenity = { version = "0.12.4", features = ["rustls_backend"] }
sqlx = { version = "0.8.6", features = [
"runtime-tokio-rustls",
"postgres",
"chrono",
"json",
"macros",
] }
thiserror = "2.0.16"
time = "0.3.43"
tokio = { version = "1.47.1", features = ["full"] }
tokio-util = "0.7"
tl = "0.7.8"
tracing = "0.1.41"
tracing-subscriber = { version = "0.3.20", features = ["env-filter", "json"] }
url = "2.5"
governor = "0.10.1"
once_cell = "1.21.3"
serde_path_to_error = "0.1.17"
num-format = "0.4.4"
tower-http = { version = "0.6.0", features = ["fs", "cors", "trace", "timeout"] }
rust-embed = { version = "8.0", features = ["debug-embed", "include-exclude"] }
mime_guess = "2.0"
clap = { version = "4.5", features = ["derive"] }
rapidhash = "4.1.0"
yansi = "1.0.1"
[dev-dependencies]
# A 'release mode' profile that compiles quickly, but still 'appears' like a release build, useful for debugging
[profile.dev-release]
inherits = "dev"
debug-assertions = false

116
Dockerfile Normal file
View File

@@ -0,0 +1,116 @@
# Build arguments
ARG RUST_VERSION=1.89.0
ARG RAILWAY_GIT_COMMIT_SHA
# --- Frontend Build Stage ---
FROM oven/bun:1 AS frontend-builder
WORKDIR /app
# Copy backend Cargo.toml for build-time version retrieval
COPY ./Cargo.toml ./
# Copy frontend package files
COPY ./web/package.json ./web/bun.lock* ./
# Install dependencies
RUN bun install --frozen-lockfile
# Copy frontend source code
COPY ./web ./
# Build frontend
RUN bun run build
# --- Chef Base Stage ---
FROM lukemathwalker/cargo-chef:latest-rust-${RUST_VERSION} AS chef
WORKDIR /app
# --- Planner Stage ---
FROM chef AS planner
COPY Cargo.toml Cargo.lock ./
COPY build.rs ./
COPY src ./src
# Migrations & .sqlx specifically left out to avoid invalidating cache
RUN cargo chef prepare --recipe-path recipe.json --bin banner
# --- Rust Build Stage ---
FROM chef AS builder
# Set build-time environment variable for Railway Git commit SHA
ARG RAILWAY_GIT_COMMIT_SHA
ENV RAILWAY_GIT_COMMIT_SHA=${RAILWAY_GIT_COMMIT_SHA}
# Copy recipe from planner and build dependencies only
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json --bin banner
# Install build dependencies for final compilation
RUN apt-get update && apt-get install -y \
pkg-config \
libssl-dev \
git \
&& rm -rf /var/lib/apt/lists/*
# Copy source code and built frontend assets
COPY Cargo.toml Cargo.lock ./
COPY build.rs ./
COPY .git* ./
COPY src ./src
COPY migrations ./migrations
COPY --from=frontend-builder /app/dist ./web/dist
# Build web app with embedded assets
RUN cargo build --release --bin banner
# Strip the binary to reduce size
RUN strip target/release/banner
# --- Runtime Stage ---
FROM debian:12-slim
ARG APP=/usr/src/app
ARG APP_USER=appuser
ARG UID=1000
ARG GID=1000
# Install runtime dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
tzdata \
wget \
&& rm -rf /var/lib/apt/lists/*
ARG TZ=Etc/UTC
ENV TZ=${TZ}
# Create user with specific UID/GID
RUN addgroup --gid $GID $APP_USER \
&& adduser --uid $UID --disabled-password --gecos "" --ingroup $APP_USER $APP_USER \
&& mkdir -p ${APP}
# Copy application binary
COPY --from=builder --chown=$APP_USER:$APP_USER /app/target/release/banner ${APP}/banner
# Set proper permissions
RUN chmod +x ${APP}/banner
USER $APP_USER
WORKDIR ${APP}
# Build-time arg for PORT, default to 8000
ARG PORT=8000
# Runtime environment var for PORT, default to build-time arg
ENV PORT=${PORT}
EXPOSE ${PORT}
# Add health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:${PORT}/health || exit 1
# Can be explicitly overriden with different hosts & ports
ENV HOSTS=0.0.0.0,[::]
# Implicitly uses PORT environment variable
# temporary: running without 'scraper' service
CMD ["sh", "-c", "exec ./banner --services web,bot"]

79
Justfile Normal file
View File

@@ -0,0 +1,79 @@
default_services := "bot,web,scraper"
default:
just --list
# Run all checks (format, clippy, tests, lint)
check:
cargo fmt --all -- --check
cargo clippy --all-features -- --deny warnings
cargo nextest run
bun run --cwd web typecheck
bun run --cwd web lint
# Format all Rust and TypeScript code
format:
cargo fmt --all
bun run --cwd web format
# Check formatting without modifying (CI-friendly)
format-check:
cargo fmt --all -- --check
bun run --cwd web format:check
# Start PostgreSQL in Docker and update .env with connection string
db:
#!/usr/bin/env bash
set -euo pipefail
# Find available port
PORT=$(shuf -i 49152-65535 -n 1)
while ss -tlnp 2>/dev/null | grep -q ":$PORT "; do
PORT=$(shuf -i 49152-65535 -n 1)
done
# Start PostgreSQL container
docker run -d \
--name banner-postgres \
-e POSTGRES_PASSWORD=banner \
-e POSTGRES_USER=banner \
-e POSTGRES_DB=banner \
-p "$PORT:5432" \
postgres:17-alpine
# Update .env file
DB_URL="postgresql://banner:banner@localhost:$PORT/banner"
if [ -f .env ]; then
sed -i.bak "s|^DATABASE_URL=.*|DATABASE_URL=$DB_URL|" .env
else
echo "DATABASE_URL=$DB_URL" > .env
fi
echo "PostgreSQL started on port $PORT"
echo "DATABASE_URL=$DB_URL"
echo "Run: sqlx migrate run"
# Auto-reloading frontend server
frontend:
bun run --cwd web dev
# Production build of frontend
build-frontend:
bun run --cwd web build
# Auto-reloading backend server
backend *ARGS:
bacon --headless run -- -- {{ARGS}}
# Production build
build:
bun run --cwd web build
cargo build --release --bin banner
# Run auto-reloading development build with release characteristics
dev-build *ARGS='--services web --tracing pretty': build-frontend
bacon --headless run -- --profile dev-release -- {{ARGS}}
# Auto-reloading development build for both frontend and backend
[parallel]
dev *ARGS='--services web,bot': frontend (backend ARGS)

142
README.md
View File

@@ -1,125 +1,51 @@
# banner
A discord bot for executing queries & searches on the Ellucian Banner instance hosting all of UTSA's class data.
A complex multi-service system providing a Discord bot and browser-based interface to UTSA's course data.
## Feature Wishlist
## Services
- Commands
- ICS Download (get a ICS download of your classes with location & timing perfectly - set for every class you're in)
- Classes Now (find classes happening)
- Autocomplete
- Class Title
- Course Number
- Term/Part of Term
- Professor
- Attribute
- Component Pagination
- RateMyProfessor Integration (Linked/Embedded)
- Smart term selection (i.e. Summer 2024 will be selected automatically when opened)
- Rate Limiting (bursting with global/user limits)
- DMs Integration (allow usage of the bot in DMs)
- Class Change Notifications (get notified when details about a class change)
- Multi-term Querying (currently the backend for searching is kinda weird)
- Full Autocomplete for Every Search Option
- Metrics, Log Query, Privileged Error Feedback
- Search for Classes
- Major, Professor, Location, Name, Time of Day
- Subscribe to Classes
- Availability (seat, pre-seat)
- Waitlist Movement
- Detail Changes (meta, time, location, seats, professor)
- `time` Start, End, Days of Week
- `seats` Any change in seat/waitlist data
- `meta`
- Lookup via Course Reference Number (CRN)
- Smart Time of Day Handling
- "2 PM" -> Start within 2:00 PM to 2:59 PM
- "2-3 PM" -> Start within 2:00 PM to 3:59 PM
- "ends by 2 PM" -> Ends within 12:00 AM to 2:00 PM
- "after 2 PM" -> Start within 2:01 PM to 11:59 PM
- "before 2 PM" -> Ends within 12:00 AM to 1:59 PM
- Get By Section Command
- CS 4393 001 =>
- Will require SQL to be able to search for a class by its section number
The application consists of three modular services that can be run independently or together:
## Analysis Required
- Discord Bot ([`bot`][src-bot])
Some of the features and architecture of Ellucian's Banner system are not clear.
The follow features, JSON, and more require validation & analysis:
- Primary interface for course monitoring and data queries
- Built with [Serenity][serenity] and [Poise][poise] frameworks for robust command handling
- Uses slash commands with comprehensive error handling and logging
- Struct Nullability
- Much of the responses provided by Ellucian contain nulls, and most of them are uncertain as to when and why they're null.
- Analysis must be conducted to be sure of when to use a string and when it should nillable (pointer).
- Multiple Professors / Primary Indicator
- Multiple Meeting Times
- Meeting Schedule Types
- AFF vs AIN vs AHB etc.
- Do CRNs repeat between years?
- Check whether partOfTerm is always filled in, and it's meaning for various class results.
- Check which API calls are affected by change in term/sessionID term select
- SessionIDs
- How long does a session ID work?
- Do I really require a separate one per term?
- How many can I activate, are there any restrictions?
- How should session IDs be checked as 'invalid'?
- What action(s) keep a session ID 'active', if any?
- Are there any courses with multiple meeting times?
- Google Calendar link generation, as an alternative to ICS file generation
- Web Server ([`web`][src-web])
## Change Identification
- [Axum][axum]-based server with Vite/React-based frontend
- [Embeds static assets][rust-embed] at compile time with E-Tags & Cache-Control headers
- Important attributes of a class will be parsed on both the old and new data.
- These attributes will be compared and given identifiers that can be subscribed to.
- When a user subscribes to one of these identifiers, any changes identified will be sent to the user.
- Scraper ([`scraper`][src-scraper])
## Real-time Suggestions
- Intelligent data collection system with priority-based queuing inside PostgreSQL via [`sqlx`][sqlx]
- Rate-limited scraping with burst handling to respect UTSA's systems
- Handles course data updates, availability changes, and metadata synchronization
Various commands arguments have the ability to have suggestions appear.
## Quick Start
- They must be fast. As ephemeral suggestions that are only relevant for seconds or less, they need to be delivered in less than a second.
- They need to be easy to acquire. With as many commands & arguments to search as I do, it is paramount that the API be easy to understand & use.
- It cannot be complicated. I only have so much time to develop this.
- It does not need to be persistent. Since the data is scraped and rolled periodically from the Banner system, the data used will be deleted and re-requested occasionally.
```bash
bun install --cwd web # Install frontend dependencies
cargo build # Build the backend
For these reasons, I believe SQLite to be the ideal place for this data to be stored.
It is exceptionally fast, works well in-memory, and is less complicated compared to most other solutions.
just dev # Runs auto-reloading dev build
just dev --services bot,web # Runs auto-reloading dev build, running only the bot and web services
just dev-build # Development build with release characteristics (frontend is embedded, non-auto-reloading)
- Only required data about the class will be stored, along with the JSON-encoded string.
- For now, this would only be the CRN (and possibly the Term).
- Potentially, a binary encoding could be used for performance, but it is unlikely to be better.
- Database dumping into R2 would be good to ensure that over-scraping of the Banner system does not occur.
- Upon a safe close requested
- Must be done quickly (<8 seconds)
- Every 30 minutes, if any scraping ocurred.
- May cause locking of commands.
just build # Production build that embeds assets
```
## Scraping
## Documentation
In order to keep the in-memory database of the bot up-to-date with the Banner system, the API must be scraped.
Scraping will be separated by major to allow for priority majors (namely, Computer Science) to be scraped more often compared to others.
This will lower the overall load on the Banner system while ensuring that data presented by the app is still relevant.
Comprehensive documentation is available in the [`docs/`][documentation] folder.
For now, all majors will be scraped fully every 4 hours with at least 5 minutes between each one.
- On startup, priority majors will be scraped first (if required).
- Other majors will be scraped in arbitrary order (if required).
- Scrape timing will be stored in Redis.
- CRNs will be the Primary Key within SQLite
- If CRNs are duplicated between terms, then the primary key will be (CRN, Term)
Considerations
- Change in metadata should decrease the interval
- The number of courses scraped should change the interval (2 hours per 500 courses involved)
## Rate Limiting, Costs & Bursting
Ideally, this application would implement dynamic rate limiting to ensure overload on the server does not occur.
Better, it would also ensure that priority requests (commands) are dispatched faster than background processes (scraping), while making sure different requests are weighted differently.
For example, a recent scrape of 350 classes should be weighted 5x more than a search for 8 classes by a user.
Still, even if the cap does not normally allow for this request to be processed immediately, the small user search should proceed with a small bursting cap.
The requirements to this hypothetical system would be:
- Conditional Bursting: background processes or other requests deemed "low priority" are not allowed to use bursting.
- Arbitrary Costs: rate limiting is considered in the form of the request size/speed more or less, such that small simple requests can be made more frequently, unlike large requests.
[documentation]: docs/README.md
[src-bot]: src/bot
[src-web]: src/web
[src-scraper]: src/scraper
[serenity]: https://github.com/serenity-rs/serenity
[poise]: https://github.com/serenity-rs/poise
[axum]: https://github.com/tokio-rs/axum
[rust-embed]: https://lib.rs/crates/rust-embed
[sqlx]: https://github.com/launchbadge/sqlx

52
bacon.toml Normal file
View File

@@ -0,0 +1,52 @@
# This is a configuration file for the bacon tool
#
# Complete help on configuration: https://dystroy.org/bacon/config/
#
# You may check the current default at
# https://github.com/Canop/bacon/blob/main/defaults/default-bacon.toml
default_job = "check"
env.CARGO_TERM_COLOR = "always"
[jobs.check]
command = ["cargo", "check", "--all-targets"]
need_stdout = false
[jobs.clippy]
command = ["cargo", "clippy", "--all-targets"]
need_stdout = false
[jobs.test]
command = [
"cargo", "nextest", "run",
]
need_stdout = true
analyzer = "nextest"
[jobs.run]
command = [
"cargo", "run",
]
need_stdout = true
allow_warnings = true
background = false
on_change_strategy = "kill_then_restart"
# kill = ["pkill", "-TERM", "-P"]'
[jobs.dev]
command = [
"just", "dev"
]
need_stdout = true
allow_warnings = true
background = false
on_change_strategy = "kill_then_restart"
# You may define here keybindings that would be specific to
# a project, for example a shortcut to launch a specific job.
# Shortcuts to internal functions (scrolling, toggling, etc.)
# should go in your personal global prefs.toml file instead.
[keybindings]
c = "job:clippy" # comment this to have 'c' run clippy on only the default target
shift-c = "job:check"
d = "job:dev"

36
build.rs Normal file
View File

@@ -0,0 +1,36 @@
use std::process::Command;
fn main() {
// Try to get Git commit hash from Railway environment variable first
let git_hash = std::env::var("RAILWAY_GIT_COMMIT_SHA").unwrap_or_else(|_| {
// Fallback to git command if not on Railway
let output = Command::new("git").args(["rev-parse", "HEAD"]).output();
match output {
Ok(output) => {
if output.status.success() {
String::from_utf8_lossy(&output.stdout).trim().to_string()
} else {
"unknown".to_string()
}
}
Err(_) => "unknown".to_string(),
}
});
// Get the short hash (first 7 characters)
let short_hash = if git_hash != "unknown" && git_hash.len() >= 7 {
git_hash[..7].to_string()
} else {
git_hash.clone()
};
// Set the environment variables that will be available at compile time
println!("cargo:rustc-env=GIT_COMMIT_HASH={}", git_hash);
println!("cargo:rustc-env=GIT_COMMIT_SHORT={}", short_hash);
// Rebuild if the Git commit changes (only works when .git directory is available)
if std::path::Path::new(".git/HEAD").exists() {
println!("cargo:rerun-if-changed=.git/HEAD");
println!("cargo:rerun-if-changed=.git/refs/heads");
}
}

94
docs/ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,94 @@
# Architecture
## System Overview
The Banner project is built as a multi-service application with the following components:
- **Discord Bot Service**: Handles Discord interactions and commands
- **Web Service**: Serves the React frontend and provides API endpoints
- **Scraper Service**: Background data collection and synchronization
- **Database Layer**: PostgreSQL for persistent storage
## Technical Analysis
### Banner System Integration
Some of the features and architecture of Ellucian's Banner system are not clear.
The following features, JSON, and more require validation & analysis:
- Struct Nullability
- Much of the responses provided by Ellucian contain nulls, and most of them are uncertain as to when and why they're null.
- Analysis must be conducted to be sure of when to use a string and when it should nillable (pointer).
- Multiple Professors / Primary Indicator
- Multiple Meeting Times
- Meeting Schedule Types
- AFF vs AIN vs AHB etc.
- Do CRNs repeat between years?
- Check whether partOfTerm is always filled in, and it's meaning for various class results.
- Check which API calls are affected by change in term/sessionID term select
- SessionIDs
- How long does a session ID work?
- Do I really require a separate one per term?
- How many can I activate, are there any restrictions?
- How should session IDs be checked as 'invalid'?
- What action(s) keep a session ID 'active', if any?
- Are there any courses with multiple meeting times?
- Google Calendar link generation, as an alternative to ICS file generation
## Change Identification
- Important attributes of a class will be parsed on both the old and new data.
- These attributes will be compared and given identifiers that can be subscribed to.
- When a user subscribes to one of these identifiers, any changes identified will be sent to the user.
## Real-time Suggestions
Various commands arguments have the ability to have suggestions appear.
- They must be fast. As ephemeral suggestions that are only relevant for seconds or less, they need to be delivered in less than a second.
- They need to be easy to acquire. With as many commands & arguments to search as I do, it is paramount that the API be easy to understand & use.
- It cannot be complicated. I only have so much time to develop this.
- It does not need to be persistent. Since the data is scraped and rolled periodically from the Banner system, the data used will be deleted and re-requested occasionally.
For these reasons, I believe PostgreSQL to be the ideal place for this data to be stored.
It is exceptionally fast, works well in-memory, and is less complicated compared to most other solutions.
- Only required data about the class will be stored, along with the JSON-encoded string.
- For now, this would only be the CRN (and possibly the Term).
- Potentially, a binary encoding could be used for performance, but it is unlikely to be better.
- Database dumping into R2 would be good to ensure that over-scraping of the Banner system does not occur.
- Upon a safe close requested
- Must be done quickly (<8 seconds)
- Every 30 minutes, if any scraping ocurred.
- May cause locking of commands.
## Scraping System
In order to keep the in-memory database of the bot up-to-date with the Banner system, the API must be scraped.
Scraping will be separated by major to allow for priority majors (namely, Computer Science) to be scraped more often compared to others.
This will lower the overall load on the Banner system while ensuring that data presented by the app is still relevant.
For now, all majors will be scraped fully every 4 hours with at least 5 minutes between each one.
- On startup, priority majors will be scraped first (if required).
- Other majors will be scraped in arbitrary order (if required).
- Scrape timing will be stored in database.
- CRNs will be the Primary Key within database
- If CRNs are duplicated between terms, then the primary key will be (CRN, Term)
Considerations
- Change in metadata should decrease the interval
- The number of courses scraped should change the interval (2 hours per 500 courses involved)
## Rate Limiting, Costs & Bursting
Ideally, this application would implement dynamic rate limiting to ensure overload on the server does not occur.
Better, it would also ensure that priority requests (commands) are dispatched faster than background processes (scraping), while making sure different requests are weighted differently.
For example, a recent scrape of 350 classes should be weighted 5x more than a search for 8 classes by a user.
Still, even if the cap does not normally allow for this request to be processed immediately, the small user search should proceed with a small bursting cap.
The requirements to this hypothetical system would be:
- Conditional Bursting: background processes or other requests deemed "low priority" are not allowed to use bursting.
- Arbitrary Costs: rate limiting is considered in the form of the request size/speed more or less, such that small simple requests can be made more frequently, unlike large requests.

View File

@@ -1,11 +1,17 @@
# Sessions
# Banner
All notes on the internal workings of the Banner system by Ellucian.
## Sessions
All notes on the internal workings of Sessions in the Banner system.
- Sessions are generated on demand with a random string of characters.
- The format `{5 random characters}{milliseconds since epoch}`
- Example: ``
- Sessions are invalidated after 30 minutes, but may change.
- This delay can be found in the original HTML returned, find `meta[name="maxInactiveInterval"]` and read the `content` attribute.
- This is read at runtime by the javascript on initialization.
- This is read at runtime (in the browser, by javascript) on initialization.
- Multiple timers exist, one is for the Inactivity Timer.
- A dialog will appear asking the user to continue their session.
- If they click the button, the session will be extended via the keepAliveURL (see `meta[name="keepAliveURL"]`).

58
docs/FEATURES.md Normal file
View File

@@ -0,0 +1,58 @@
# Features
## Current Features
### Discord Bot Commands
- **search** - Search for courses with various filters (title, course code, keywords)
- **terms** - List available terms or search for a specific term
- **time** - Get meeting times for a specific course (CRN)
- **ics** - Generate ICS calendar file for a course with holiday exclusions
- **gcal** - Generate Google Calendar link for a course
### Data Pipeline
- Intelligent scraping system with priority queues
- Rate limiting and burst handling
- Background data synchronization
## Feature Wishlist
### Commands
- ICS Download (get a ICS download of your classes with location & timing perfectly - set for every class you're in)
- Classes Now (find classes happening)
- Autocomplete
- Class Title
- Course Number
- Term/Part of Term
- Professor
- Attribute
- Component Pagination
- RateMyProfessor Integration (Linked/Embedded)
- Smart term selection (i.e. Summer 2024 will be selected automatically when opened)
- Rate Limiting (bursting with global/user limits)
- DMs Integration (allow usage of the bot in DMs)
- Class Change Notifications (get notified when details about a class change)
- Multi-term Querying (currently the backend for searching is kinda weird)
- Full Autocomplete for Every Search Option
- Metrics, Log Query, Privileged Error Feedback
- Search for Classes
- Major, Professor, Location, Name, Time of Day
- Subscribe to Classes
- Availability (seat, pre-seat)
- Waitlist Movement
- Detail Changes (meta, time, location, seats, professor)
- `time` Start, End, Days of Week
- `seats` Any change in seat/waitlist data
- `meta`
- Lookup via Course Reference Number (CRN)
- Smart Time of Day Handling
- "2 PM" -> Start within 2:00 PM to 2:59 PM
- "2-3 PM" -> Start within 2:00 PM to 3:59 PM
- "ends by 2 PM" -> Ends within 12:00 AM to 2:00 PM
- "after 2 PM" -> Start within 2:01 PM to 11:59 PM
- "before 2 PM" -> Ends within 12:00 AM to 1:59 PM
- Get By Section Command
- CS 4393 001 =>
- Will require SQL to be able to search for a class by its section number

42
docs/README.md Normal file
View File

@@ -0,0 +1,42 @@
# Documentation
This folder contains detailed documentation for the Banner project. This file acts as the index.
## Files
- [`FEATURES.md`](FEATURES.md) - Current features, implemented functionality, and future roadmap
- [`BANNER.md`](BANNER.md) - General API documentation on the Banner system
- [`ARCHITECTURE.md`](ARCHITECTURE.md) - Technical implementation details, system design, and analysis
## Samples
The `samples/` folder contains real Banner API response examples:
- `search/` - Course search API responses with various filters
- [`searchResults.json`](samples/search/searchResults.json)
- [`searchResults_500.json`](samples/search/searchResults_500.json)
- [`searchResults_CS500.json`](samples/search/searchResults_CS500.json)
- [`searchResults_malware.json`](samples/search/searchResults_malware.json)
- `meta/` - Metadata API responses (terms, subjects, instructors, etc.)
- [`get_attribute.json`](samples/meta/get_attribute.json)
- [`get_campus.json`](samples/meta/get_campus.json)
- [`get_instructionalMethod.json`](samples/meta/get_instructionalMethod.json)
- [`get_instructor.json`](samples/meta/get_instructor.json)
- [`get_partOfTerm.json`](samples/meta/get_partOfTerm.json)
- [`get_subject.json`](samples/meta/get_subject.json)
- [`getTerms.json`](samples/meta/getTerms.json)
- `course/` - Course detail API responses (HTML and JSON)
- [`getFacultyMeetingTimes.json`](samples/course/getFacultyMeetingTimes.json)
- [`getClassDetails.html`](samples/course/getClassDetails.html)
- [`getCorequisites.html`](samples/course/getCorequisites.html)
- [`getCourseDescription.html`](samples/course/getCourseDescription.html)
- [`getEnrollmentInfo.html`](samples/course/getEnrollmentInfo.html)
- [`getFees.html`](samples/course/getFees.html)
- [`getLinkedSections.html`](samples/course/getLinkedSections.html)
- [`getRestrictions.html`](samples/course/getRestrictions.html)
- [`getSectionAttributes.html`](samples/course/getSectionAttributes.html)
- [`getSectionBookstoreDetails.html`](samples/course/getSectionBookstoreDetails.html)
- [`getSectionPrerequisites.html`](samples/course/getSectionPrerequisites.html)
- [`getXlistSections.html`](samples/course/getXlistSections.html)
These samples are used for development, testing, and understanding the Banner API structure.

View File

@@ -0,0 +1,56 @@
-- Drop all old tables
DROP TABLE IF EXISTS scrape_jobs;
DROP TABLE IF EXISTS course_metrics;
DROP TABLE IF EXISTS course_audits;
DROP TABLE IF EXISTS courses;
-- Enums for scrape_jobs
CREATE TYPE scrape_priority AS ENUM ('Low', 'Medium', 'High', 'Critical');
CREATE TYPE target_type AS ENUM ('Subject', 'CourseRange', 'CrnList', 'SingleCrn');
-- Main course data table
CREATE TABLE courses (
id SERIAL PRIMARY KEY,
crn VARCHAR NOT NULL,
subject VARCHAR NOT NULL,
course_number VARCHAR NOT NULL,
title VARCHAR NOT NULL,
term_code VARCHAR NOT NULL,
enrollment INTEGER NOT NULL,
max_enrollment INTEGER NOT NULL,
wait_count INTEGER NOT NULL,
wait_capacity INTEGER NOT NULL,
last_scraped_at TIMESTAMPTZ NOT NULL,
UNIQUE(crn, term_code)
);
-- Time-series data for course enrollment
CREATE TABLE course_metrics (
id SERIAL PRIMARY KEY,
course_id INTEGER NOT NULL REFERENCES courses(id) ON DELETE CASCADE,
timestamp TIMESTAMPTZ NOT NULL,
enrollment INTEGER NOT NULL,
wait_count INTEGER NOT NULL,
seats_available INTEGER NOT NULL
);
-- Audit trail for changes to course data
CREATE TABLE course_audits (
id SERIAL PRIMARY KEY,
course_id INTEGER NOT NULL REFERENCES courses(id) ON DELETE CASCADE,
timestamp TIMESTAMPTZ NOT NULL,
field_changed VARCHAR NOT NULL,
old_value TEXT NOT NULL,
new_value TEXT NOT NULL
);
-- Job queue for the scraper
CREATE TABLE scrape_jobs (
id SERIAL PRIMARY KEY,
target_type target_type NOT NULL,
target_payload JSONB NOT NULL,
priority scrape_priority NOT NULL,
execute_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
locked_at TIMESTAMPTZ
);

View File

@@ -0,0 +1,3 @@
-- Add retry tracking columns to scrape_jobs table
ALTER TABLE scrape_jobs ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0 CHECK (retry_count >= 0);
ALTER TABLE scrape_jobs ADD COLUMN max_retries INTEGER NOT NULL DEFAULT 5 CHECK (max_retries >= 0);

View File

@@ -0,0 +1,45 @@
-- Performance optimization indexes
-- Index for term-based queries (most common access pattern)
CREATE INDEX IF NOT EXISTS idx_courses_term_code ON courses(term_code);
-- Index for subject-based filtering
CREATE INDEX IF NOT EXISTS idx_courses_subject ON courses(subject);
-- Composite index for subject + term queries
CREATE INDEX IF NOT EXISTS idx_courses_subject_term ON courses(subject, term_code);
-- Index for course number lookups
CREATE INDEX IF NOT EXISTS idx_courses_course_number ON courses(course_number);
-- Index for last scraped timestamp (useful for finding stale data)
CREATE INDEX IF NOT EXISTS idx_courses_last_scraped ON courses(last_scraped_at);
-- Index for course metrics time-series queries
-- BRIN index is optimal for time-series data
CREATE INDEX IF NOT EXISTS idx_course_metrics_timestamp ON course_metrics USING BRIN(timestamp);
-- B-tree index for specific course metric lookups
CREATE INDEX IF NOT EXISTS idx_course_metrics_course_timestamp
ON course_metrics(course_id, timestamp DESC);
-- Partial index for pending scrape jobs (only unlocked jobs)
CREATE INDEX IF NOT EXISTS idx_scrape_jobs_pending
ON scrape_jobs(execute_at ASC)
WHERE locked_at IS NULL;
-- Index for high-priority job processing
CREATE INDEX IF NOT EXISTS idx_scrape_jobs_priority_pending
ON scrape_jobs(priority DESC, execute_at ASC)
WHERE locked_at IS NULL;
-- Index for retry tracking
CREATE INDEX IF NOT EXISTS idx_scrape_jobs_retry_count
ON scrape_jobs(retry_count)
WHERE retry_count > 0 AND locked_at IS NULL;
-- Analyze tables to update statistics
ANALYZE courses;
ANALYZE course_metrics;
ANALYZE course_audits;
ANALYZE scrape_jobs;

View File

@@ -0,0 +1,53 @@
-- Index Optimization Follow-up Migration
-- Reason: Redundant with composite index idx_courses_subject_term
DROP INDEX IF EXISTS idx_courses_subject;
-- Remove: idx_scrape_jobs_retry_count
DROP INDEX IF EXISTS idx_scrape_jobs_retry_count;
-- Purpose: Optimize the scheduler's frequent query (runs every 60 seconds)
CREATE INDEX IF NOT EXISTS idx_scrape_jobs_scheduler_lookup
ON scrape_jobs(target_type, target_payload)
WHERE locked_at IS NULL;
-- Note: We use (target_type, target_payload) instead of including locked_at
-- in the index columns because:
-- 1. The WHERE clause filters locked_at IS NULL (partial index optimization)
-- 2. target_payload is JSONB and already large; keeping it as an indexed column
-- allows PostgreSQL to use index-only scans for the SELECT target_payload query
-- 3. This design minimizes index size while maximizing query performance
-- Purpose: Enable efficient audit trail queries by course
CREATE INDEX IF NOT EXISTS idx_course_audits_course_timestamp
ON course_audits(course_id, timestamp DESC);
-- Purpose: Enable queries like "Show all changes in the last 24 hours"
CREATE INDEX IF NOT EXISTS idx_course_audits_timestamp
ON course_audits(timestamp DESC);
-- The BRIN index on course_metrics(timestamp) assumes data is inserted in
-- chronological order. BRIN indexes are only effective when data is physically
-- ordered on disk. If you perform:
-- - Backfills of historical data
-- - Out-of-order inserts
-- - Frequent UPDATEs that move rows
--
-- Then the BRIN index effectiveness will degrade. Monitor with:
-- SELECT * FROM brin_page_items(get_raw_page('idx_course_metrics_timestamp', 1));
--
-- If you see poor selectivity, consider:
-- 1. REINDEX to rebuild after bulk loads
-- 2. Switch to B-tree if inserts are not time-ordered
-- 3. Use CLUSTER to physically reorder the table (requires downtime)
COMMENT ON INDEX idx_course_metrics_timestamp IS
'BRIN index - requires chronologically ordered inserts for efficiency. Monitor selectivity.';
-- Update statistics for query planner
ANALYZE courses;
ANALYZE course_metrics;
ANALYZE course_audits;
ANALYZE scrape_jobs;

168
src/app.rs Normal file
View File

@@ -0,0 +1,168 @@
use crate::banner::BannerApi;
use crate::cli::ServiceName;
use crate::config::Config;
use crate::scraper::ScraperService;
use crate::services::bot::BotService;
use crate::services::manager::ServiceManager;
use crate::services::web::WebService;
use crate::state::AppState;
use crate::web::routes::BannerState;
use figment::value::UncasedStr;
use figment::{Figment, providers::Env};
use sqlx::postgres::PgPoolOptions;
use std::process::ExitCode;
use std::sync::Arc;
use std::time::Duration;
use tracing::{error, info};
/// Main application struct containing all necessary components
pub struct App {
config: Config,
db_pool: sqlx::PgPool,
banner_api: Arc<BannerApi>,
app_state: AppState,
banner_state: BannerState,
service_manager: ServiceManager,
}
impl App {
/// Create a new App instance with all necessary components initialized
pub async fn new() -> Result<Self, anyhow::Error> {
// Load configuration
let config: Config = Figment::new()
.merge(Env::raw().map(|k| {
if k == UncasedStr::new("RAILWAY_DEPLOYMENT_DRAINING_SECONDS") {
"SHUTDOWN_TIMEOUT".into()
} else {
k.into()
}
}))
.extract()
.expect("Failed to load config");
// Check if the database URL is via private networking
let is_private = config.database_url.contains("railway.internal");
let slow_threshold = Duration::from_millis(if is_private { 200 } else { 500 });
// Create database connection pool
let db_pool = PgPoolOptions::new()
.min_connections(0)
.max_connections(4)
.acquire_slow_threshold(slow_threshold)
.acquire_timeout(Duration::from_secs(4))
.idle_timeout(Duration::from_secs(60 * 2))
.max_lifetime(Duration::from_secs(60 * 30))
.connect(&config.database_url)
.await
.expect("Failed to create database pool");
info!(
is_private = is_private,
slow_threshold = format!("{:.2?}", slow_threshold),
"database pool established"
);
// Run database migrations
info!("Running database migrations...");
sqlx::migrate!("./migrations")
.run(&db_pool)
.await
.expect("Failed to run database migrations");
info!("Database migrations completed successfully");
// Create BannerApi and AppState
let banner_api = BannerApi::new_with_config(
config.banner_base_url.clone(),
config.rate_limiting.clone().into(),
)
.expect("Failed to create BannerApi");
let banner_api_arc = Arc::new(banner_api);
let app_state = AppState::new(banner_api_arc.clone(), db_pool.clone());
// Create BannerState for web service
let banner_state = BannerState {};
Ok(App {
config,
db_pool,
banner_api: banner_api_arc,
app_state,
banner_state,
service_manager: ServiceManager::new(),
})
}
/// Setup and register services based on enabled service list
pub fn setup_services(&mut self, services: &[ServiceName]) -> Result<(), anyhow::Error> {
// Register enabled services with the manager
if services.contains(&ServiceName::Web) {
let web_service =
Box::new(WebService::new(self.config.port, self.banner_state.clone()));
self.service_manager
.register_service(ServiceName::Web.as_str(), web_service);
}
if services.contains(&ServiceName::Scraper) {
let scraper_service = Box::new(ScraperService::new(
self.db_pool.clone(),
self.banner_api.clone(),
));
self.service_manager
.register_service(ServiceName::Scraper.as_str(), scraper_service);
}
// Check if any services are enabled
if !self.service_manager.has_services() && !services.contains(&ServiceName::Bot) {
error!("No services enabled. Cannot start application.");
return Err(anyhow::anyhow!("No services enabled"));
}
Ok(())
}
/// Setup bot service if enabled
pub async fn setup_bot_service(&mut self) -> Result<(), anyhow::Error> {
use std::sync::Arc;
use tokio::sync::{Mutex, broadcast};
// Create shutdown channel for status update task
let (status_shutdown_tx, status_shutdown_rx) = broadcast::channel(1);
let status_task_handle = Arc::new(Mutex::new(None));
let client = BotService::create_client(
&self.config,
self.app_state.clone(),
status_task_handle.clone(),
status_shutdown_rx,
)
.await
.expect("Failed to create Discord client");
let bot_service = Box::new(BotService::new(
client,
status_task_handle,
status_shutdown_tx,
));
self.service_manager
.register_service(ServiceName::Bot.as_str(), bot_service);
Ok(())
}
/// Start all registered services
pub fn start_services(&mut self) {
self.service_manager.spawn_all();
}
/// Run the application and handle shutdown signals
pub async fn run(self) -> ExitCode {
use crate::signals::handle_shutdown_signals;
handle_shutdown_signals(self.service_manager, self.config.shutdown_timeout).await
}
/// Get a reference to the configuration
pub fn config(&self) -> &Config {
&self.config
}
}

View File

@@ -1,48 +0,0 @@
//! Application state shared across components (bot, web, scheduler).
use crate::banner::BannerApi;
use crate::banner::Course;
use anyhow::Result;
use redis::AsyncCommands;
use redis::Client;
use serde_json;
#[derive(Clone, Debug)]
pub struct AppState {
pub banner_api: std::sync::Arc<BannerApi>,
pub redis: std::sync::Arc<Client>,
}
impl AppState {
pub fn new(
banner_api: BannerApi,
redis_url: &str,
) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
let redis_client = Client::open(redis_url)?;
Ok(Self {
banner_api: std::sync::Arc::new(banner_api),
redis: std::sync::Arc::new(redis_client),
})
}
/// Get a course by CRN with Redis cache fallback to Banner API
pub async fn get_course_or_fetch(&self, term: &str, crn: &str) -> Result<Course> {
let mut conn = self.redis.get_multiplexed_async_connection().await?;
let key = format!("class:{}", crn);
if let Some(serialized) = conn.get::<_, Option<String>>(&key).await? {
let course: Course = serde_json::from_str(&serialized)?;
return Ok(course);
}
// Fallback: fetch from Banner API
if let Some(course) = self.banner_api.get_course_by_crn(term, crn).await? {
let serialized = serde_json::to_string(&course)?;
let _: () = conn.set(&key, serialized).await?;
return Ok(course);
}
Err(anyhow::anyhow!("Course not found for CRN {}", crn))
}
}

View File

@@ -1,81 +1,233 @@
//! Main Banner API client implementation.
use crate::banner::{SessionManager, models::*, query::SearchQuery};
use anyhow::{Context, Result};
use axum::http::HeaderValue;
use reqwest::Client;
use serde_json;
use std::{
collections::{HashMap, VecDeque},
sync::{Arc, Mutex},
time::Instant,
};
// use tracing::debug;
use crate::banner::{
BannerSession, SessionPool, create_shared_rate_limiter,
errors::BannerApiError,
json::parse_json_with_context,
middleware::TransparentMiddleware,
models::*,
nonce,
query::SearchQuery,
rate_limit_middleware::RateLimitMiddleware,
rate_limiter::{RateLimitConfig, SharedRateLimiter},
util::user_agent,
};
use anyhow::{Context, Result, anyhow};
use cookie::Cookie;
use dashmap::DashMap;
use http::HeaderValue;
use reqwest::{Client, Request, Response};
use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
use serde_json;
use tl;
use tracing::{Level, Metadata, Span, debug, error, field::ValueSet, info, span, trace, warn};
/// Main Banner API client.
#[derive(Debug)]
pub struct BannerApi {
session_manager: SessionManager,
client: Client,
pub sessions: SessionPool,
http: ClientWithMiddleware,
base_url: String,
}
#[allow(dead_code)]
impl BannerApi {
/// Creates a new Banner API client.
pub fn new(base_url: String) -> Result<Self> {
let client = Client::builder()
.cookie_store(true)
.user_agent(user_agent())
.tcp_keepalive(Some(std::time::Duration::from_secs(60 * 5)))
.read_timeout(std::time::Duration::from_secs(10))
.connect_timeout(std::time::Duration::from_secs(10))
.timeout(std::time::Duration::from_secs(30))
.build()
.context("Failed to create HTTP client")?;
Self::new_with_config(base_url, RateLimitConfig::default())
}
let session_manager = SessionManager::new(base_url.clone(), client.clone());
/// Creates a new Banner API client with custom rate limiting configuration.
pub fn new_with_config(base_url: String, rate_limit_config: RateLimitConfig) -> Result<Self> {
let rate_limiter = create_shared_rate_limiter(Some(rate_limit_config));
let http = ClientBuilder::new(
Client::builder()
.cookie_store(false)
.user_agent(user_agent())
.tcp_keepalive(Some(std::time::Duration::from_secs(60 * 5)))
.read_timeout(std::time::Duration::from_secs(10))
.connect_timeout(std::time::Duration::from_secs(10))
.timeout(std::time::Duration::from_secs(30))
.build()
.context("Failed to create HTTP client")?,
)
.with(TransparentMiddleware)
.with(RateLimitMiddleware::new(rate_limiter.clone()))
.build();
Ok(Self {
session_manager,
client,
sessions: SessionPool::new(http.clone(), base_url.clone()),
http,
base_url,
})
}
/// Sets up the API client by initializing session cookies.
pub async fn setup(&self) -> Result<()> {
self.session_manager.setup().await
/// Validates offset parameter for search methods.
fn validate_offset(offset: i32) -> Result<()> {
if offset <= 0 {
Err(anyhow::anyhow!("Offset must be greater than 0"))
} else {
Ok(())
}
}
/// Retrieves a list of terms from the Banner API.
pub async fn get_terms(
/// Builds common search parameters for list endpoints.
fn build_list_params(
&self,
search: &str,
page: i32,
term: &str,
offset: i32,
max_results: i32,
) -> Result<Vec<BannerTerm>> {
if page <= 0 {
return Err(anyhow::anyhow!("Page must be greater than 0"));
}
session_id: &str,
) -> Vec<(&str, String)> {
vec![
("searchTerm", search.to_string()),
("term", term.to_string()),
("offset", offset.to_string()),
("max", max_results.to_string()),
("uniqueSessionId", session_id.to_string()),
("_", nonce()),
]
}
let url = format!("{}/classSearch/getTerms", self.base_url);
let params = [
("searchTerm", search),
("offset", &page.to_string()),
("max", &max_results.to_string()),
("_", &timestamp_nonce()),
];
/// Makes a GET request to a list endpoint and parses JSON response.
async fn get_list_endpoint<T>(
&self,
endpoint: &str,
search: &str,
term: &str,
offset: i32,
max_results: i32,
) -> Result<Vec<T>>
where
T: for<'de> serde::Deserialize<'de>,
{
Self::validate_offset(offset)?;
let session = self.sessions.acquire(term.parse()?).await?;
let url = format!("{}/classSearch/{}", self.base_url, endpoint);
let params = self.build_list_params(search, term, offset, max_results, &session.id());
let response = self
.client
.http
.get(&url)
.query(&params)
.send()
.await
.context("Failed to get terms")?;
.with_context(|| format!("Failed to get {}", endpoint))?;
let terms: Vec<BannerTerm> = response
let data: Vec<T> = response
.json()
.await
.context("Failed to parse terms response")?;
.with_context(|| format!("Failed to parse {} response", endpoint))?;
Ok(terms)
Ok(data)
}
/// Builds search parameters for course search methods.
fn build_search_params(
&self,
query: &SearchQuery,
term: &str,
session_id: &str,
sort: &str,
sort_descending: bool,
) -> HashMap<String, String> {
let mut params = query.to_params();
params.insert("txt_term".to_string(), term.to_string());
params.insert("uniqueSessionId".to_string(), session_id.to_string());
params.insert("sortColumn".to_string(), sort.to_string());
params.insert(
"sortDirection".to_string(),
if sort_descending { "desc" } else { "asc" }.to_string(),
);
params.insert("startDatepicker".to_string(), String::new());
params.insert("endDatepicker".to_string(), String::new());
params
}
/// Performs a course search and handles common response processing.
#[tracing::instrument(
skip(self, query),
fields(
term = %term,
subject = %query.get_subject().unwrap_or(&"all".to_string())
)
)]
async fn perform_search(
&self,
term: &str,
query: &SearchQuery,
sort: &str,
sort_descending: bool,
) -> Result<SearchResult, BannerApiError> {
let mut session = self.sessions.acquire(term.parse()?).await?;
if session.been_used() {
self.http
.post(format!("{}/classSearch/resetDataForm", self.base_url))
.header("Cookie", session.cookie())
.send()
.await
.map_err(|e| BannerApiError::RequestFailed(e.into()))?;
}
session.touch();
let params = self.build_search_params(query, term, &session.id(), sort, sort_descending);
debug!(
term = term,
subject = query.get_subject().map(|s| s.as_str()).unwrap_or("all"),
max_results = query.get_max_results(),
"Searching for courses"
);
let response = self
.http
.get(format!("{}/searchResults/searchResults", self.base_url))
.header("Cookie", session.cookie())
.query(&params)
.send()
.await
.context("Failed to search courses")?;
let status = response.status();
let url = response.url().clone();
let body = response
.text()
.await
.with_context(|| format!("Failed to read body (status={status})"))?;
let search_result: SearchResult = parse_json_with_context(&body).map_err(|e| {
BannerApiError::RequestFailed(anyhow!(
"Failed to parse search response (status={status}, url={url}): {e}"
))
})?;
// Check for signs of an invalid session
if search_result.path_mode.is_none() {
return Err(BannerApiError::InvalidSession(
"Search result path mode is none".to_string(),
));
} else if search_result.data.is_none() {
return Err(BannerApiError::InvalidSession(
"Search result data is none".to_string(),
));
}
if !search_result.success {
return Err(BannerApiError::RequestFailed(anyhow!(
"Search marked as unsuccessful by Banner API"
)));
}
Ok(search_result)
}
/// Retrieves a list of subjects from the Banner API.
@@ -86,35 +238,8 @@ impl BannerApi {
offset: i32,
max_results: i32,
) -> Result<Vec<Pair>> {
if offset <= 0 {
return Err(anyhow::anyhow!("Offset must be greater than 0"));
}
let session_id = self.session_manager.ensure_session()?;
let url = format!("{}/classSearch/get_subject", self.base_url);
let params = [
("searchTerm", search),
("term", term),
("offset", &offset.to_string()),
("max", &max_results.to_string()),
("uniqueSessionId", &session_id),
("_", &timestamp_nonce()),
];
let response = self
.client
.get(&url)
.query(&params)
.send()
self.get_list_endpoint("get_subject", search, term, offset, max_results)
.await
.context("Failed to get subjects")?;
let subjects: Vec<Pair> = response
.json()
.await
.context("Failed to parse subjects response")?;
Ok(subjects)
}
/// Retrieves a list of instructors from the Banner API.
@@ -125,87 +250,33 @@ impl BannerApi {
offset: i32,
max_results: i32,
) -> Result<Vec<Instructor>> {
if offset <= 0 {
return Err(anyhow::anyhow!("Offset must be greater than 0"));
}
let session_id = self.session_manager.ensure_session()?;
let url = format!("{}/classSearch/get_instructor", self.base_url);
let params = [
("searchTerm", search),
("term", term),
("offset", &offset.to_string()),
("max", &max_results.to_string()),
("uniqueSessionId", &session_id),
("_", &timestamp_nonce()),
];
let response = self
.client
.get(&url)
.query(&params)
.send()
self.get_list_endpoint("get_instructor", search, term, offset, max_results)
.await
.context("Failed to get instructors")?;
let instructors: Vec<Instructor> = response
.json()
.await
.context("Failed to parse instructors response")?;
Ok(instructors)
}
/// Retrieves a list of campuses from the Banner API.
pub async fn get_campuses(
&self,
search: &str,
term: i32,
term: &str,
offset: i32,
max_results: i32,
) -> Result<Vec<Pair>> {
if offset <= 0 {
return Err(anyhow::anyhow!("Offset must be greater than 0"));
}
let session_id = self.session_manager.ensure_session()?;
let url = format!("{}/classSearch/get_campus", self.base_url);
let params = [
("searchTerm", search),
("term", &term.to_string()),
("offset", &offset.to_string()),
("max", &max_results.to_string()),
("uniqueSessionId", &session_id),
("_", &timestamp_nonce()),
];
let response = self
.client
.get(&url)
.query(&params)
.send()
self.get_list_endpoint("get_campus", search, term, offset, max_results)
.await
.context("Failed to get campuses")?;
let campuses: Vec<Pair> = response
.json()
.await
.context("Failed to parse campuses response")?;
Ok(campuses)
}
/// Retrieves meeting time information for a course.
pub async fn get_course_meeting_time(
&self,
term: &str,
crn: i32,
crn: &str,
) -> Result<Vec<MeetingScheduleInfo>> {
let url = format!("{}/searchResults/getFacultyMeetingTimes", self.base_url);
let params = [("term", term), ("courseReferenceNumber", &crn.to_string())];
let params = [("term", term), ("courseReferenceNumber", crn)];
let response = self
.client
.http
.get(&url)
.query(&params)
.send()
@@ -236,14 +307,14 @@ impl BannerApi {
));
}
#[derive(serde::Deserialize)]
struct ResponseWrapper {
fmt: Vec<MeetingTimeResponse>,
}
let response: MeetingTimesApiResponse =
response.json().await.context("Failed to parse response")?;
let wrapper: ResponseWrapper = response.json().await.context("Failed to parse response")?;
Ok(wrapper.fmt.into_iter().map(|m| m.schedule_info()).collect())
Ok(response
.fmt
.into_iter()
.map(|m| m.schedule_info())
.collect())
}
/// Performs a search for courses.
@@ -253,95 +324,33 @@ impl BannerApi {
query: &SearchQuery,
sort: &str,
sort_descending: bool,
) -> Result<SearchResult> {
self.session_manager.reset_data_form().await?;
let session_id = self.session_manager.ensure_session()?;
let mut params = query.to_params();
// Add additional parameters
params.insert("txt_term".to_string(), term.to_string());
params.insert("uniqueSessionId".to_string(), session_id);
params.insert("sortColumn".to_string(), sort.to_string());
params.insert(
"sortDirection".to_string(),
if sort_descending { "desc" } else { "asc" }.to_string(),
);
params.insert("startDatepicker".to_string(), String::new());
params.insert("endDatepicker".to_string(), String::new());
let url = format!("{}/searchResults/searchResults", self.base_url);
let response = self
.client
.get(&url)
.query(&params)
.send()
) -> Result<SearchResult, BannerApiError> {
self.perform_search(term, query, sort, sort_descending)
.await
.context("Failed to search courses")?;
let search_result: SearchResult = response
.json()
.await
.context("Failed to parse search response")?;
if !search_result.success {
return Err(anyhow::anyhow!(
"Search marked as unsuccessful by Banner API"
));
}
Ok(search_result)
}
/// Selects a term for the current session.
pub async fn select_term(&self, term: &str) -> Result<()> {
self.session_manager.select_term(term).await
}
/// Retrieves a single course by CRN by issuing a minimal search
pub async fn get_course_by_crn(&self, term: &str, crn: &str) -> Result<Option<Course>> {
self.session_manager.reset_data_form().await?;
// Ensure session is configured for this term
self.select_term(term).await?;
let session_id = self.session_manager.ensure_session()?;
pub async fn get_course_by_crn(
&self,
term: &str,
crn: &str,
) -> Result<Option<Course>, BannerApiError> {
debug!(term = term, crn = crn, "Looking up course by CRN");
let query = SearchQuery::new()
.course_reference_number(crn)
.max_results(1);
let mut params = query.to_params();
params.insert("txt_term".to_string(), term.to_string());
params.insert("uniqueSessionId".to_string(), session_id);
params.insert("sortColumn".to_string(), "subjectDescription".to_string());
params.insert("sortDirection".to_string(), "asc".to_string());
params.insert("startDatepicker".to_string(), String::new());
params.insert("endDatepicker".to_string(), String::new());
let search_result = self
.perform_search(term, &query, "subjectDescription", false)
.await?;
let url = format!("{}/searchResults/searchResults", self.base_url);
let response = self
.client
.get(&url)
.query(&params)
.send()
.await
.context("Failed to search course by CRN")?;
let status = response.status();
let body = response
.text()
.await
.with_context(|| format!("Failed to read body (status={status})"))?;
let search_result: SearchResult = parse_json_with_context(&body).map_err(|e| {
anyhow::anyhow!(
"Failed to parse search response for CRN (status={status}, url={url}): {e}",
)
})?;
if !search_result.success {
return Err(anyhow::anyhow!(
"Search marked as unsuccessful by Banner API"
// Additional validation for CRN search
if search_result.path_mode == Some("registration".to_string())
&& search_result.data.is_none()
{
return Err(BannerApiError::InvalidSession(
"Search result path mode is registration and data is none".to_string(),
));
}
@@ -351,16 +360,16 @@ impl BannerApi {
}
/// Gets course details (placeholder - needs implementation).
pub async fn get_course_details(&self, term: i32, crn: i32) -> Result<ClassDetails> {
pub async fn get_course_details(&self, term: &str, crn: &str) -> Result<ClassDetails> {
let body = serde_json::json!({
"term": term.to_string(),
"courseReferenceNumber": crn.to_string(),
"term": term,
"courseReferenceNumber": crn,
"first": "first"
});
let url = format!("{}/searchResults/getClassDetails", self.base_url);
let response = self
.client
.http
.post(&url)
.json(&body)
.send()
@@ -375,54 +384,3 @@ impl BannerApi {
Ok(details)
}
}
/// Generates a timestamp-based nonce.
fn timestamp_nonce() -> String {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis()
.to_string()
}
/// Returns a browser-like user agent string.
fn user_agent() -> &'static str {
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
}
/// Attempt to parse JSON and, on failure, include a contextual snippet around the error location
fn parse_json_with_context<T: serde::de::DeserializeOwned>(body: &str) -> Result<T> {
match serde_json::from_str::<T>(body) {
Ok(value) => Ok(value),
Err(err) => {
let (line, column) = (err.line(), err.column());
let snippet = build_error_snippet(body, line as usize, column as usize, 120);
Err(anyhow::anyhow!(
"{} at line {}, column {}\nSnippet:\n{}",
err,
line,
column,
snippet
))
}
}
}
fn build_error_snippet(body: &str, line: usize, column: usize, max_len: usize) -> String {
let target_line = body.lines().nth(line.saturating_sub(1)).unwrap_or("");
if target_line.is_empty() {
return String::new();
}
let start = column.saturating_sub(max_len.min(column));
let end = (column + max_len).min(target_line.len());
let slice = &target_line[start..end];
let mut indicator = String::new();
if column > start {
indicator.push_str(&" ".repeat(column - start - 1));
indicator.push('^');
}
format!("{}\n{}", slice, indicator)
}

11
src/banner/errors.rs Normal file
View File

@@ -0,0 +1,11 @@
//! Error types for the Banner API client.
use thiserror::Error;
#[derive(Debug, thiserror::Error)]
pub enum BannerApiError {
#[error("Banner session is invalid or expired: {0}")]
InvalidSession(String),
#[error(transparent)]
RequestFailed(#[from] anyhow::Error),
}

415
src/banner/json.rs Normal file
View File

@@ -0,0 +1,415 @@
//! JSON parsing utilities for the Banner API client.
use anyhow::Result;
use serde_json::{self, Value};
/// Attempt to parse JSON and, on failure, include a contextual snippet of the
/// line where the error occurred.
///
/// In debug builds, this provides detailed context including the full JSON object
/// containing the error and type mismatch information. In release builds, it shows
/// a minimal snippet to prevent dumping huge JSON bodies to production logs.
pub fn parse_json_with_context<T: serde::de::DeserializeOwned>(body: &str) -> Result<T> {
let jd = &mut serde_json::Deserializer::from_str(body);
match serde_path_to_error::deserialize(jd) {
Ok(value) => Ok(value),
Err(err) => {
let inner_err = err.inner();
let (line, column) = (inner_err.line(), inner_err.column());
let path = err.path().to_string();
let msg = inner_err.to_string();
let loc = format!(" at line {line} column {column}");
let msg_without_loc = msg.strip_suffix(&loc).unwrap_or(&msg).to_string();
// Build error message differently for debug vs release builds
let final_err = if cfg!(debug_assertions) {
// Debug mode: provide detailed context
let type_info = parse_type_mismatch(&msg_without_loc);
let context = extract_json_object_at_path(body, err.path(), line, column);
let mut err_msg = String::new();
if !path.is_empty() && path != "." {
err_msg.push_str(&format!("for path '{}'\n", path));
}
err_msg.push_str(&format!(
"({}) at line {} column {}\n\n",
type_info, line, column
));
err_msg.push_str(&context);
err_msg
} else {
// Release mode: minimal snippet to keep logs concise
let snippet = build_error_snippet(body, line, column, 20);
let mut err_msg = String::new();
if !path.is_empty() && path != "." {
err_msg.push_str(&format!("for path '{}' ", path));
}
err_msg.push_str(&format!(
"({}) at line {} column {}",
msg_without_loc, line, column
));
err_msg.push_str(&format!("\n{}", snippet));
err_msg
};
Err(anyhow::anyhow!(final_err))
}
}
}
/// Extract type mismatch information from a serde error message.
///
/// Parses error messages like "invalid type: null, expected a string" to extract
/// the expected and actual types for clearer error reporting.
///
/// Returns a formatted string like "(expected a string, got null)" or the original
/// message if parsing fails.
fn parse_type_mismatch(error_msg: &str) -> String {
// Try to parse "invalid type: X, expected Y" format
if let Some(invalid_start) = error_msg.find("invalid type: ") {
let after_prefix = &error_msg[invalid_start + "invalid type: ".len()..];
if let Some(comma_pos) = after_prefix.find(", expected ") {
let actual_type = &after_prefix[..comma_pos];
let expected_part = &after_prefix[comma_pos + ", expected ".len()..];
// Clean up expected part (remove " at line X column Y" if present)
let expected_type = expected_part
.split(" at line ")
.next()
.unwrap_or(expected_part)
.trim();
return format!("expected {}, got {}", expected_type, actual_type);
}
}
// Try to parse "expected X at line Y" format
if error_msg.starts_with("expected ")
&& let Some(expected_part) = error_msg.split(" at line ").next()
{
return expected_part.to_string();
}
// Fallback: return original message without location info
error_msg.to_string()
}
/// Extract and pretty-print the JSON object/array containing the parse error.
///
/// This function navigates to the error location using the serde path and extracts
/// the parent object or array to provide better context for debugging.
///
/// # Arguments
/// * `body` - The raw JSON string
/// * `path` - The serde path to the error (e.g., "data[0].faculty[0].displayName")
/// * `line` - Line number of the error (for fallback)
/// * `column` - Column number of the error (for fallback)
///
/// # Returns
/// A formatted string containing the JSON object with the error, or a fallback snippet
fn extract_json_object_at_path(
body: &str,
path: &serde_path_to_error::Path,
line: usize,
column: usize,
) -> String {
// Try to parse the entire JSON structure
let root_value: Value = match serde_json::from_str(body) {
Ok(v) => v,
Err(_) => {
// If we can't parse the JSON at all, fall back to line snippet
return build_error_snippet(body, line, column, 20);
}
};
// Navigate to the error location using the path
let path_str = path.to_string();
let segments = parse_path_segments(&path_str);
let (context_value, context_name) = navigate_to_context(&root_value, &segments);
// Pretty-print the context value with limited depth to avoid huge output
match serde_json::to_string_pretty(&context_value) {
Ok(pretty) => {
// Limit output to ~50 lines to prevent log spam
let lines: Vec<&str> = pretty.lines().collect();
let truncated = if lines.len() > 50 {
let mut result = lines[..47].join("\n");
result.push_str("\n ... (truncated, ");
result.push_str(&(lines.len() - 47).to_string());
result.push_str(" more lines)");
result
} else {
pretty
};
format!("{} at '{}':\n{}", context_name, path_str, truncated)
}
Err(_) => {
// Fallback to simple snippet if pretty-print fails
build_error_snippet(body, line, column, 20)
}
}
}
/// Parse a JSON path string into segments for navigation.
///
/// Converts paths like "data[0].faculty[1].displayName" into a sequence of
/// object keys and array indices.
fn parse_path_segments(path: &str) -> Vec<PathSegment> {
let mut segments = Vec::new();
let mut current = String::new();
let mut in_bracket = false;
for ch in path.chars() {
match ch {
'.' if !in_bracket => {
if !current.is_empty() {
segments.push(PathSegment::Key(current.clone()));
current.clear();
}
}
'[' => {
if !current.is_empty() {
segments.push(PathSegment::Key(current.clone()));
current.clear();
}
in_bracket = true;
}
']' => {
if in_bracket && !current.is_empty() {
if let Ok(index) = current.parse::<usize>() {
segments.push(PathSegment::Index(index));
}
current.clear();
}
in_bracket = false;
}
_ => current.push(ch),
}
}
if !current.is_empty() {
segments.push(PathSegment::Key(current));
}
segments
}
/// Represents a segment in a JSON path (either an object key or array index).
#[derive(Debug)]
enum PathSegment {
Key(String),
Index(usize),
}
/// Navigate through a JSON value using path segments and return the appropriate context.
///
/// This function walks the JSON structure and returns the parent object/array that
/// contains the error, providing meaningful context for debugging.
///
/// # Returns
/// A tuple of (context_value, description) where context_value is the JSON to display
/// and description is a human-readable name for what we're showing.
fn navigate_to_context<'a>(
mut current: &'a Value,
segments: &[PathSegment],
) -> (&'a Value, &'static str) {
// If path is empty or just root, return the whole value
if segments.is_empty() {
return (current, "Root object");
}
// Try to navigate to the parent of the error location
// We want to show the containing object/array, not just the failing field
let parent_depth = segments.len().saturating_sub(1);
for (i, segment) in segments.iter().enumerate() {
// Stop one level before the end to show the parent context
if i >= parent_depth {
break;
}
match segment {
PathSegment::Key(key) => {
if let Some(next) = current.get(key) {
current = next;
} else {
// Can't navigate further, return what we have
return (current, "Partial context (navigation stopped)");
}
}
PathSegment::Index(idx) => {
if let Some(next) = current.get(idx) {
current = next;
} else {
return (current, "Partial context (index out of bounds)");
}
}
}
}
(current, "Object containing error")
}
fn build_error_snippet(body: &str, line: usize, column: usize, context_len: usize) -> String {
let target_line = body.lines().nth(line.saturating_sub(1)).unwrap_or("");
if target_line.is_empty() {
return "(empty line)".to_string();
}
// column is 1-based, convert to 0-based for slicing
let error_idx = column.saturating_sub(1);
let half_len = context_len / 2;
let start = error_idx.saturating_sub(half_len);
let end = (error_idx + half_len).min(target_line.len());
let slice = &target_line[start..end];
let indicator_pos = error_idx - start;
let indicator = " ".repeat(indicator_pos) + "^";
format!("...{slice}...\n {indicator}")
}
#[cfg(test)]
mod tests {
use super::*;
use serde::Deserialize;
#[test]
fn test_parse_type_mismatch_invalid_type() {
let msg = "invalid type: null, expected a string at line 45 column 29";
let result = parse_type_mismatch(msg);
assert_eq!(result, "expected a string, got null");
}
#[test]
fn test_parse_type_mismatch_expected() {
let msg = "expected value at line 1 column 1";
let result = parse_type_mismatch(msg);
assert_eq!(result, "expected value");
}
#[test]
fn test_parse_path_segments_simple() {
let segments = parse_path_segments("data.name");
assert_eq!(segments.len(), 2);
match &segments[0] {
PathSegment::Key(k) => assert_eq!(k, "data"),
_ => panic!("Expected Key segment"),
}
}
#[test]
fn test_parse_path_segments_with_array() {
let segments = parse_path_segments("data[0].faculty[1].displayName");
assert_eq!(segments.len(), 5);
match &segments[0] {
PathSegment::Key(k) => assert_eq!(k, "data"),
_ => panic!("Expected Key segment"),
}
match &segments[1] {
PathSegment::Index(i) => assert_eq!(*i, 0),
_ => panic!("Expected Index segment"),
}
}
#[test]
fn test_parse_json_with_context_null_value() {
#[derive(Debug, Deserialize)]
struct TestStruct {
name: String,
}
let json = r#"{"name": null}"#;
let result: Result<TestStruct> = parse_json_with_context(json);
assert!(result.is_err());
let err_msg = result.unwrap_err().to_string();
// Should contain path info
assert!(err_msg.contains("name"));
// In debug mode, should contain detailed context
if cfg!(debug_assertions) {
assert!(err_msg.contains("expected"));
}
}
#[test]
fn test_navigate_to_context() {
let json = r#"{"data": [{"faculty": [{"name": "John"}]}]}"#;
let value: Value = serde_json::from_str(json).unwrap();
let segments = parse_path_segments("data[0].faculty[0].name");
let (context, _) = navigate_to_context(&value, &segments);
// Should return the faculty[0] object (parent of 'name')
assert!(context.is_object());
assert!(context.get("name").is_some());
}
#[test]
fn test_realistic_banner_error() {
#[derive(Debug, Deserialize)]
struct Course {
#[allow(dead_code)]
#[serde(rename = "courseTitle")]
course_title: String,
faculty: Vec<Faculty>,
}
#[derive(Debug, Deserialize)]
struct Faculty {
#[serde(rename = "displayName")]
display_name: String,
#[allow(dead_code)]
email: String,
}
#[derive(Debug, Deserialize)]
struct SearchResult {
data: Vec<Course>,
}
// Simulate Banner API response with null faculty displayName
// This mimics the actual error from SPN subject scrape
let json = r#"{
"data": [
{
"courseTitle": "Spanish Conversation",
"faculty": [
{
"displayName": null,
"email": "instructor@utsa.edu"
}
]
}
]
}"#;
let result: Result<SearchResult> = parse_json_with_context(json);
assert!(result.is_err());
let err_msg = result.unwrap_err().to_string();
println!("\n=== Error output in debug mode ===\n{}\n", err_msg);
// Verify error contains key information
assert!(err_msg.contains("data[0].faculty[0].displayName"));
// In debug mode, should show detailed context
if cfg!(debug_assertions) {
// Should show type mismatch info
assert!(err_msg.contains("expected") && err_msg.contains("got"));
// Should show surrounding JSON context with the faculty object
assert!(err_msg.contains("email"));
}
}
}

75
src/banner/middleware.rs Normal file
View File

@@ -0,0 +1,75 @@
//! HTTP middleware for the Banner API client.
use http::Extensions;
use reqwest::{Request, Response};
use reqwest_middleware::{Middleware, Next};
use tracing::{debug, trace, warn};
pub struct TransparentMiddleware;
/// Threshold for logging slow requests at DEBUG level (in milliseconds)
const SLOW_REQUEST_THRESHOLD_MS: u128 = 1000;
#[async_trait::async_trait]
impl Middleware for TransparentMiddleware {
async fn handle(
&self,
req: Request,
extensions: &mut Extensions,
next: Next<'_>,
) -> std::result::Result<Response, reqwest_middleware::Error> {
let method = req.method().to_string();
let path = req.url().path().to_string();
let start = std::time::Instant::now();
let response_result = next.run(req, extensions).await;
let duration = start.elapsed();
match response_result {
Ok(response) => {
if response.status().is_success() {
let duration_ms = duration.as_millis();
if duration_ms >= SLOW_REQUEST_THRESHOLD_MS {
debug!(
method = method,
path = path,
status = response.status().as_u16(),
duration_ms = duration_ms,
"Request completed (slow)"
);
} else {
trace!(
method = method,
path = path,
status = response.status().as_u16(),
duration_ms = duration_ms,
"Request completed"
);
}
Ok(response)
} else {
let e = response.error_for_status_ref().unwrap_err();
warn!(
method = method,
path = path,
error = ?e,
status = response.status().as_u16(),
duration_ms = duration.as_millis(),
"Request failed"
);
Ok(response)
}
}
Err(error) => {
warn!(
method = method,
path = path,
error = ?error,
duration_ms = duration.as_millis(),
"Request failed"
);
Err(error)
}
}
}
}

View File

@@ -1,18 +1,26 @@
#![allow(unused_imports)]
//! Banner API module for interacting with Ellucian Banner systems.
//!
//! This module provides functionality to:
//! - Search for courses and retrieve course information
//! - Manage Banner API sessions and authentication
//! - Scrape course data and cache it in Redis
//! - Generate ICS files and calendar links
pub mod api;
pub mod errors;
pub mod json;
pub mod middleware;
pub mod models;
pub mod query;
pub mod scraper;
pub mod rate_limit_middleware;
pub mod rate_limiter;
pub mod session;
pub mod util;
pub use api::*;
pub use errors::*;
pub use models::*;
pub use query::*;
pub use rate_limiter::*;
pub use session::*;

View File

@@ -59,6 +59,24 @@ pub struct Course {
pub meetings_faculty: Vec<MeetingTimeResponse>,
}
impl Course {
/// Returns the course title in the format "SUBJ #### - Course Title"
pub fn display_title(&self) -> String {
format!(
"{} {} - {}",
self.subject, self.course_number, self.course_title
)
}
/// Returns the name of the primary instructor, or "Unknown" if not available
pub fn primary_instructor_name(&self) -> &str {
self.faculty
.first()
.map(|f| f.display_name.as_str())
.unwrap_or("Unknown")
}
}
/// Class details (to be implemented)
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ClassDetails {

View File

@@ -1,7 +1,7 @@
use bitflags::{Flags, bitflags};
use chrono::{DateTime, NaiveDate, NaiveTime, Timelike, Utc};
use serde::{Deserialize, Deserializer, Serialize};
use std::{cmp::Ordering, str::FromStr};
use std::{cmp::Ordering, collections::HashSet, fmt::Display, str::FromStr};
use super::terms::Term;
@@ -33,7 +33,7 @@ pub struct FacultyItem {
#[serde(deserialize_with = "deserialize_string_to_u32")]
pub course_reference_number: u32, // CRN, e.g 27294
pub display_name: String, // "LastName, FirstName"
pub email_address: String, // e.g. FirstName.LastName@utsaedu
pub email_address: Option<String>, // e.g. FirstName.LastName@utsaedu
pub primary_indicator: bool,
pub term: String, // e.g "202420"
}
@@ -42,11 +42,11 @@ pub struct FacultyItem {
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct MeetingTime {
pub start_date: String, // MM/DD/YYYY, e.g 08/26/2025
pub end_date: String, // MM/DD/YYYY, e.g 08/26/2025
pub begin_time: String, // HHMM, e.g 1000
pub end_time: String, // HHMM, e.g 1100
pub category: String, // unknown meaning, e.g. 01, 02, etc
pub start_date: String, // MM/DD/YYYY, e.g 08/26/2025
pub end_date: String, // MM/DD/YYYY, e.g 08/26/2025
pub begin_time: Option<String>, // HHMM, e.g 1000
pub end_time: Option<String>, // HHMM, e.g 1100
pub category: String, // unknown meaning, e.g. 01, 02, etc
pub class: String, // internal class name, e.g. net.hedtech.banner.general.overallMeetingTimeDecorator
pub monday: bool, // true if the meeting time occurs on Monday
pub tuesday: bool, // true if the meeting time occurs on Tuesday
@@ -55,15 +55,15 @@ pub struct MeetingTime {
pub friday: bool, // true if the meeting time occurs on Friday
pub saturday: bool, // true if the meeting time occurs on Saturday
pub sunday: bool, // true if the meeting time occurs on Sunday
pub room: String, // e.g. 1238
pub room: Option<String>, // e.g. 1.238
#[serde(deserialize_with = "deserialize_string_to_term")]
pub term: Term, // e.g 202510
pub building: String, // e.g NPB
pub building_description: String, // e.g North Paseo Building
pub campus: String, // campus code, e.g 11
pub campus_description: String, // name of campus, e.g Main Campus
pub building: Option<String>, // e.g NPB
pub building_description: Option<String>, // e.g North Paseo Building
pub campus: Option<String>, // campus code, e.g 11
pub campus_description: Option<String>, // name of campus, e.g Main Campus
pub course_reference_number: String, // CRN, e.g 27294
pub credit_hour_session: f64, // e.g. 30
pub credit_hour_session: Option<f64>, // e.g. 30
pub hours_week: f64, // e.g. 30
pub meeting_schedule_type: String, // e.g AFF
pub meeting_type: String, // e.g HB, H2, H1, OS, OA, OH, ID, FF
@@ -148,20 +148,22 @@ pub enum DayOfWeek {
impl DayOfWeek {
/// Convert to short string representation
pub fn to_short_string(&self) -> &'static str {
///
/// Do not change these, these are used for ICS generation. Casing does not matter though.
pub fn to_short_string(self) -> &'static str {
match self {
DayOfWeek::Monday => "M",
DayOfWeek::Monday => "Mo",
DayOfWeek::Tuesday => "Tu",
DayOfWeek::Wednesday => "W",
DayOfWeek::Wednesday => "We",
DayOfWeek::Thursday => "Th",
DayOfWeek::Friday => "F",
DayOfWeek::Friday => "Fr",
DayOfWeek::Saturday => "Sa",
DayOfWeek::Sunday => "Su",
}
}
/// Convert to full string representation
pub fn to_string(&self) -> &'static str {
pub fn to_full_string(self) -> &'static str {
match self {
DayOfWeek::Monday => "Monday",
DayOfWeek::Tuesday => "Tuesday",
@@ -196,10 +198,9 @@ impl TryFrom<MeetingDays> for DayOfWeek {
});
}
return Err(anyhow::anyhow!(
"Cannot convert multiple days to a single day: {:?}",
days
));
Err(anyhow::anyhow!(
"Cannot convert multiple days to a single day: {days:?}"
))
}
}
@@ -252,18 +253,12 @@ impl TimeRange {
let hour = time.hour();
let minute = time.minute();
if hour == 0 {
format!("12:{:02}AM", minute)
} else if hour < 12 {
format!("{}:{:02}AM", hour, minute)
} else if hour == 12 {
format!("12:{:02}PM", minute)
} else {
format!("{}:{:02}PM", hour - 12, minute)
}
let meridiem = if hour < 12 { "AM" } else { "PM" };
format!("{hour}:{minute:02}{meridiem}")
}
/// Get duration in minutes
#[allow(dead_code)]
pub fn duration_minutes(&self) -> i64 {
let start_minutes = self.start.hour() as i64 * 60 + self.start.minute() as i64;
let end_minutes = self.end.hour() as i64 * 60 + self.end.minute() as i64;
@@ -308,6 +303,7 @@ impl DateRange {
}
/// Check if a specific date falls within this range
#[allow(dead_code)]
pub fn contains_date(&self, date: NaiveDate) -> bool {
date >= self.start && date <= self.end
}
@@ -355,37 +351,58 @@ impl MeetingType {
/// Meeting location information
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MeetingLocation {
pub campus: String,
pub building: String,
pub building_description: String,
pub room: String,
pub is_online: bool,
pub enum MeetingLocation {
Online,
InPerson {
campus: String,
campus_description: String,
building: String,
building_description: String,
room: String,
},
}
impl MeetingLocation {
/// Create from raw MeetingTime data
pub fn from_meeting_time(meeting_time: &MeetingTime) -> Self {
let is_online = meeting_time.room.is_empty();
if meeting_time.campus.is_none()
|| meeting_time.building.is_none()
|| meeting_time.building_description.is_none()
|| meeting_time.room.is_none()
|| meeting_time.campus_description.is_none()
|| meeting_time
.campus_description
.eq(&Some("Internet".to_string()))
{
return MeetingLocation::Online;
}
MeetingLocation {
campus: meeting_time.campus_description.clone(),
building: meeting_time.building.clone(),
building_description: meeting_time.building_description.clone(),
room: meeting_time.room.clone(),
is_online,
MeetingLocation::InPerson {
campus: meeting_time.campus.as_ref().unwrap().clone(),
campus_description: meeting_time.campus_description.as_ref().unwrap().clone(),
building: meeting_time.building.as_ref().unwrap().clone(),
building_description: meeting_time.building_description.as_ref().unwrap().clone(),
room: meeting_time.room.as_ref().unwrap().clone(),
}
}
}
/// Convert to formatted string
pub fn to_string(&self) -> String {
if self.is_online {
"Online".to_string()
} else {
format!(
"{} | {} | {} {}",
self.campus, self.building_description, self.building, self.room
)
impl Display for MeetingLocation {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
MeetingLocation::Online => write!(f, "Online"),
MeetingLocation::InPerson {
campus,
building,
building_description,
room,
..
} => write!(
f,
"{campus} | {building_name} | {building_code} {room}",
building_name = building_description,
building_code = building,
),
}
}
}
@@ -405,7 +422,11 @@ impl MeetingScheduleInfo {
/// Create from raw MeetingTime data
pub fn from_meeting_time(meeting_time: &MeetingTime) -> Self {
let days = MeetingDays::from_meeting_time(meeting_time);
let time_range = TimeRange::from_hhmm(&meeting_time.begin_time, &meeting_time.end_time);
let time_range = match (&meeting_time.begin_time, &meeting_time.end_time) {
(Some(begin), Some(end)) => TimeRange::from_hhmm(begin, end),
_ => None,
};
let date_range =
DateRange::from_mm_dd_yyyy(&meeting_time.start_date, &meeting_time.end_date)
.unwrap_or_else(|| {
@@ -439,32 +460,52 @@ impl MeetingScheduleInfo {
}
/// Get formatted days string
pub fn days_string(&self) -> String {
pub fn days_string(&self) -> Option<String> {
if self.days.is_empty() {
"None".to_string()
} else if self.days.is_all() {
"Everyday".to_string()
} else {
self.days_of_week()
.iter()
.map(|day| day.to_short_string())
.collect::<Vec<_>>()
.join("")
return None;
}
if self.days.is_all() {
return Some("Everyday".to_string());
}
let days_of_week = self.days_of_week();
if days_of_week.len() == 1 {
return Some(days_of_week[0].to_full_string().to_string());
}
// Mapper function to get the short string representation of the day of week
let mapper = {
let ambiguous = self.days.intersects(
MeetingDays::Tuesday
| MeetingDays::Thursday
| MeetingDays::Saturday
| MeetingDays::Sunday,
);
if ambiguous {
|day: &DayOfWeek| day.to_short_string().to_string()
} else {
|day: &DayOfWeek| day.to_short_string().chars().next().unwrap().to_string()
}
};
Some(days_of_week.iter().map(mapper).collect::<String>())
}
/// Returns a formatted string representing the location of the meeting
pub fn place_string(&self) -> String {
if self.location.room.is_empty() {
"Online".to_string()
} else {
format!(
match &self.location {
MeetingLocation::Online => "Online".to_string(),
MeetingLocation::InPerson {
campus,
building,
building_description,
room,
..
} => format!(
"{} | {} | {} {}",
self.location.campus,
self.location.building_description,
self.location.building,
self.location.room
)
campus, building_description, building, room
),
}
}

View File

@@ -10,8 +10,8 @@ pub struct SearchResult {
pub total_count: i32,
pub page_offset: i32,
pub page_max_size: i32,
pub path_mode: String,
pub search_results_config: Vec<SearchResultConfig>,
pub path_mode: Option<String>,
pub search_results_config: Option<Vec<SearchResultConfig>>,
pub data: Option<Vec<Course>>,
}

View File

@@ -13,7 +13,7 @@ const CURRENT_YEAR: u32 = compile_time::date!().year() as u32;
const VALID_YEARS: RangeInclusive<u32> = 2007..=(CURRENT_YEAR + 10);
/// Represents a term in the Banner system
#[derive(Debug, Clone, Serialize, Deserialize)]
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, Hash)]
pub struct Term {
pub year: u32, // 2024, 2025, etc
pub season: Season,
@@ -29,7 +29,7 @@ pub enum TermPoint {
}
/// Represents a season within a term
#[derive(Debug, Clone, Serialize, Deserialize)]
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, Hash)]
pub enum Season {
Fall,
Spring,
@@ -46,7 +46,7 @@ impl Term {
/// Returns the current term status for a specific date
pub fn get_status_for_date(date: NaiveDate) -> TermPoint {
let literal_year = date.year() as u32;
let day_of_year = date.ordinal() as u32;
let day_of_year = date.ordinal();
let ranges = Self::get_season_ranges(literal_year);
// If we're past the end of the summer term, we're 'in' the next school year.
@@ -115,22 +115,22 @@ impl Term {
fn get_season_ranges(year: u32) -> SeasonRanges {
let spring_start = NaiveDate::from_ymd_opt(year as i32, 1, 14)
.unwrap()
.ordinal() as u32;
.ordinal();
let spring_end = NaiveDate::from_ymd_opt(year as i32, 5, 1)
.unwrap()
.ordinal() as u32;
.ordinal();
let summer_start = NaiveDate::from_ymd_opt(year as i32, 5, 25)
.unwrap()
.ordinal() as u32;
.ordinal();
let summer_end = NaiveDate::from_ymd_opt(year as i32, 8, 15)
.unwrap()
.ordinal() as u32;
.ordinal();
let fall_start = NaiveDate::from_ymd_opt(year as i32, 8, 18)
.unwrap()
.ordinal() as u32;
.ordinal();
let fall_end = NaiveDate::from_ymd_opt(year as i32, 12, 10)
.unwrap()
.ordinal() as u32;
.ordinal();
SeasonRanges {
spring: YearDayRange {
@@ -147,11 +147,6 @@ impl Term {
},
}
}
/// Returns a long string representation of the term (e.g., "Fall 2025")
pub fn to_long_string(&self) -> String {
format!("{} {}", self.season, self.year)
}
}
impl TermPoint {
@@ -179,16 +174,21 @@ struct YearDayRange {
end: u32,
}
impl ToString for Term {
impl std::fmt::Display for Term {
/// Returns the term in the format YYYYXX, where YYYY is the year and XX is the season code
fn to_string(&self) -> String {
format!("{}{}", self.year, self.season.to_str())
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
f,
"{year}{season}",
year = self.year,
season = self.season.to_str()
)
}
}
impl Season {
/// Returns the season code as a string
fn to_str(&self) -> &'static str {
fn to_str(self) -> &'static str {
match self {
Season::Fall => "10",
Season::Spring => "20",
@@ -215,7 +215,7 @@ impl FromStr for Season {
"10" => Season::Fall,
"20" => Season::Spring,
"30" => Season::Summer,
_ => return Err(anyhow::anyhow!("Invalid season: {}", s)),
_ => return Err(anyhow::anyhow!("Invalid season: {s}")),
};
Ok(season)
}

View File

@@ -32,6 +32,7 @@ pub struct SearchQuery {
course_number_range: Option<Range>,
}
#[allow(dead_code)]
impl SearchQuery {
/// Creates a new SearchQuery with default values
pub fn new() -> Self {
@@ -160,6 +161,16 @@ impl SearchQuery {
self
}
/// Gets the subject field
pub fn get_subject(&self) -> Option<&String> {
self.subject.as_ref()
}
/// Gets the max_results field
pub fn get_max_results(&self) -> i32 {
self.max_results
}
/// Converts the query into URL parameters for the Banner API
pub fn to_params(&self) -> HashMap<String, String> {
let mut params = HashMap::new();
@@ -270,7 +281,7 @@ impl std::fmt::Display for SearchQuery {
let mut parts = Vec::new();
if let Some(ref subject) = self.subject {
parts.push(format!("subject={}", subject));
parts.push(format!("subject={subject}"));
}
if let Some(ref title) = self.title {
parts.push(format!("title={}", title.trim()));
@@ -296,21 +307,21 @@ impl std::fmt::Display for SearchQuery {
.map(|i| i.to_string())
.collect::<Vec<_>>()
.join(",");
parts.push(format!("instructor={}", instructor_str));
parts.push(format!("instructor={instructor_str}"));
}
if let Some(start_time) = self.start_time {
let (hour, minute, meridiem) = format_time_parameter(start_time);
parts.push(format!("startTime={}:{}:{}", hour, minute, meridiem));
parts.push(format!("startTime={hour}:{minute}:{meridiem}"));
}
if let Some(end_time) = self.end_time {
let (hour, minute, meridiem) = format_time_parameter(end_time);
parts.push(format!("endTime={}:{}:{}", hour, minute, meridiem));
parts.push(format!("endTime={hour}:{minute}:{meridiem}"));
}
if let Some(min_credits) = self.min_credits {
parts.push(format!("minCredits={}", min_credits));
parts.push(format!("minCredits={min_credits}"));
}
if let Some(max_credits) = self.max_credits {
parts.push(format!("maxCredits={}", max_credits));
parts.push(format!("maxCredits={max_credits}"));
}
if let Some(ref range) = self.course_number_range {
parts.push(format!("courseNumberRange={}-{}", range.low, range.high));

View File

@@ -0,0 +1,84 @@
//! HTTP middleware that enforces rate limiting for Banner API requests.
use crate::banner::rate_limiter::{RequestType, SharedRateLimiter};
use http::Extensions;
use reqwest::{Request, Response};
use reqwest_middleware::{Middleware, Next};
use tracing::debug;
use url::Url;
/// Middleware that enforces rate limiting based on request URL patterns
pub struct RateLimitMiddleware {
rate_limiter: SharedRateLimiter,
}
impl RateLimitMiddleware {
/// Creates a new rate limiting middleware
pub fn new(rate_limiter: SharedRateLimiter) -> Self {
Self { rate_limiter }
}
/// Returns a human-readable description of the rate limit for a request type
fn get_rate_limit_description(request_type: RequestType) -> &'static str {
match request_type {
RequestType::Session => "6 rpm (~10s interval)",
RequestType::Search => "30 rpm (~2s interval)",
RequestType::Metadata => "20 rpm (~3s interval)",
RequestType::Reset => "10 rpm (~6s interval)",
}
}
/// Determines the request type based on the URL path
fn get_request_type(url: &Url) -> RequestType {
let path = url.path();
if path.contains("/registration")
|| path.contains("/selfServiceMenu")
|| path.contains("/term/termSelection")
{
RequestType::Session
} else if path.contains("/searchResults") || path.contains("/classSearch") {
RequestType::Search
} else if path.contains("/getTerms")
|| path.contains("/getSubjects")
|| path.contains("/getCampuses")
{
RequestType::Metadata
} else if path.contains("/resetDataForm") {
RequestType::Reset
} else {
// Default to search for unknown endpoints
RequestType::Search
}
}
}
#[async_trait::async_trait]
impl Middleware for RateLimitMiddleware {
async fn handle(
&self,
req: Request,
extensions: &mut Extensions,
next: Next<'_>,
) -> std::result::Result<Response, reqwest_middleware::Error> {
let request_type = Self::get_request_type(req.url());
let start = std::time::Instant::now();
self.rate_limiter.wait_for_permission(request_type).await;
let wait_duration = start.elapsed();
// Only log if rate limiting caused significant delay (>= 500ms)
if wait_duration.as_millis() >= 500 {
let limit_desc = Self::get_rate_limit_description(request_type);
debug!(
request_type = ?request_type,
wait_ms = wait_duration.as_millis(),
rate_limit = limit_desc,
"Rate limit caused delay"
);
}
// Make the actual request
next.run(req, extensions).await
}
}

131
src/banner/rate_limiter.rs Normal file
View File

@@ -0,0 +1,131 @@
//! Rate limiting for Banner API requests to prevent overwhelming the server.
use governor::{
Quota, RateLimiter,
clock::DefaultClock,
state::{InMemoryState, NotKeyed},
};
use std::num::NonZeroU32;
use std::sync::Arc;
use std::time::Duration;
/// Different types of Banner API requests with different rate limits
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum RequestType {
/// Session creation and management (very conservative)
Session,
/// Course search requests (moderate)
Search,
/// Term and metadata requests (moderate)
Metadata,
/// Data form resets (low priority)
Reset,
}
/// Rate limiter configuration for different request types
#[derive(Debug, Clone)]
pub struct RateLimitConfig {
/// Requests per minute for session operations
pub session_rpm: u32,
/// Requests per minute for search operations
pub search_rpm: u32,
/// Requests per minute for metadata operations
pub metadata_rpm: u32,
/// Requests per minute for reset operations
pub reset_rpm: u32,
/// Burst allowance (extra requests allowed in short bursts)
pub burst_allowance: u32,
}
impl Default for RateLimitConfig {
fn default() -> Self {
Self {
// Very conservative for session creation
session_rpm: 6, // 1 every 10 seconds
// Moderate for search operations
search_rpm: 30, // 1 every 2 seconds
// Moderate for metadata
metadata_rpm: 20, // 1 every 3 seconds
// Low for resets
reset_rpm: 10, // 1 every 6 seconds
// Allow small bursts
burst_allowance: 3,
}
}
}
/// A rate limiter that manages different request types with different limits
pub struct BannerRateLimiter {
session_limiter: RateLimiter<NotKeyed, InMemoryState, DefaultClock>,
search_limiter: RateLimiter<NotKeyed, InMemoryState, DefaultClock>,
metadata_limiter: RateLimiter<NotKeyed, InMemoryState, DefaultClock>,
reset_limiter: RateLimiter<NotKeyed, InMemoryState, DefaultClock>,
}
impl BannerRateLimiter {
/// Creates a new rate limiter with the given configuration
pub fn new(config: RateLimitConfig) -> Self {
let session_quota = Quota::with_period(Duration::from_secs(60) / config.session_rpm)
.unwrap()
.allow_burst(NonZeroU32::new(config.burst_allowance).unwrap());
let search_quota = Quota::with_period(Duration::from_secs(60) / config.search_rpm)
.unwrap()
.allow_burst(NonZeroU32::new(config.burst_allowance).unwrap());
let metadata_quota = Quota::with_period(Duration::from_secs(60) / config.metadata_rpm)
.unwrap()
.allow_burst(NonZeroU32::new(config.burst_allowance).unwrap());
let reset_quota = Quota::with_period(Duration::from_secs(60) / config.reset_rpm)
.unwrap()
.allow_burst(NonZeroU32::new(config.burst_allowance).unwrap());
Self {
session_limiter: RateLimiter::direct(session_quota),
search_limiter: RateLimiter::direct(search_quota),
metadata_limiter: RateLimiter::direct(metadata_quota),
reset_limiter: RateLimiter::direct(reset_quota),
}
}
/// Waits for permission to make a request of the given type
pub async fn wait_for_permission(&self, request_type: RequestType) {
let limiter = match request_type {
RequestType::Session => &self.session_limiter,
RequestType::Search => &self.search_limiter,
RequestType::Metadata => &self.metadata_limiter,
RequestType::Reset => &self.reset_limiter,
};
// Wait until we can make the request (logging handled by middleware)
limiter.until_ready().await;
}
}
impl Default for BannerRateLimiter {
fn default() -> Self {
Self::new(RateLimitConfig::default())
}
}
/// A shared rate limiter instance
pub type SharedRateLimiter = Arc<BannerRateLimiter>;
/// Creates a new shared rate limiter with custom configuration
pub fn create_shared_rate_limiter(config: Option<RateLimitConfig>) -> SharedRateLimiter {
Arc::new(BannerRateLimiter::new(config.unwrap_or_default()))
}
/// Conversion from config module's RateLimitingConfig to this module's RateLimitConfig
impl From<crate::config::RateLimitingConfig> for RateLimitConfig {
fn from(config: crate::config::RateLimitingConfig) -> Self {
Self {
session_rpm: config.session_rpm,
search_rpm: config.search_rpm,
metadata_rpm: config.metadata_rpm,
reset_rpm: config.reset_rpm,
burst_allowance: config.burst_allowance,
}
}
}

View File

@@ -1,293 +0,0 @@
//! Course scraping functionality for the Banner API.
use crate::banner::{api::BannerApi, models::*, query::SearchQuery};
use anyhow::{Context, Result};
use redis::AsyncCommands;
use std::time::Duration;
use tokio::time;
use tracing::{debug, error, info, warn};
/// Priority majors that should be scraped more frequently
const PRIORITY_MAJORS: &[&str] = &["CS", "CPE", "MAT", "EE", "IS"];
/// Maximum number of courses to fetch per page
const MAX_PAGE_SIZE: i32 = 500;
/// Course scraper for Banner API
pub struct CourseScraper {
api: BannerApi,
redis_client: redis::Client,
}
impl CourseScraper {
/// Creates a new course scraper
pub fn new(api: BannerApi, redis_url: &str) -> Result<Self> {
let redis_client =
redis::Client::open(redis_url).context("Failed to create Redis client")?;
Ok(Self { api, redis_client })
}
/// Scrapes all courses and stores them in Redis
pub async fn scrape_all(&self, term: &str) -> Result<()> {
// Get all subjects
let subjects = self
.api
.get_subjects("", term, 1, 100)
.await
.context("Failed to get subjects for scraping")?;
if subjects.is_empty() {
return Err(anyhow::anyhow!("No subjects found for term {}", term));
}
// Categorize subjects
let (priority_subjects, other_subjects): (Vec<_>, Vec<_>) = subjects
.into_iter()
.partition(|subject| PRIORITY_MAJORS.contains(&subject.code.as_str()));
// Get expired subjects that need scraping
let mut expired_subjects = Vec::new();
expired_subjects.extend(self.get_expired_subjects(&priority_subjects, term).await?);
expired_subjects.extend(self.get_expired_subjects(&other_subjects, term).await?);
if expired_subjects.is_empty() {
info!("No expired subjects found, skipping scrape");
return Ok(());
}
info!(
"Scraping {} subjects for term {}",
expired_subjects.len(),
term
);
// Scrape each expired subject
for subject in expired_subjects {
if let Err(e) = self.scrape_subject(&subject.code, term).await {
error!("Failed to scrape subject {}: {}", subject.code, e);
}
// Rate limiting between subjects
time::sleep(Duration::from_secs(2)).await;
}
Ok(())
}
/// Gets subjects that have expired and need to be scraped
async fn get_expired_subjects(&self, subjects: &[Pair], term: &str) -> Result<Vec<Pair>> {
let mut conn = self
.redis_client
.get_multiplexed_async_connection()
.await
.context("Failed to get Redis connection")?;
let mut expired = Vec::new();
for subject in subjects {
let key = format!("scraped:{}:{}", subject.code, term);
let scraped: Option<String> = conn
.get(&key)
.await
.context("Failed to check scrape status in Redis")?;
// If not scraped or marked as expired (empty/0), add to list
if scraped.is_none() || scraped.as_deref() == Some("0") {
expired.push(subject.clone());
}
}
Ok(expired)
}
/// Scrapes all courses for a specific subject
pub async fn scrape_subject(&self, subject: &str, term: &str) -> Result<()> {
let mut offset = 0;
let mut total_courses = 0;
loop {
let query = SearchQuery::new()
.subject(subject)
.offset(offset)
.max_results(MAX_PAGE_SIZE * 2);
// Ensure session term is selected before searching
self.api.select_term(term).await?;
let result = self
.api
.search(term, &query, "subjectDescription", false)
.await
.with_context(|| {
format!(
"Failed to search for subject {} at offset {}",
subject, offset
)
})?;
if !result.success {
return Err(anyhow::anyhow!(
"Search marked unsuccessful for subject {}",
subject
));
}
let course_count = result.data.as_ref().map(|v| v.len() as i32).unwrap_or(0);
total_courses += course_count;
debug!(
"Retrieved {} courses for subject {} at offset {}",
course_count, subject, offset
);
// Store each course in Redis
for course in result.data.unwrap_or_default() {
if let Err(e) = self.store_course(&course).await {
error!(
"Failed to store course {}: {}",
course.course_reference_number, e
);
}
}
// Check if we got a full page and should continue
if course_count >= MAX_PAGE_SIZE {
if course_count > MAX_PAGE_SIZE {
warn!(
"Course count {} exceeds max page size {}",
course_count, MAX_PAGE_SIZE
);
}
offset += MAX_PAGE_SIZE;
debug!(
"Continuing to next page for subject {} at offset {}",
subject, offset
);
// Rate limiting between pages
time::sleep(Duration::from_secs(3)).await;
continue;
}
break;
}
info!(
"Scraped {} total courses for subject {}",
total_courses, subject
);
// Mark subject as scraped with expiry
self.mark_subject_scraped(subject, term, total_courses)
.await?;
Ok(())
}
/// Stores a course in Redis
async fn store_course(&self, course: &Course) -> Result<()> {
let mut conn = self
.redis_client
.get_multiplexed_async_connection()
.await
.context("Failed to get Redis connection")?;
let key = format!("class:{}", course.course_reference_number);
let serialized = serde_json::to_string(course).context("Failed to serialize course")?;
let _: () = conn
.set(&key, serialized)
.await
.context("Failed to store course in Redis")?;
Ok(())
}
/// Marks a subject as scraped with appropriate expiry time
async fn mark_subject_scraped(
&self,
subject: &str,
term: &str,
course_count: i32,
) -> Result<()> {
let mut conn = self
.redis_client
.get_multiplexed_async_connection()
.await
.context("Failed to get Redis connection")?;
let key = format!("scraped:{}:{}", subject, term);
let expiry = self.calculate_expiry(subject, course_count);
let value = if course_count == 0 { -1 } else { course_count };
let _: () = conn
.set_ex(&key, value, expiry.as_secs() as u64)
.await
.context("Failed to mark subject as scraped")?;
debug!(
"Marked subject {} as scraped with {} courses, expiry: {:?}",
subject, course_count, expiry
);
Ok(())
}
/// Calculates expiry time for a scraped subject based on various factors
fn calculate_expiry(&self, subject: &str, course_count: i32) -> Duration {
// Base calculation: 1 hour per 100 courses
let mut base_expiry = Duration::from_secs(3600 * (course_count as u64 / 100).max(1));
// Special handling for subjects with few courses
if course_count < 50 {
// Linear interpolation: 1 course = 12 hours, 49 courses = 1 hour
let hours = 12.0 - ((course_count as f64 - 1.0) / 48.0) * 11.0;
base_expiry = Duration::from_secs((hours * 3600.0) as u64);
}
// Priority subjects get shorter expiry (more frequent updates)
if PRIORITY_MAJORS.contains(&subject) {
base_expiry = base_expiry / 3;
}
// Add random variance (±15%)
let variance = (base_expiry.as_secs() as f64 * 0.15) as u64;
let random_offset = (rand::random::<f64>() - 0.5) * 2.0 * variance as f64;
let final_expiry = if random_offset > 0.0 {
base_expiry + Duration::from_secs(random_offset as u64)
} else {
base_expiry.saturating_sub(Duration::from_secs((-random_offset) as u64))
};
// Ensure minimum of 1 hour
final_expiry.max(Duration::from_secs(3600))
}
/// Gets a course from Redis cache
pub async fn get_course(&self, crn: &str) -> Result<Option<Course>> {
let mut conn = self
.redis_client
.get_multiplexed_async_connection()
.await
.context("Failed to get Redis connection")?;
let key = format!("class:{}", crn);
let serialized: Option<String> = conn
.get(&key)
.await
.context("Failed to get course from Redis")?;
match serialized {
Some(data) => {
let course: Course = serde_json::from_str(&data)
.context("Failed to deserialize course from Redis")?;
Ok(Some(course))
}
None => Ok(None),
}
}
}

View File

@@ -1,120 +1,425 @@
//! Session management for Banner API.
use anyhow::Result;
use rand::distributions::{Alphanumeric, DistString};
use reqwest::Client;
use std::sync::Mutex;
use crate::banner::BannerTerm;
use crate::banner::models::Term;
use anyhow::{Context, Result};
use cookie::Cookie;
use dashmap::DashMap;
use governor::state::InMemoryState;
use governor::{Quota, RateLimiter};
use once_cell::sync::Lazy;
use rand::distr::{Alphanumeric, SampleString};
use reqwest_middleware::ClientWithMiddleware;
use std::collections::{HashMap, VecDeque};
use std::num::NonZeroU32;
use std::ops::{Deref, DerefMut};
use std::sync::Arc;
use std::time::{Duration, Instant};
use tracing::{debug, info};
use tokio::sync::{Mutex, Notify};
use tracing::{debug, info, trace};
use url::Url;
/// Session manager for Banner API interactions
#[derive(Debug)]
pub struct SessionManager {
current_session: Mutex<Option<SessionData>>,
base_url: String,
client: Client,
}
const SESSION_EXPIRY: Duration = Duration::from_secs(25 * 60); // 25 minutes
// A global rate limiter to ensure we only try to create one new session every 10 seconds,
// preventing us from overwhelming the server with session creation requests.
static SESSION_CREATION_RATE_LIMITER: Lazy<
RateLimiter<governor::state::direct::NotKeyed, InMemoryState, governor::clock::DefaultClock>,
> = Lazy::new(|| RateLimiter::direct(Quota::with_period(Duration::from_secs(10)).unwrap()));
/// Represents an active anonymous session within the Banner API.
/// Identified by multiple persistent cookies, as well as a client-generated "unique session ID".
#[derive(Debug, Clone)]
struct SessionData {
session_id: String,
pub struct BannerSession {
// Randomly generated
pub unique_session_id: String,
// Timestamp of creation
created_at: Instant,
// Timestamp of last activity
last_activity: Option<Instant>,
// Cookie values from initial registration page
jsessionid: String,
ssb_cookie: String,
}
impl SessionManager {
const SESSION_EXPIRY: Duration = Duration::from_secs(25 * 60); // 25 minutes
/// Generates a new session ID mimicking Banner's format
fn generate_session_id() -> String {
let random_part = Alphanumeric.sample_string(&mut rand::rng(), 5);
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis();
format!("{}{}", random_part, timestamp)
}
/// Creates a new session manager
pub fn new(base_url: String, client: Client) -> Self {
/// Generates a timestamp-based nonce
pub fn nonce() -> String {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis()
.to_string()
}
impl BannerSession {
/// Creates a new session
pub async fn new(unique_session_id: &str, jsessionid: &str, ssb_cookie: &str) -> Result<Self> {
let now = Instant::now();
Ok(Self {
created_at: now,
last_activity: None,
unique_session_id: unique_session_id.to_string(),
jsessionid: jsessionid.to_string(),
ssb_cookie: ssb_cookie.to_string(),
})
}
/// Returns the unique session ID
pub fn id(&self) -> String {
self.unique_session_id.clone()
}
/// Updates the last activity timestamp
pub fn touch(&mut self) {
self.last_activity = Some(Instant::now());
}
/// Returns true if the session is expired
pub fn is_expired(&self) -> bool {
self.last_activity.unwrap_or(self.created_at).elapsed() > SESSION_EXPIRY
}
/// Returns a string used to for the "Cookie" header
pub fn cookie(&self) -> String {
format!(
"JSESSIONID={}; SSB_COOKIE={}",
self.jsessionid, self.ssb_cookie
)
}
pub fn been_used(&self) -> bool {
self.last_activity.is_some()
}
}
/// A smart pointer that returns a BannerSession to the pool when dropped.
pub struct PooledSession {
session: Option<BannerSession>,
// This Arc points directly to the term-specific pool.
pool: Arc<TermPool>,
}
impl PooledSession {
pub fn been_used(&self) -> bool {
self.session.as_ref().unwrap().been_used()
}
}
impl Deref for PooledSession {
type Target = BannerSession;
fn deref(&self) -> &Self::Target {
// The option is only ever None after drop is called, so this is safe.
self.session.as_ref().unwrap()
}
}
impl DerefMut for PooledSession {
fn deref_mut(&mut self) -> &mut Self::Target {
self.session.as_mut().unwrap()
}
}
/// The magic happens here: when the guard goes out of scope, this is called.
impl Drop for PooledSession {
fn drop(&mut self) {
if let Some(session) = self.session.take() {
let pool = self.pool.clone();
// Since drop() cannot be async, we spawn a task to return the session.
tokio::spawn(async move {
pool.release(session).await;
});
}
}
}
pub struct TermPool {
sessions: Mutex<VecDeque<BannerSession>>,
notifier: Notify,
is_creating: Mutex<bool>,
}
impl TermPool {
fn new() -> Self {
Self {
current_session: Mutex::new(None),
base_url,
client,
sessions: Mutex::new(VecDeque::new()),
notifier: Notify::new(),
is_creating: Mutex::new(false),
}
}
/// Ensures a valid session is available, creating one if necessary
pub fn ensure_session(&self) -> Result<String> {
let mut session_guard = self.current_session.lock().unwrap();
async fn release(&self, session: BannerSession) {
let id = session.unique_session_id.clone();
if session.is_expired() {
debug!(id = id, "Session expired, dropping");
// Wake up a waiter, as it might need to create a new session
// if this was the last one.
self.notifier.notify_one();
return;
}
if let Some(ref session) = *session_guard {
if session.created_at.elapsed() < Self::SESSION_EXPIRY {
return Ok(session.session_id.clone());
let mut queue = self.sessions.lock().await;
queue.push_back(session);
drop(queue); // Release lock before notifying
self.notifier.notify_one();
}
}
pub struct SessionPool {
sessions: DashMap<Term, Arc<TermPool>>,
http: ClientWithMiddleware,
base_url: String,
}
impl SessionPool {
pub fn new(http: ClientWithMiddleware, base_url: String) -> Self {
Self {
sessions: DashMap::new(),
http,
base_url,
}
}
/// Acquires a session from the pool.
/// If no sessions are available, a new one is created on demand,
/// respecting the global rate limit.
pub async fn acquire(&self, term: Term) -> Result<PooledSession> {
let term_pool = self
.sessions
.entry(term)
.or_insert_with(|| Arc::new(TermPool::new()))
.clone();
let start = Instant::now();
let mut waited_for_creation = false;
loop {
// Fast path: Try to get an existing, non-expired session.
{
let mut queue = term_pool.sessions.lock().await;
if let Some(session) = queue.pop_front() {
if !session.is_expired() {
return Ok(PooledSession {
session: Some(session),
pool: Arc::clone(&term_pool),
});
} else {
debug!(id = session.unique_session_id, "Discarded expired session");
}
}
} // MutexGuard is dropped, lock is released.
// Slow path: No sessions available. We must either wait or become the creator.
let mut is_creating_guard = term_pool.is_creating.lock().await;
if *is_creating_guard {
// Another task is already creating a session. Release the lock and wait.
drop(is_creating_guard);
if !waited_for_creation {
trace!("Waiting for another task to create session");
waited_for_creation = true;
}
term_pool.notifier.notified().await;
// Loop back to the top to try the fast path again.
continue;
}
// This task is now the designated creator.
*is_creating_guard = true;
drop(is_creating_guard);
// Race: wait for a session to be returned OR for the rate limiter to allow a new one.
trace!("Pool empty, creating new session");
tokio::select! {
_ = term_pool.notifier.notified() => {
// A session was returned while we were waiting!
// We are no longer the creator. Reset the flag and loop to race for the new session.
let mut guard = term_pool.is_creating.lock().await;
*guard = false;
drop(guard);
continue;
}
_ = SESSION_CREATION_RATE_LIMITER.until_ready() => {
// The rate limit has elapsed. It's our job to create the session.
let new_session_result = self.create_session(&term).await;
// After creation, we are no longer the creator. Reset the flag
// and notify all other waiting tasks.
let mut guard = term_pool.is_creating.lock().await;
*guard = false;
drop(guard);
term_pool.notifier.notify_waiters();
match new_session_result {
Ok(new_session) => {
let elapsed = start.elapsed();
debug!(
id = new_session.unique_session_id,
elapsed_ms = elapsed.as_millis(),
"Created new session"
);
return Ok(PooledSession {
session: Some(new_session),
pool: term_pool,
});
}
Err(e) => {
// Propagate the error if session creation failed.
return Err(e.context("Failed to create new session in pool"));
}
}
}
}
}
// Generate new session
let session_id = self.generate_session_id();
*session_guard = Some(SessionData {
session_id: session_id.clone(),
created_at: Instant::now(),
});
debug!("Generated new Banner session: {}", session_id);
Ok(session_id)
}
/// Generates a new session ID mimicking Banner's format
fn generate_session_id(&self) -> String {
let random_part = Alphanumeric.sample_string(&mut rand::thread_rng(), 5);
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis();
format!("{}{}", random_part, timestamp)
}
/// Sets up initial session cookies by making required Banner API requests
pub async fn setup(&self) -> Result<()> {
info!("Setting up Banner session...");
pub async fn create_session(&self, term: &Term) -> Result<BannerSession> {
info!(term = %term, "setting up banner session");
let request_paths = ["/registration/registration", "/selfServiceMenu/data"];
// The 'register' or 'search' registration page
let initial_registration = self
.http
.get(format!("{}/registration", self.base_url))
.send()
.await?;
// TODO: Validate success
for path in &request_paths {
let url = format!("{}{}", self.base_url, path);
let response = self
.client
.get(&url)
.query(&[("_", timestamp_nonce())])
.header("User-Agent", user_agent())
.send()
.await?;
let cookies = initial_registration
.headers()
.get_all("Set-Cookie")
.iter()
.filter_map(|header_value| {
if let Ok(cookie_str) = header_value.to_str() {
if let Ok(cookie) = Cookie::parse(cookie_str) {
Some((cookie.name().to_string(), cookie.value().to_string()))
} else {
None
}
} else {
None
}
})
.collect::<HashMap<String, String>>();
if !response.status().is_success() {
return Err(anyhow::anyhow!(
"Failed to setup session, request to {} returned {}",
path,
response.status()
));
}
if !cookies.contains_key("JSESSIONID") || !cookies.contains_key("SSB_COOKIE") {
return Err(anyhow::anyhow!("Failed to get cookies"));
}
// Note: Cookie validation would require additional setup in a real implementation
debug!("Session setup complete");
Ok(())
let jsessionid = cookies
.get("JSESSIONID")
.ok_or_else(|| anyhow::anyhow!("JSESSIONID cookie missing after validation"))?;
let ssb_cookie = cookies
.get("SSB_COOKIE")
.ok_or_else(|| anyhow::anyhow!("SSB_COOKIE cookie missing after validation"))?;
let cookie_header = format!("JSESSIONID={}; SSB_COOKIE={}", jsessionid, ssb_cookie);
self.http
.get(format!("{}/selfServiceMenu/data", self.base_url))
.header("Cookie", &cookie_header)
.send()
.await?
.error_for_status()
.context("Failed to get data page")?;
self.http
.get(format!("{}/term/termSelection", self.base_url))
.header("Cookie", &cookie_header)
.query(&[("mode", "search")])
.send()
.await?
.error_for_status()
.context("Failed to get term selection page")?;
// TOOD: Validate success
let terms = self.get_terms("", 1, 10).await?;
if !terms.iter().any(|t| t.code == term.to_string()) {
return Err(anyhow::anyhow!("Failed to get term search response"));
}
let specific_term_search_response = self.get_terms(&term.to_string(), 1, 10).await?;
if !specific_term_search_response
.iter()
.any(|t| t.code == term.to_string())
{
return Err(anyhow::anyhow!("Failed to get term search response"));
}
let unique_session_id = generate_session_id();
self.select_term(&term.to_string(), &unique_session_id, &cookie_header)
.await?;
BannerSession::new(&unique_session_id, jsessionid, ssb_cookie).await
}
/// Retrieves a list of terms from the Banner API.
pub async fn get_terms(
&self,
search: &str,
page: i32,
max_results: i32,
) -> Result<Vec<BannerTerm>> {
if page <= 0 {
return Err(anyhow::anyhow!("Page must be greater than 0"));
}
let url = format!("{}/classSearch/getTerms", self.base_url);
let params = [
("searchTerm", search),
("offset", &page.to_string()),
("max", &max_results.to_string()),
("_", &nonce()),
];
let response = self
.http
.get(&url)
.query(&params)
.send()
.await
.with_context(|| "Failed to get terms".to_string())?;
let terms: Vec<BannerTerm> = response
.json()
.await
.context("Failed to parse terms response")?;
Ok(terms)
}
/// Selects a term for the current session
pub async fn select_term(&self, term: &str) -> Result<()> {
let session_id = self.ensure_session()?;
pub async fn select_term(
&self,
term: &str,
unique_session_id: &str,
cookie_header: &str,
) -> Result<()> {
let form_data = [
("term", term),
("studyPath", ""),
("studyPathText", ""),
("startDatepicker", ""),
("endDatepicker", ""),
("uniqueSessionId", &session_id),
("uniqueSessionId", unique_session_id),
];
let url = format!("{}/term/search", self.base_url);
let response = self
.client
.http
.post(&url)
.header("Cookie", cookie_header)
.query(&[("mode", "search")])
.form(&form_data)
.header("User-Agent", user_agent())
.header("Content-Type", "application/x-www-form-urlencoded")
.send()
.await?;
@@ -128,18 +433,36 @@ impl SessionManager {
#[derive(serde::Deserialize)]
struct RedirectResponse {
#[serde(rename = "fwdUrl")]
#[serde(rename = "fwdURL")]
fwd_url: String,
}
let redirect: RedirectResponse = response.json().await?;
let base_url_path = self
.base_url
.parse::<Url>()
.context("Failed to parse base URL")?
.path()
.to_string();
let non_overlap_redirect =
redirect
.fwd_url
.strip_prefix(&base_url_path)
.ok_or_else(|| {
anyhow::anyhow!(
"Redirect URL '{}' does not start with expected prefix '{}'",
redirect.fwd_url,
base_url_path
)
})?;
// Follow the redirect
let redirect_url = format!("{}{}", self.base_url, redirect.fwd_url);
let redirect_url = format!("{}{}", self.base_url, non_overlap_redirect);
let redirect_response = self
.client
.http
.get(&redirect_url)
.header("User-Agent", user_agent())
.header("Cookie", cookie_header)
.send()
.await?;
@@ -150,41 +473,6 @@ impl SessionManager {
));
}
debug!("Successfully selected term: {}", term);
Ok(())
}
/// Resets the data form (required before new searches)
pub async fn reset_data_form(&self) -> Result<()> {
let url = format!("{}/classSearch/resetDataForm", self.base_url);
let response = self
.client
.post(&url)
.header("User-Agent", user_agent())
.send()
.await?;
if !response.status().is_success() {
return Err(anyhow::anyhow!(
"Failed to reset data form: {}",
response.status()
));
}
Ok(())
}
}
/// Generates a timestamp-based nonce
fn timestamp_nonce() -> String {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis()
.to_string()
}
/// Returns a browser-like user agent string
fn user_agent() -> &'static str {
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
}

6
src/banner/util.rs Normal file
View File

@@ -0,0 +1,6 @@
//! Utility functions for the Banner module.
/// Returns a browser-like user agent string.
pub fn user_agent() -> &'static str {
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
}

132
src/bin/search.rs Normal file
View File

@@ -0,0 +1,132 @@
use banner::banner::{BannerApi, SearchQuery, Term};
use banner::config::Config;
use banner::error::Result;
use figment::{Figment, providers::Env};
use futures::future;
use tracing::{error, info};
use tracing_subscriber::{EnvFilter, FmtSubscriber};
#[tokio::main]
async fn main() -> Result<()> {
// Configure logging
let filter = EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new("info,banner=trace,reqwest=debug,hyper=info"));
let subscriber = FmtSubscriber::builder()
.with_env_filter(filter)
.with_target(true)
.finish();
tracing::subscriber::set_global_default(subscriber).expect("setting default subscriber failed");
info!("Starting Banner search test");
dotenvy::dotenv().ok();
// Load configuration
let config: Config = Figment::new()
.merge(Env::raw())
.extract()
.expect("Failed to load config");
info!(
banner_base_url = config.banner_base_url,
"Configuration loaded"
);
// Create Banner API client
let banner_api =
BannerApi::new_with_config(config.banner_base_url, config.rate_limiting.into())
.expect("Failed to create BannerApi");
// Get current term
let term = Term::get_current().inner().to_string();
info!(term = term, "Using current term");
// Define multiple search queries
let queries = vec![
(
"CS Courses",
SearchQuery::new().subject("CS").max_results(10),
),
(
"Math Courses",
SearchQuery::new().subject("MAT").max_results(10),
),
(
"3000-level CS",
SearchQuery::new()
.subject("CS")
.course_numbers(3000, 3999)
.max_results(8),
),
(
"High Credit Courses",
SearchQuery::new().credits(4, 6).max_results(8),
),
(
"Programming Courses",
SearchQuery::new().keyword("programming").max_results(6),
),
];
info!(query_count = queries.len(), "Executing concurrent searches");
// Execute all searches concurrently
let search_futures = queries.into_iter().map(|(label, query)| {
info!(label = %label, "Starting search");
let banner_api = &banner_api;
let term = &term;
async move {
let result = banner_api
.search(term, &query, "subjectDescription", false)
.await;
(label, result)
}
});
// Wait for all searches to complete
let search_results = future::join_all(search_futures)
.await
.into_iter()
.filter_map(|(label, result)| match result {
Ok(search_result) => {
info!(
label = label,
success = search_result.success,
total_count = search_result.total_count,
"Search completed successfully"
);
Some((label, search_result))
}
Err(e) => {
error!(label = label, error = ?e, "Search failed");
None
}
})
.collect::<Vec<_>>();
// Process and display results
for (label, search_result) in search_results {
println!("\n=== {} ===", label);
if let Some(courses) = &search_result.data {
if courses.is_empty() {
println!(" No courses found");
} else {
println!(" Found {} courses:", courses.len());
for course in courses {
println!(
" {} {} - {} (CRN: {})",
course.subject,
course.course_number,
course.course_title,
course.course_reference_number
);
}
}
} else {
println!(" No courses found");
}
}
info!("Search test completed");
Ok(())
}

View File

@@ -1,10 +1,10 @@
//! Google Calendar command implementation.
use crate::banner::{Course, DayOfWeek, MeetingScheduleInfo, Term};
use crate::bot::{Context, Error};
use crate::banner::{Course, DayOfWeek, MeetingScheduleInfo};
use crate::bot::{Context, Error, utils};
use chrono::NaiveDate;
use std::collections::HashMap;
use tracing::{error, info};
use tracing::info;
use url::Url;
/// Generate a link to create a Google Calendar event for a course
@@ -18,36 +18,16 @@ pub async fn gcal(
ctx.defer().await?;
let app_state = &ctx.data().app_state;
let banner_api = &app_state.banner_api;
// Get current term dynamically
let current_term_status = Term::get_current();
let term = current_term_status.inner();
// Fetch live course data from Redis cache via AppState
let course = match app_state
.get_course_or_fetch(&term.to_string(), &crn.to_string())
.await
{
Ok(course) => course,
Err(e) => {
error!(%e, crn, "Failed to fetch course data");
return Err(Error::from(e));
}
};
let course = utils::get_course_by_crn(&ctx, crn).await?;
let term = course.term.clone();
// Get meeting times
let meeting_times = match banner_api
.get_course_meeting_time(&term.to_string(), crn)
.await
{
Ok(meeting_time) => meeting_time,
Err(e) => {
error!("Failed to get meeting times: {}", e);
return Err(Error::from(e));
}
};
let meeting_times = ctx
.data()
.app_state
.banner_api
.get_course_meeting_time(&term, &crn.to_string())
.await?;
struct LinkDetail {
link: String,
@@ -74,8 +54,10 @@ pub async fn gcal(
.map(|m| {
let link = generate_gcal_url(&course, m)?;
let detail = match &m.time_range {
Some(range) => format!("{} {}", m.days_string(), range.format_12hr()),
None => m.days_string(),
Some(range) => {
format!("{} {}", m.days_string().unwrap(), range.format_12hr())
}
None => m.days_string().unwrap(),
};
Ok(LinkDetail { link, detail })
})
@@ -95,7 +77,7 @@ pub async fn gcal(
)
.await?;
info!("gcal command completed for CRN: {}", crn);
info!(crn = %crn, "gcal command completed");
Ok(())
}
@@ -104,10 +86,7 @@ fn generate_gcal_url(
course: &Course,
meeting_time: &MeetingScheduleInfo,
) -> Result<String, anyhow::Error> {
let course_text = format!(
"{} {} - {}",
course.subject, course.course_number, course.course_title
);
let course_text = course.display_title();
let dates_text = {
let (start, end) = meeting_time.datetime_range();
@@ -119,18 +98,14 @@ fn generate_gcal_url(
};
// Get instructor name
let instructor_name = if !course.faculty.is_empty() {
&course.faculty[0].display_name
} else {
"Unknown"
};
let instructor_name = course.primary_instructor_name();
// The event description
let details_text = format!(
"CRN: {}\nInstructor: {}\nDays: {}",
course.course_reference_number,
instructor_name,
meeting_time.days_string()
meeting_time.days_string().unwrap()
);
// The event location

View File

@@ -1,6 +1,110 @@
//! ICS command implementation for generating calendar files.
use crate::bot::{Context, Error};
use crate::banner::{Course, MeetingScheduleInfo};
use crate::bot::{Context, Error, utils};
use chrono::{Datelike, NaiveDate, Utc};
use serenity::all::CreateAttachment;
use tracing::info;
/// Represents a holiday or special day that should be excluded from class schedules
#[derive(Debug, Clone)]
enum Holiday {
/// A single-day holiday
Single { month: u32, day: u32 },
/// A multi-day holiday range
Range {
month: u32,
start_day: u32,
end_day: u32,
},
}
impl Holiday {
/// Check if a specific date falls within this holiday
fn contains_date(&self, date: NaiveDate) -> bool {
match self {
Holiday::Single { month, day, .. } => date.month() == *month && date.day() == *day,
Holiday::Range {
month,
start_day,
end_day,
..
} => date.month() == *month && date.day() >= *start_day && date.day() <= *end_day,
}
}
/// Get all dates in this holiday for a given year
fn get_dates_for_year(&self, year: i32) -> Vec<NaiveDate> {
match self {
Holiday::Single { month, day, .. } => {
if let Some(date) = NaiveDate::from_ymd_opt(year, *month, *day) {
vec![date]
} else {
Vec::new()
}
}
Holiday::Range {
month,
start_day,
end_day,
..
} => {
let mut dates = Vec::new();
for day in *start_day..=*end_day {
if let Some(date) = NaiveDate::from_ymd_opt(year, *month, day) {
dates.push(date);
}
}
dates
}
}
}
}
/// University holidays that should be excluded from class schedules
const UNIVERSITY_HOLIDAYS: &[(&str, Holiday)] = &[
("Labor Day", Holiday::Single { month: 9, day: 1 }),
(
"Fall Break",
Holiday::Range {
month: 10,
start_day: 13,
end_day: 14,
},
),
(
"Unspecified Holiday",
Holiday::Single { month: 11, day: 26 },
),
(
"Thanksgiving",
Holiday::Range {
month: 11,
start_day: 28,
end_day: 29,
},
),
("Student Study Day", Holiday::Single { month: 12, day: 5 }),
(
"Winter Holiday",
Holiday::Range {
month: 12,
start_day: 23,
end_day: 31,
},
),
("New Year's Day", Holiday::Single { month: 1, day: 1 }),
("MLK Day", Holiday::Single { month: 1, day: 20 }),
(
"Spring Break",
Holiday::Range {
month: 3,
start_day: 10,
end_day: 15,
},
),
("Student Study Day", Holiday::Single { month: 5, day: 9 }),
];
/// Generate an ICS file for a course
#[poise::command(slash_command, prefix_command)]
@@ -10,16 +114,323 @@ pub async fn ics(
) -> Result<(), Error> {
ctx.defer().await?;
// TODO: Get BannerApi from context or global state
// TODO: Get current term dynamically
let term = 202510; // Hardcoded for now
let course = utils::get_course_by_crn(&ctx, crn).await?;
let term = course.term.clone();
// TODO: Implement actual ICS file generation
ctx.say(format!(
"ICS command not yet implemented - BannerApi integration needed\nCRN: {}, Term: {}",
crn, term
))
// Get meeting times
let meeting_times = ctx
.data()
.app_state
.banner_api
.get_course_meeting_time(&term, &crn.to_string())
.await?;
if meeting_times.is_empty() {
ctx.say("No meeting times found for this course.").await?;
return Ok(());
}
// Sort meeting times by start time
let mut sorted_meeting_times = meeting_times.to_vec();
sorted_meeting_times.sort_unstable_by(|a, b| match (&a.time_range, &b.time_range) {
(Some(a_time), Some(b_time)) => a_time.start.cmp(&b_time.start),
(Some(_), None) => std::cmp::Ordering::Less,
(None, Some(_)) => std::cmp::Ordering::Greater,
(None, None) => a.days.bits().cmp(&b.days.bits()),
});
// Generate ICS content
let (ics_content, excluded_holidays) =
generate_ics_content(&course, &term, &sorted_meeting_times)?;
// Create file attachment
let filename = format!(
"{subject}_{number}_{section}.ics",
subject = course.subject.replace(" ", "_"),
number = course.course_number,
section = course.sequence_number,
);
let file = CreateAttachment::bytes(ics_content.into_bytes(), filename.clone());
// Build response content
let mut response_content = format!(
"📅 Generated ICS calendar for **{}**\n\n**Meeting Times:**\n{}",
course.display_title(),
sorted_meeting_times
.iter()
.enumerate()
.map(|(i, m)| {
let time_info = match &m.time_range {
Some(range) => format!(
"{} {}",
m.days_string().unwrap_or("TBA".to_string()),
range.format_12hr()
),
None => m.days_string().unwrap_or("TBA".to_string()),
};
format!("{}. {}", i + 1, time_info)
})
.collect::<Vec<_>>()
.join("\n")
);
// Add holiday exclusion information
if !excluded_holidays.is_empty() {
let count = excluded_holidays.len();
let count_text = if count == 1 {
"1 date was".to_string()
} else {
format!("{} dates were", count)
};
response_content.push_str(&format!("\n\n{} excluded from the ICS file:\n", count_text));
response_content.push_str(
&excluded_holidays
.iter()
.map(|s| format!("- {}", s))
.collect::<Vec<_>>()
.join("\n"),
);
}
ctx.send(
poise::CreateReply::default()
.content(response_content)
.attachment(file),
)
.await?;
info!(crn = %crn, "ics command completed");
Ok(())
}
/// Generate ICS content for a course and its meeting times
fn generate_ics_content(
course: &Course,
term: &str,
meeting_times: &[MeetingScheduleInfo],
) -> Result<(String, Vec<String>), anyhow::Error> {
let mut ics_content = String::new();
let mut excluded_holidays = Vec::new();
// ICS header
ics_content.push_str("BEGIN:VCALENDAR\r\n");
ics_content.push_str("VERSION:2.0\r\n");
ics_content.push_str("PRODID:-//Banner Bot//Course Calendar//EN\r\n");
ics_content.push_str("CALSCALE:GREGORIAN\r\n");
ics_content.push_str("METHOD:PUBLISH\r\n");
// Calendar name
ics_content.push_str(&format!(
"X-WR-CALNAME:{} - {}\r\n",
course.display_title(),
term
));
// Generate events for each meeting time
for (index, meeting_time) in meeting_times.iter().enumerate() {
let (event_content, holidays) = generate_event_content(course, meeting_time, index)?;
ics_content.push_str(&event_content);
excluded_holidays.extend(holidays);
}
// ICS footer
ics_content.push_str("END:VCALENDAR\r\n");
Ok((ics_content, excluded_holidays))
}
/// Generate ICS event content for a single meeting time
fn generate_event_content(
course: &Course,
meeting_time: &MeetingScheduleInfo,
index: usize,
) -> Result<(String, Vec<String>), anyhow::Error> {
let course_title = course.display_title();
let instructor_name = course.primary_instructor_name();
let location = meeting_time.place_string();
// Create event title with meeting index if multiple meetings
let event_title = if index > 0 {
format!("{} (Meeting {})", course_title, index + 1)
} else {
course_title
};
// Create event description
let description = format!(
"CRN: {}\\nInstructor: {}\\nDays: {}\\nMeeting Type: {}",
course.course_reference_number,
instructor_name,
meeting_time.days_string().unwrap_or("TBA".to_string()),
meeting_time.meeting_type.description()
);
// Get start and end times
let (start_dt, end_dt) = meeting_time.datetime_range();
// Format datetimes for ICS (UTC format)
let start_utc = start_dt.with_timezone(&Utc);
let end_utc = end_dt.with_timezone(&Utc);
let start_str = start_utc.format("%Y%m%dT%H%M%SZ").to_string();
let end_str = end_utc.format("%Y%m%dT%H%M%SZ").to_string();
// Generate unique ID for the event
let uid = format!(
"{}-{}-{}@banner-bot.local",
course.course_reference_number,
index,
start_utc.timestamp()
);
let mut event_content = String::new();
// Event header
event_content.push_str("BEGIN:VEVENT\r\n");
event_content.push_str(&format!("UID:{}\r\n", uid));
event_content.push_str(&format!("DTSTART:{}\r\n", start_str));
event_content.push_str(&format!("DTEND:{}\r\n", end_str));
event_content.push_str(&format!("SUMMARY:{}\r\n", escape_ics_text(&event_title)));
event_content.push_str(&format!(
"DESCRIPTION:{}\r\n",
escape_ics_text(&description)
));
event_content.push_str(&format!("LOCATION:{}\r\n", escape_ics_text(&location)));
// Add recurrence rule if there are specific days and times
if !meeting_time.days.is_empty() && meeting_time.time_range.is_some() {
let days_of_week = meeting_time.days_of_week();
let by_day: Vec<String> = days_of_week
.iter()
.map(|day| day.to_short_string().to_uppercase())
.collect();
if !by_day.is_empty() {
let until_date = meeting_time
.date_range
.end
.format("%Y%m%dT000000Z")
.to_string();
event_content.push_str(&format!(
"RRULE:FREQ=WEEKLY;BYDAY={};UNTIL={}\r\n",
by_day.join(","),
until_date
));
// Add holiday exceptions (EXDATE) if the class would meet on holiday dates
let holiday_exceptions = get_holiday_exceptions(meeting_time);
if let Some(exdate_property) = generate_exdate_property(&holiday_exceptions, start_utc)
{
event_content.push_str(&format!("{}\r\n", exdate_property));
}
// Collect holiday names for reporting
let mut holiday_names = Vec::new();
for (holiday_name, holiday) in UNIVERSITY_HOLIDAYS {
for &exception_date in &holiday_exceptions {
if holiday.contains_date(exception_date) {
holiday_names.push(format!(
"{} ({})",
holiday_name,
exception_date.format("%a, %b %d")
));
}
}
}
holiday_names.sort();
holiday_names.dedup();
return Ok((event_content, holiday_names));
}
}
// Event footer
event_content.push_str("END:VEVENT\r\n");
Ok((event_content, Vec::new()))
}
/// Convert chrono::Weekday to the custom DayOfWeek enum
fn chrono_weekday_to_day_of_week(weekday: chrono::Weekday) -> crate::banner::meetings::DayOfWeek {
use crate::banner::meetings::DayOfWeek;
match weekday {
chrono::Weekday::Mon => DayOfWeek::Monday,
chrono::Weekday::Tue => DayOfWeek::Tuesday,
chrono::Weekday::Wed => DayOfWeek::Wednesday,
chrono::Weekday::Thu => DayOfWeek::Thursday,
chrono::Weekday::Fri => DayOfWeek::Friday,
chrono::Weekday::Sat => DayOfWeek::Saturday,
chrono::Weekday::Sun => DayOfWeek::Sunday,
}
}
/// Check if a class meets on a specific date based on its meeting days
fn class_meets_on_date(meeting_time: &MeetingScheduleInfo, date: NaiveDate) -> bool {
let weekday = chrono_weekday_to_day_of_week(date.weekday());
let meeting_days = meeting_time.days_of_week();
meeting_days.contains(&weekday)
}
/// Get holiday dates that fall within the course date range and would conflict with class meetings
fn get_holiday_exceptions(meeting_time: &MeetingScheduleInfo) -> Vec<NaiveDate> {
let mut exceptions = Vec::new();
// Get the year range from the course date range
let start_year = meeting_time.date_range.start.year();
let end_year = meeting_time.date_range.end.year();
for (_, holiday) in UNIVERSITY_HOLIDAYS {
// Check for the holiday in each year of the course
for year in start_year..=end_year {
let holiday_dates = holiday.get_dates_for_year(year);
for holiday_date in holiday_dates {
// Check if the holiday falls within the course date range
if holiday_date >= meeting_time.date_range.start
&& holiday_date <= meeting_time.date_range.end
{
// Check if the class would actually meet on this day
if class_meets_on_date(meeting_time, holiday_date) {
exceptions.push(holiday_date);
}
}
}
}
}
exceptions
}
/// Generate EXDATE property for holiday exceptions
fn generate_exdate_property(
exceptions: &[NaiveDate],
start_time: chrono::DateTime<Utc>,
) -> Option<String> {
if exceptions.is_empty() {
return None;
}
let mut exdate_values = Vec::new();
for &exception_date in exceptions {
// Create a datetime for the exception using the same time as the start time
let exception_datetime = exception_date.and_time(start_time.time()).and_utc();
let exdate_str = exception_datetime.format("%Y%m%dT%H%M%SZ").to_string();
exdate_values.push(exdate_str);
}
Some(format!("EXDATE:{}", exdate_values.join(",")))
}
/// Escape text for ICS format
fn escape_ics_text(text: &str) -> String {
text.replace("\\", "\\\\")
.replace(";", "\\;")
.replace(",", "\\,")
.replace("\n", "\\n")
.replace("\r", "")
}

View File

@@ -1,13 +1,13 @@
//! Bot commands module.
pub mod gcal;
pub mod ics;
pub mod search;
pub mod terms;
pub mod time;
pub mod ics;
pub mod gcal;
pub use gcal::gcal;
pub use ics::ics;
pub use search::search;
pub use terms::terms;
pub use time::time;
pub use ics::ics;
pub use gcal::gcal;

View File

@@ -1,8 +1,10 @@
//! Course search command implementation.
use crate::banner::SearchQuery;
use crate::banner::{SearchQuery, Term};
use crate::bot::{Context, Error};
use anyhow::anyhow;
use regex::Regex;
use tracing::info;
/// Search for courses with various filters
#[poise::command(slash_command, prefix_command)]
@@ -40,12 +42,37 @@ pub async fn search(
query = query.max_results(max_results.min(25)); // Cap at 25
}
// TODO: Get current term dynamically
// TODO: Get BannerApi from context or global state
// For now, we'll return an error
ctx.say("Search functionality not yet implemented - BannerApi integration needed")
let term = Term::get_current().inner().to_string();
let search_result = ctx
.data()
.app_state
.banner_api
.search(&term, &query, "subjectDescription", false)
.await?;
let response = if let Some(courses) = search_result.data {
if courses.is_empty() {
"No courses found with the specified criteria.".to_string()
} else {
courses
.iter()
.map(|course| {
format!(
"**{}**: {} ({})",
course.display_title(),
course.primary_instructor_name(),
course.course_reference_number
)
})
.collect::<Vec<_>>()
.join("\n")
}
} else {
"No courses found with the specified criteria.".to_string()
};
ctx.say(response).await?;
info!("search command completed");
Ok(())
}
@@ -65,22 +92,22 @@ fn parse_course_code(input: &str) -> Result<(i32, i32), Error> {
};
if low > high {
return Err("Invalid range: low value greater than high value".into());
return Err(anyhow!("Invalid range: low value greater than high value"));
}
if low < 1000 || high > 9999 {
return Err("Course codes must be between 1000 and 9999".into());
return Err(anyhow!("Course codes must be between 1000 and 9999"));
}
return Ok((low, high));
}
return Err("Invalid range format".into());
return Err(anyhow!("Invalid range format"));
}
// Handle wildcard format (e.g, "34xx")
if input.contains('x') {
if input.len() != 4 {
return Err("Wildcard format must be exactly 4 characters".into());
return Err(anyhow!("Wildcard format must be exactly 4 characters"));
}
let re = Regex::new(r"(\d+)(x+)").unwrap();
@@ -92,22 +119,22 @@ fn parse_course_code(input: &str) -> Result<(i32, i32), Error> {
let high = low + 10_i32.pow(x_count as u32) - 1;
if low < 1000 || high > 9999 {
return Err("Course codes must be between 1000 and 9999".into());
return Err(anyhow!("Course codes must be between 1000 and 9999"));
}
return Ok((low, high));
}
return Err("Invalid wildcard format".into());
return Err(anyhow!("Invalid wildcard format"));
}
// Handle single course code
if input.len() == 4 {
let code: i32 = input.parse()?;
if code < 1000 || code > 9999 {
return Err("Course codes must be between 1000 and 9999".into());
if !(1000..=9999).contains(&code) {
return Err(anyhow!("Course codes must be between 1000 and 9999"));
}
return Ok((code, code));
}
Err("Invalid course code format".into())
Err(anyhow!("Invalid course code format"))
}

View File

@@ -1,6 +1,8 @@
//! Terms command implementation.
use crate::banner::{BannerTerm, Term};
use crate::bot::{Context, Error};
use tracing::info;
/// List available terms or search for a specific term
#[poise::command(slash_command, prefix_command)]
@@ -13,14 +15,45 @@ pub async fn terms(
let search_term = search.unwrap_or_default();
let page_number = page.unwrap_or(1).max(1);
let max_results = 10;
// TODO: Get BannerApi from context or global state
// For now, we'll return a placeholder response
ctx.say(format!(
"Terms command not yet implemented - BannerApi integration needed\nSearch: '{}', Page: {}",
search_term, page_number
))
.await?;
let terms = ctx
.data()
.app_state
.banner_api
.sessions
.get_terms(&search_term, page_number, max_results)
.await?;
let response = if terms.is_empty() {
"No terms found.".to_string()
} else {
let current_term_code = Term::get_current().inner().to_string();
terms
.iter()
.map(|term| format_term(term, &current_term_code))
.collect::<Vec<_>>()
.join("\n")
};
ctx.say(response).await?;
info!("terms command completed");
Ok(())
}
fn format_term(term: &BannerTerm, current_term_code: &str) -> String {
let is_current = if term.code == current_term_code {
" (current)"
} else {
""
};
let is_archived = if term.is_archived() {
" (archived)"
} else {
""
};
format!(
"- `{}`: {}{}{}",
term.code, term.description, is_current, is_archived
)
}

View File

@@ -1,6 +1,7 @@
//! Time command implementation for course meeting times.
use crate::bot::{Context, Error};
use crate::bot::{Context, Error, utils};
use tracing::info;
/// Get meeting times for a specific course
#[poise::command(slash_command, prefix_command)]
@@ -10,16 +11,15 @@ pub async fn time(
) -> Result<(), Error> {
ctx.defer().await?;
// TODO: Get BannerApi from context or global state
// TODO: Get current term dynamically
let term = 202510; // Hardcoded for now
let course = utils::get_course_by_crn(&ctx, crn).await?;
// TODO: Implement actual meeting time retrieval
// TODO: Implement actual meeting time retrieval and display
ctx.say(format!(
"Time command not yet implemented - BannerApi integration needed\nCRN: {}, Term: {}",
crn, term
"Meeting time display for '{}' is not yet implemented.",
course.display_title()
))
.await?;
info!(crn = %crn, "time command completed");
Ok(())
}

View File

@@ -1,12 +1,12 @@
use crate::app_state::AppState;
use crate::error::Error;
use crate::state::AppState;
pub mod commands;
pub mod utils;
#[derive(Debug)]
pub struct Data {
pub app_state: AppState,
} // User data, which is stored and accessible in all command invocations
pub type Error = Box<dyn std::error::Error + Send + Sync>;
pub type Context<'a> = poise::Context<'a, Data, Error>;
/// Get all available commands

24
src/bot/utils.rs Normal file
View File

@@ -0,0 +1,24 @@
//! Bot command utilities.
use crate::banner::{Course, Term};
use crate::bot::Context;
use crate::error::Result;
use tracing::error;
/// Gets a course by its CRN for the current term.
pub async fn get_course_by_crn(ctx: &Context<'_>, crn: i32) -> Result<Course> {
let app_state = &ctx.data().app_state;
// Get current term dynamically
let current_term_status = Term::get_current();
let term = current_term_status.inner();
// Fetch live course data from database via AppState
app_state
.get_course_or_fetch(&term.to_string(), &crn.to_string())
.await
.map_err(|e| {
error!(error = %e, crn = %crn, "failed to fetch course data");
e
})
}

104
src/cli.rs Normal file
View File

@@ -0,0 +1,104 @@
use clap::Parser;
/// Banner Discord Bot - Course availability monitoring
///
/// This application runs multiple services that can be controlled via CLI arguments:
/// - bot: Discord bot for course monitoring commands
/// - web: HTTP server for web interface and API
/// - scraper: Background service for scraping course data
///
/// Use --services to specify which services to run, or --disable-services to exclude specific services.
#[derive(Parser, Debug)]
#[command(author, version, about, long_about = None)]
pub struct Args {
/// Log formatter to use
#[arg(long, value_enum, default_value_t = default_tracing_format())]
pub tracing: TracingFormat,
/// Services to run (comma-separated). Default: all services
///
/// Examples:
/// --services bot,web # Run only bot and web services
/// --services scraper # Run only the scraper service
#[arg(long, value_delimiter = ',', conflicts_with = "disable_services")]
pub services: Option<Vec<ServiceName>>,
/// Services to disable (comma-separated)
///
/// Examples:
/// --disable-services bot # Run web and scraper only
/// --disable-services bot,web # Run only the scraper service
#[arg(long, value_delimiter = ',', conflicts_with = "services")]
pub disable_services: Option<Vec<ServiceName>>,
}
#[derive(clap::ValueEnum, Clone, Debug)]
pub enum TracingFormat {
/// Use pretty formatter (default in debug mode)
Pretty,
/// Use JSON formatter (default in release mode)
Json,
}
#[derive(clap::ValueEnum, Clone, Debug, PartialEq)]
pub enum ServiceName {
/// Discord bot for course monitoring commands
Bot,
/// HTTP server for web interface and API
Web,
/// Background service for scraping course data
Scraper,
}
impl ServiceName {
/// Get all available services
pub fn all() -> Vec<ServiceName> {
vec![ServiceName::Bot, ServiceName::Web, ServiceName::Scraper]
}
/// Convert to string for service registration
pub fn as_str(&self) -> &'static str {
match self {
ServiceName::Bot => "bot",
ServiceName::Web => "web",
ServiceName::Scraper => "scraper",
}
}
}
/// Determine which services should be enabled based on CLI arguments
pub fn determine_enabled_services(args: &Args) -> Result<Vec<ServiceName>, anyhow::Error> {
match (&args.services, &args.disable_services) {
(Some(services), None) => {
// User specified which services to run
Ok(services.clone())
}
(None, Some(disabled)) => {
// User specified which services to disable
let enabled: Vec<ServiceName> = ServiceName::all()
.into_iter()
.filter(|s| !disabled.contains(s))
.collect();
Ok(enabled)
}
(None, None) => {
// Default: run all services
Ok(ServiceName::all())
}
(Some(_), Some(_)) => {
// This should be prevented by clap's conflicts_with, but just in case
Err(anyhow::anyhow!(
"Cannot specify both --services and --disable-services"
))
}
}
}
#[cfg(debug_assertions)]
const DEFAULT_TRACING_FORMAT: TracingFormat = TracingFormat::Pretty;
#[cfg(not(debug_assertions))]
const DEFAULT_TRACING_FORMAT: TracingFormat = TracingFormat::Json;
fn default_tracing_format() -> TracingFormat {
DEFAULT_TRACING_FORMAT
}

View File

@@ -3,28 +3,28 @@
//! This module handles loading and parsing configuration from environment variables
//! using the figment crate. It supports flexible duration parsing that accepts both
//! numeric values (interpreted as seconds) and duration strings with units.
//!
//! All configuration is loaded from environment variables with the `APP_` prefix:
use fundu::{DurationParser, TimeUnit};
use serde::{Deserialize, Deserializer};
use std::time::Duration;
/// Application configuration loaded from environment variables
/// Main application configuration containing all sub-configurations
#[derive(Deserialize)]
pub struct Config {
/// Discord bot token for authentication
pub bot_token: String,
/// Log level for the application
///
/// This value is used to set the log level for this application's target specifically.
/// e.g. "debug" would be similar to "warn,banner=debug,..."
///
/// Valid values are: "trace", "debug", "info", "warn", "error"
/// Defaults to "info" if not specified
#[serde(default = "default_log_level")]
pub log_level: String,
/// Port for the web server (default: 8080)
#[serde(default = "default_port")]
pub port: u16,
/// Database connection URL
pub database_url: String,
/// Redis connection URL
pub redis_url: String,
/// Base URL for banner generation service
pub banner_base_url: String,
/// Target Discord guild ID where the bot operates
pub bot_target_guild: u64,
/// Discord application ID
pub bot_app_id: u64,
/// Graceful shutdown timeout duration
///
/// Accepts both numeric values (seconds) and duration strings
@@ -34,6 +34,29 @@ pub struct Config {
deserialize_with = "deserialize_duration"
)]
pub shutdown_timeout: Duration,
/// Discord bot token for authentication
pub bot_token: String,
/// Target Discord guild ID where the bot operates
pub bot_target_guild: u64,
/// Base URL for banner generation service
///
/// Defaults to "https://ssbprod.utsa.edu/StudentRegistrationSsb/ssb" if not specified
#[serde(default = "default_banner_base_url")]
pub banner_base_url: String,
/// Rate limiting configuration for Banner API requests
#[serde(default = "default_rate_limiting")]
pub rate_limiting: RateLimitingConfig,
}
/// Default log level of "info"
fn default_log_level() -> String {
"info".to_string()
}
/// Default port of 8080
fn default_port() -> u16 {
8080
}
/// Default shutdown timeout of 8 seconds
@@ -41,6 +64,67 @@ fn default_shutdown_timeout() -> Duration {
Duration::from_secs(8)
}
/// Default banner base URL
fn default_banner_base_url() -> String {
"https://ssbprod.utsa.edu/StudentRegistrationSsb/ssb".to_string()
}
/// Rate limiting configuration for Banner API requests
#[derive(Deserialize, Clone, Debug)]
pub struct RateLimitingConfig {
/// Requests per minute for session operations (very conservative)
#[serde(default = "default_session_rpm")]
pub session_rpm: u32,
/// Requests per minute for search operations (moderate)
#[serde(default = "default_search_rpm")]
pub search_rpm: u32,
/// Requests per minute for metadata operations (moderate)
#[serde(default = "default_metadata_rpm")]
pub metadata_rpm: u32,
/// Requests per minute for reset operations (low priority)
#[serde(default = "default_reset_rpm")]
pub reset_rpm: u32,
/// Burst allowance (extra requests allowed in short bursts)
#[serde(default = "default_burst_allowance")]
pub burst_allowance: u32,
}
/// Default rate limiting configuration
fn default_rate_limiting() -> RateLimitingConfig {
RateLimitingConfig {
session_rpm: default_session_rpm(),
search_rpm: default_search_rpm(),
metadata_rpm: default_metadata_rpm(),
reset_rpm: default_reset_rpm(),
burst_allowance: default_burst_allowance(),
}
}
/// Default session requests per minute (6 = 1 every 10 seconds)
fn default_session_rpm() -> u32 {
6
}
/// Default search requests per minute (30 = 1 every 2 seconds)
fn default_search_rpm() -> u32 {
30
}
/// Default metadata requests per minute (20 = 1 every 3 seconds)
fn default_metadata_rpm() -> u32 {
20
}
/// Default reset requests per minute (10 = 1 every 6 seconds)
fn default_reset_rpm() -> u32 {
10
}
/// Default burst allowance (3 extra requests)
fn default_burst_allowance() -> u32 {
3
}
/// Duration parser configured to handle various time units with seconds as default
///
/// Supports:

135
src/data/batch.rs Normal file
View File

@@ -0,0 +1,135 @@
//! Batch database operations for improved performance.
use crate::banner::Course;
use crate::error::Result;
use sqlx::PgPool;
use std::time::Instant;
use tracing::info;
/// Batch upsert courses in a single database query.
///
/// This function performs a bulk INSERT...ON CONFLICT DO UPDATE for all courses
/// in a single round-trip to the database, significantly reducing overhead compared
/// to individual inserts.
///
/// # Performance
/// - Reduces N database round-trips to 1
/// - Typical usage: 50-200 courses per batch
/// - PostgreSQL parameter limit: 65,535 (we use ~10 per course)
///
/// # Arguments
/// * `courses` - Slice of Course structs from the Banner API
/// * `db_pool` - PostgreSQL connection pool
///
/// # Returns
/// * `Ok(())` on success
/// * `Err(_)` if the database operation fails
///
/// # Example
/// ```no_run
/// use banner::data::batch::batch_upsert_courses;
/// use banner::banner::Course;
/// use sqlx::PgPool;
///
/// async fn example(courses: &[Course], pool: &PgPool) -> anyhow::Result<()> {
/// batch_upsert_courses(courses, pool).await?;
/// Ok(())
/// }
/// ```
pub async fn batch_upsert_courses(courses: &[Course], db_pool: &PgPool) -> Result<()> {
// Early return for empty batches
if courses.is_empty() {
info!("No courses to upsert, skipping batch operation");
return Ok(());
}
let start = Instant::now();
let course_count = courses.len();
// Extract course fields into vectors for UNNEST
let crns: Vec<&str> = courses
.iter()
.map(|c| c.course_reference_number.as_str())
.collect();
let subjects: Vec<&str> = courses.iter().map(|c| c.subject.as_str()).collect();
let course_numbers: Vec<&str> = courses.iter().map(|c| c.course_number.as_str()).collect();
let titles: Vec<&str> = courses.iter().map(|c| c.course_title.as_str()).collect();
let term_codes: Vec<&str> = courses.iter().map(|c| c.term.as_str()).collect();
let enrollments: Vec<i32> = courses.iter().map(|c| c.enrollment).collect();
let max_enrollments: Vec<i32> = courses.iter().map(|c| c.maximum_enrollment).collect();
let wait_counts: Vec<i32> = courses.iter().map(|c| c.wait_count).collect();
let wait_capacities: Vec<i32> = courses.iter().map(|c| c.wait_capacity).collect();
// Perform batch upsert using UNNEST for efficient bulk insertion
let result = sqlx::query(
r#"
INSERT INTO courses (
crn, subject, course_number, title, term_code,
enrollment, max_enrollment, wait_count, wait_capacity, last_scraped_at
)
SELECT * FROM UNNEST(
$1::text[], $2::text[], $3::text[], $4::text[], $5::text[],
$6::int4[], $7::int4[], $8::int4[], $9::int4[],
array_fill(NOW()::timestamptz, ARRAY[$10])
) AS t(
crn, subject, course_number, title, term_code,
enrollment, max_enrollment, wait_count, wait_capacity, last_scraped_at
)
ON CONFLICT (crn, term_code)
DO UPDATE SET
subject = EXCLUDED.subject,
course_number = EXCLUDED.course_number,
title = EXCLUDED.title,
enrollment = EXCLUDED.enrollment,
max_enrollment = EXCLUDED.max_enrollment,
wait_count = EXCLUDED.wait_count,
wait_capacity = EXCLUDED.wait_capacity,
last_scraped_at = EXCLUDED.last_scraped_at
"#,
)
.bind(&crns)
.bind(&subjects)
.bind(&course_numbers)
.bind(&titles)
.bind(&term_codes)
.bind(&enrollments)
.bind(&max_enrollments)
.bind(&wait_counts)
.bind(&wait_capacities)
.bind(course_count as i32)
.execute(db_pool)
.await
.map_err(|e| anyhow::anyhow!("Failed to batch upsert courses: {}", e))?;
let duration = start.elapsed();
info!(
courses_count = course_count,
rows_affected = result.rows_affected(),
duration_ms = duration.as_millis(),
"Batch upserted courses"
);
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_empty_batch_returns_ok() {
// This is a basic compile-time test
// Runtime tests would require sqlx::test macro and a test database
let courses: Vec<Course> = vec![];
assert_eq!(courses.len(), 0);
}
}

4
src/data/mod.rs Normal file
View File

@@ -0,0 +1,4 @@
//! Database models and schema.
pub mod batch;
pub mod models;

79
src/data/models.rs Normal file
View File

@@ -0,0 +1,79 @@
//! `sqlx` models for the database schema.
use chrono::{DateTime, Utc};
use serde_json::Value;
#[allow(dead_code)]
#[derive(sqlx::FromRow, Debug, Clone)]
pub struct Course {
pub id: i32,
pub crn: String,
pub subject: String,
pub course_number: String,
pub title: String,
pub term_code: String,
pub enrollment: i32,
pub max_enrollment: i32,
pub wait_count: i32,
pub wait_capacity: i32,
pub last_scraped_at: DateTime<Utc>,
}
#[allow(dead_code)]
#[derive(sqlx::FromRow, Debug, Clone)]
pub struct CourseMetric {
pub id: i32,
pub course_id: i32,
pub timestamp: DateTime<Utc>,
pub enrollment: i32,
pub wait_count: i32,
pub seats_available: i32,
}
#[allow(dead_code)]
#[derive(sqlx::FromRow, Debug, Clone)]
pub struct CourseAudit {
pub id: i32,
pub course_id: i32,
pub timestamp: DateTime<Utc>,
pub field_changed: String,
pub old_value: String,
pub new_value: String,
}
/// The priority level of a scrape job.
#[derive(sqlx::Type, Copy, Debug, Clone)]
#[sqlx(type_name = "scrape_priority", rename_all = "PascalCase")]
pub enum ScrapePriority {
Low,
Medium,
High,
Critical,
}
/// The type of target for a scrape job, determining how the payload is interpreted.
#[derive(sqlx::Type, Copy, Debug, Clone)]
#[sqlx(type_name = "target_type", rename_all = "PascalCase")]
pub enum TargetType {
Subject,
CourseRange,
CrnList,
SingleCrn,
}
/// Represents a queryable job from the database.
#[allow(dead_code)]
#[derive(sqlx::FromRow, Debug, Clone)]
pub struct ScrapeJob {
pub id: i32,
pub target_type: TargetType,
pub target_payload: Value,
pub priority: ScrapePriority,
pub execute_at: DateTime<Utc>,
pub created_at: DateTime<Utc>,
pub locked_at: Option<DateTime<Utc>>,
/// Number of retry attempts for this job (non-negative, enforced by CHECK constraint)
pub retry_count: i32,
/// Maximum number of retry attempts allowed (non-negative, enforced by CHECK constraint)
pub max_retries: i32,
}

View File

@@ -0,0 +1,4 @@
//! Application-specific error types.
pub type Error = anyhow::Error;
pub type Result<T, E = Error> = anyhow::Result<T, E>;

275
src/formatter.rs Normal file
View File

@@ -0,0 +1,275 @@
//! Custom tracing formatter
use serde::Serialize;
use serde_json::{Map, Value};
use std::fmt;
use time::macros::format_description;
use time::{OffsetDateTime, format_description::FormatItem};
use tracing::field::{Field, Visit};
use tracing::{Event, Level, Subscriber};
use tracing_subscriber::fmt::format::Writer;
use tracing_subscriber::fmt::{FmtContext, FormatEvent, FormatFields, FormattedFields};
use tracing_subscriber::registry::LookupSpan;
use yansi::Paint;
/// Cached format description for timestamps
/// Uses 3 subsecond digits on Emscripten, 5 otherwise for better performance
#[cfg(target_os = "emscripten")]
const TIMESTAMP_FORMAT: &[FormatItem<'static>] =
format_description!("[hour]:[minute]:[second].[subsecond digits:3]");
#[cfg(not(target_os = "emscripten"))]
const TIMESTAMP_FORMAT: &[FormatItem<'static>] =
format_description!("[hour]:[minute]:[second].[subsecond digits:5]");
/// A custom formatter with enhanced timestamp formatting
///
/// Re-implementation of the Full formatter with improved timestamp display.
pub struct CustomPrettyFormatter;
impl<S, N> FormatEvent<S, N> for CustomPrettyFormatter
where
S: Subscriber + for<'a> LookupSpan<'a>,
N: for<'a> FormatFields<'a> + 'static,
{
fn format_event(
&self,
ctx: &FmtContext<'_, S, N>,
mut writer: Writer<'_>,
event: &Event<'_>,
) -> fmt::Result {
let meta = event.metadata();
// 1) Timestamp (dimmed when ANSI)
let now = OffsetDateTime::now_utc();
let formatted_time = now.format(&TIMESTAMP_FORMAT).map_err(|e| {
eprintln!("Failed to format timestamp: {}", e);
fmt::Error
})?;
write_dimmed(&mut writer, formatted_time)?;
writer.write_char(' ')?;
// 2) Colored 5-char level like Full
write_colored_level(&mut writer, meta.level())?;
writer.write_char(' ')?;
// 3) Span scope chain (bold names, fields in braces, dimmed ':')
if let Some(scope) = ctx.event_scope() {
let mut saw_any = false;
for span in scope.from_root() {
write_bold(&mut writer, span.metadata().name())?;
saw_any = true;
write_dimmed(&mut writer, ":")?;
let ext = span.extensions();
if let Some(fields) = &ext.get::<FormattedFields<N>>()
&& !fields.fields.is_empty()
{
write_bold(&mut writer, "{")?;
writer.write_str(fields.fields.as_str())?;
write_bold(&mut writer, "}")?;
}
write_dimmed(&mut writer, ":")?;
}
if saw_any {
writer.write_char(' ')?;
}
}
// 4) Target (dimmed), then a space
if writer.has_ansi_escapes() {
write!(writer, "{}: ", Paint::new(meta.target()).dim())?;
} else {
write!(writer, "{}: ", meta.target())?;
}
// 5) Event fields
ctx.format_fields(writer.by_ref(), event)?;
// 6) Newline
writeln!(writer)
}
}
/// A custom JSON formatter that flattens fields to root level
///
/// Outputs logs in the format: { "message": "...", "level": "...", "customAttribute": "..." }
pub struct CustomJsonFormatter;
impl<S, N> FormatEvent<S, N> for CustomJsonFormatter
where
S: Subscriber + for<'a> LookupSpan<'a>,
N: for<'a> FormatFields<'a> + 'static,
{
fn format_event(
&self,
ctx: &FmtContext<'_, S, N>,
mut writer: Writer<'_>,
event: &Event<'_>,
) -> fmt::Result {
let meta = event.metadata();
#[derive(Serialize)]
struct EventFields {
message: String,
level: String,
target: String,
#[serde(flatten)]
spans: Map<String, Value>,
#[serde(flatten)]
fields: Map<String, Value>,
}
let (message, fields, spans) = {
let mut message: Option<String> = None;
let mut fields: Map<String, Value> = Map::new();
let mut spans: Map<String, Value> = Map::new();
struct FieldVisitor<'a> {
message: &'a mut Option<String>,
fields: &'a mut Map<String, Value>,
}
impl<'a> Visit for FieldVisitor<'a> {
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
let key = field.name();
if key == "message" {
*self.message = Some(format!("{:?}", value));
} else {
// Use typed methods for better performance
self.fields
.insert(key.to_string(), Value::String(format!("{:?}", value)));
}
}
fn record_str(&mut self, field: &Field, value: &str) {
let key = field.name();
if key == "message" {
*self.message = Some(value.to_string());
} else {
self.fields
.insert(key.to_string(), Value::String(value.to_string()));
}
}
fn record_i64(&mut self, field: &Field, value: i64) {
let key = field.name();
if key != "message" {
self.fields.insert(
key.to_string(),
Value::Number(serde_json::Number::from(value)),
);
}
}
fn record_u64(&mut self, field: &Field, value: u64) {
let key = field.name();
if key != "message" {
self.fields.insert(
key.to_string(),
Value::Number(serde_json::Number::from(value)),
);
}
}
fn record_bool(&mut self, field: &Field, value: bool) {
let key = field.name();
if key != "message" {
self.fields.insert(key.to_string(), Value::Bool(value));
}
}
}
let mut visitor = FieldVisitor {
message: &mut message,
fields: &mut fields,
};
event.record(&mut visitor);
// Collect span information from the span hierarchy
if let Some(scope) = ctx.event_scope() {
for span in scope.from_root() {
let span_name = span.metadata().name().to_string();
let mut span_fields: Map<String, Value> = Map::new();
// Try to extract fields from FormattedFields
let ext = span.extensions();
if let Some(formatted_fields) = ext.get::<FormattedFields<N>>() {
// Try to parse as JSON first
if let Ok(json_fields) = serde_json::from_str::<Map<String, Value>>(
formatted_fields.fields.as_str(),
) {
span_fields.extend(json_fields);
} else {
// If not valid JSON, treat the entire field string as a single field
span_fields.insert(
"raw".to_string(),
Value::String(formatted_fields.fields.as_str().to_string()),
);
}
}
// Insert span as a nested object directly into the spans map
spans.insert(span_name, Value::Object(span_fields));
}
}
(message, fields, spans)
};
let json = EventFields {
message: message.unwrap_or_default(),
level: meta.level().to_string(),
target: meta.target().to_string(),
spans,
fields,
};
writeln!(
writer,
"{}",
serde_json::to_string(&json).unwrap_or_else(|_| "{}".to_string())
)
}
}
/// Write the verbosity level with the same coloring/alignment as the Full formatter.
fn write_colored_level(writer: &mut Writer<'_>, level: &Level) -> fmt::Result {
if writer.has_ansi_escapes() {
let paint = match *level {
Level::TRACE => Paint::new("TRACE").magenta(),
Level::DEBUG => Paint::new("DEBUG").blue(),
Level::INFO => Paint::new(" INFO").green(),
Level::WARN => Paint::new(" WARN").yellow(),
Level::ERROR => Paint::new("ERROR").red(),
};
write!(writer, "{}", paint)
} else {
// Right-pad to width 5 like Full's non-ANSI mode
match *level {
Level::TRACE => write!(writer, "{:>5}", "TRACE"),
Level::DEBUG => write!(writer, "{:>5}", "DEBUG"),
Level::INFO => write!(writer, "{:>5}", " INFO"),
Level::WARN => write!(writer, "{:>5}", " WARN"),
Level::ERROR => write!(writer, "{:>5}", "ERROR"),
}
}
}
fn write_dimmed(writer: &mut Writer<'_>, s: impl fmt::Display) -> fmt::Result {
if writer.has_ansi_escapes() {
write!(writer, "{}", Paint::new(s).dim())
} else {
write!(writer, "{}", s)
}
}
fn write_bold(writer: &mut Writer<'_>, s: impl fmt::Display) -> fmt::Result {
if writer.has_ansi_escapes() {
write!(writer, "{}", Paint::new(s).bold())
} else {
write!(writer, "{}", s)
}
}

View File

@@ -1,5 +1,15 @@
pub mod app_state;
pub mod app;
pub mod banner;
pub mod bot;
pub mod cli;
pub mod config;
pub mod data;
pub mod error;
pub mod formatter;
pub mod logging;
pub mod scraper;
pub mod services;
pub mod signals;
pub mod state;
pub mod utils;
pub mod web;

47
src/logging.rs Normal file
View File

@@ -0,0 +1,47 @@
use crate::cli::TracingFormat;
use crate::config::Config;
use crate::formatter;
use tracing_subscriber::fmt::format::JsonFields;
use tracing_subscriber::{EnvFilter, FmtSubscriber};
/// Configure and initialize logging for the application
pub fn setup_logging(config: &Config, tracing_format: TracingFormat) {
// Configure logging based on config
// Note: Even when base_level is trace or debug, we suppress trace logs from noisy
// infrastructure modules to keep output readable. These modules use debug for important
// events and trace only for very detailed debugging.
let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| {
let base_level = &config.log_level;
EnvFilter::new(format!(
"warn,banner={},banner::rate_limiter=warn,banner::session=debug,banner::rate_limit_middleware=warn,banner::middleware=debug",
base_level
))
});
// Select formatter based on CLI args
let use_pretty = match tracing_format {
TracingFormat::Pretty => true,
TracingFormat::Json => false,
};
let subscriber: Box<dyn tracing::Subscriber + Send + Sync> = if use_pretty {
Box::new(
FmtSubscriber::builder()
.with_target(true)
.event_format(formatter::CustomPrettyFormatter)
.with_env_filter(filter)
.finish(),
)
} else {
Box::new(
FmtSubscriber::builder()
.with_target(true)
.event_format(formatter::CustomJsonFormatter)
.fmt_fields(JsonFields::new())
.with_env_filter(filter)
.finish(),
)
};
tracing::subscriber::set_global_default(subscriber).expect("setting default subscriber failed");
}

View File

@@ -1,169 +1,70 @@
use serenity::all::{ClientBuilder, GatewayIntents};
use tokio::signal;
use tracing::{debug, error, info, warn};
use tracing_subscriber::{EnvFilter, FmtSubscriber};
use crate::app::App;
use crate::cli::{Args, ServiceName, determine_enabled_services};
use crate::logging::setup_logging;
use clap::Parser;
use std::process::ExitCode;
use tracing::info;
use crate::app_state::AppState;
use crate::banner::BannerApi;
use crate::bot::{Data, get_commands};
use crate::config::Config;
use crate::services::manager::ServiceManager;
use crate::services::{ServiceResult, bot::BotService, run_service};
use figment::{Figment, providers::Env};
mod app_state;
mod app;
mod banner;
mod bot;
mod cli;
mod config;
mod data;
mod error;
mod formatter;
mod logging;
mod scraper;
mod services;
mod signals;
mod state;
mod web;
#[tokio::main]
async fn main() {
async fn main() -> ExitCode {
dotenvy::dotenv().ok();
// Configure logging
let filter =
EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("warn,banner=debug"));
let subscriber = {
#[cfg(debug_assertions)]
{
FmtSubscriber::builder()
}
#[cfg(not(debug_assertions))]
{
FmtSubscriber::builder().json()
}
}
.with_env_filter(filter)
.finish();
tracing::subscriber::set_global_default(subscriber).expect("setting default subscriber failed");
// Parse CLI arguments
let args = Args::parse();
let config: Config = Figment::new()
.merge(Env::prefixed("APP_"))
.extract()
.expect("Failed to load config");
// Determine which services should be enabled
let enabled_services: Vec<ServiceName> =
determine_enabled_services(&args).expect("Failed to determine enabled services");
// Create BannerApi and AppState
let banner_api =
BannerApi::new(config.banner_base_url.clone()).expect("Failed to create BannerApi");
banner_api
.setup()
.await
.expect("Failed to set up BannerApi session");
info!(
enabled_services = ?enabled_services,
"services configuration loaded"
);
let app_state =
AppState::new(banner_api, &config.redis_url).expect("Failed to create AppState");
// Create and initialize the application
let mut app = App::new().await.expect("Failed to initialize application");
// Configure the client with your Discord bot token in the environment
let intents = GatewayIntents::non_privileged();
// Setup logging
setup_logging(app.config(), args.tracing);
let bot_target_guild = config.bot_target_guild;
// Log application startup context
info!(
version = env!("CARGO_PKG_VERSION"),
environment = if cfg!(debug_assertions) {
"development"
} else {
"production"
},
"starting banner"
);
let framework = poise::Framework::builder()
.options(poise::FrameworkOptions {
commands: get_commands(),
..Default::default()
})
.setup(move |ctx, _ready, framework| {
let app_state = app_state.clone();
Box::pin(async move {
poise::builtins::register_in_guild(
ctx,
&framework.options().commands,
bot_target_guild.into(),
)
.await?;
poise::builtins::register_globally(ctx, &framework.options().commands).await?;
Ok(Data { app_state })
})
})
.build();
// Setup services (web, scraper)
app.setup_services(&enabled_services)
.expect("Failed to setup services");
let client = ClientBuilder::new(config.bot_token, intents)
.framework(framework)
.await
.expect("Failed to build client");
// Extract shutdown timeout before moving config
let shutdown_timeout = config.shutdown_timeout;
// Create service manager
let mut service_manager = ServiceManager::new();
// Create and add services
let bot_service = Box::new(BotService::new(client));
let bot_handle = tokio::spawn(run_service(bot_service, service_manager.subscribe()));
service_manager.add_service("bot".to_string(), bot_handle);
// Set up CTRL+C signal handling
let ctrl_c = async {
signal::ctrl_c()
// Setup bot service if enabled
if enabled_services.contains(&ServiceName::Bot) {
app.setup_bot_service()
.await
.expect("Failed to install CTRL+C signal handler");
info!("Received CTRL+C, gracefully shutting down...");
};
// Main application loop - wait for services or CTRL+C
let mut exit_code = 0;
let join = |strings: Vec<String>| {
strings
.iter()
.map(|s| format!("\"{}\"", s))
.collect::<Vec<_>>()
.join(", ")
};
tokio::select! {
(service_name, result) = service_manager.run() => {
// A service completed unexpectedly
match result {
ServiceResult::GracefulShutdown => {
info!(service = service_name, "Service completed gracefully");
}
ServiceResult::NormalCompletion => {
warn!(service = service_name, "Service completed unexpectedly");
exit_code = 1;
}
ServiceResult::Error(e) => {
error!(service = service_name, "Service failed: {e}");
exit_code = 1;
}
}
// Shutdown remaining services
match service_manager.shutdown(shutdown_timeout).await {
Ok(()) => {
debug!("Graceful shutdown complete");
}
Err(pending_services) => {
warn!(
"Graceful shutdown elapsed - the following service(s) did not complete: {}",
join(pending_services)
);
exit_code = if exit_code == 0 { 2 } else { exit_code };
}
}
}
_ = ctrl_c => {
// User requested shutdown
match service_manager.shutdown(shutdown_timeout).await {
Ok(()) => {
debug!("Graceful shutdown complete");
}
Err(pending_services) => {
warn!(
"Graceful shutdown elapsed - the following service(s) did not complete: {}",
join(pending_services)
);
exit_code = 2;
}
}
}
.expect("Failed to setup bot service");
}
info!(exit_code = exit_code, "Shutdown complete");
std::process::exit(exit_code);
// Start all services and run the application
app.start_services();
app.run().await
}

104
src/scraper/jobs/mod.rs Normal file
View File

@@ -0,0 +1,104 @@
pub mod subject;
use crate::banner::BannerApi;
use crate::data::models::TargetType;
use crate::error::Result;
use serde::{Deserialize, Serialize};
use sqlx::PgPool;
use std::fmt;
/// Errors that can occur during job parsing
#[derive(Debug)]
pub enum JobParseError {
InvalidJson(serde_json::Error),
UnsupportedTargetType(TargetType),
}
impl fmt::Display for JobParseError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
JobParseError::InvalidJson(e) => write!(f, "Invalid JSON in job payload: {}", e),
JobParseError::UnsupportedTargetType(t) => {
write!(f, "Unsupported target type: {:?}", t)
}
}
}
}
impl std::error::Error for JobParseError {
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
match self {
JobParseError::InvalidJson(e) => Some(e),
_ => None,
}
}
}
/// Errors that can occur during job processing
#[derive(Debug)]
pub enum JobError {
Recoverable(anyhow::Error), // API failures, network issues
Unrecoverable(anyhow::Error), // Parse errors, corrupted data
}
impl fmt::Display for JobError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
JobError::Recoverable(e) => write!(f, "Recoverable error: {}", e),
JobError::Unrecoverable(e) => write!(f, "Unrecoverable error: {}", e),
}
}
}
impl std::error::Error for JobError {
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
match self {
JobError::Recoverable(e) => e.source(),
JobError::Unrecoverable(e) => e.source(),
}
}
}
/// Common trait interface for all job types
#[async_trait::async_trait]
pub trait Job: Send + Sync {
/// The target type this job handles
#[allow(dead_code)]
fn target_type(&self) -> TargetType;
/// Process the job with the given API client and database pool
async fn process(&self, banner_api: &BannerApi, db_pool: &PgPool) -> Result<()>;
/// Get a human-readable description of the job
fn description(&self) -> String;
}
/// Main job enum that dispatches to specific job implementations
#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum JobType {
Subject(subject::SubjectJob),
}
impl JobType {
/// Create a job from the target type and payload
pub fn from_target_type_and_payload(
target_type: TargetType,
payload: serde_json::Value,
) -> Result<Self, JobParseError> {
match target_type {
TargetType::Subject => {
let subject_job: subject::SubjectJob =
serde_json::from_value(payload).map_err(JobParseError::InvalidJson)?;
Ok(JobType::Subject(subject_job))
}
_ => Err(JobParseError::UnsupportedTargetType(target_type)),
}
}
/// Convert to a Job trait object
pub fn boxed(self) -> Box<dyn Job> {
match self {
JobType::Subject(job) => Box::new(job),
}
}
}

View File

@@ -0,0 +1,56 @@
use super::Job;
use crate::banner::{BannerApi, SearchQuery, Term};
use crate::data::batch::batch_upsert_courses;
use crate::data::models::TargetType;
use crate::error::Result;
use serde::{Deserialize, Serialize};
use sqlx::PgPool;
use tracing::{debug, info};
/// Job implementation for scraping subject data
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SubjectJob {
pub subject: String,
}
impl SubjectJob {
pub fn new(subject: String) -> Self {
Self { subject }
}
}
#[async_trait::async_trait]
impl Job for SubjectJob {
fn target_type(&self) -> TargetType {
TargetType::Subject
}
#[tracing::instrument(skip(self, banner_api, db_pool), fields(subject = %self.subject))]
async fn process(&self, banner_api: &BannerApi, db_pool: &PgPool) -> Result<()> {
let subject_code = &self.subject;
// Get the current term
let term = Term::get_current().inner().to_string();
let query = SearchQuery::new().subject(subject_code).max_results(500);
let search_result = banner_api
.search(&term, &query, "subjectDescription", false)
.await?;
if let Some(courses_from_api) = search_result.data {
info!(
subject = subject_code,
count = courses_from_api.len(),
"Found courses"
);
batch_upsert_courses(&courses_from_api, db_pool).await?;
}
debug!(subject = subject_code, "Subject job completed");
Ok(())
}
fn description(&self) -> String {
format!("Scrape subject: {}", self.subject)
}
}

116
src/scraper/mod.rs Normal file
View File

@@ -0,0 +1,116 @@
pub mod jobs;
pub mod scheduler;
pub mod worker;
use crate::banner::BannerApi;
use crate::services::Service;
use sqlx::PgPool;
use std::sync::Arc;
use tokio::sync::broadcast;
use tokio::task::JoinHandle;
use tracing::{info, warn};
use self::scheduler::Scheduler;
use self::worker::Worker;
/// The main service that will be managed by the application's `ServiceManager`.
///
/// It holds the shared resources (database pool, API client) and manages the
/// lifecycle of the Scheduler and Worker tasks.
pub struct ScraperService {
db_pool: PgPool,
banner_api: Arc<BannerApi>,
scheduler_handle: Option<JoinHandle<()>>,
worker_handles: Vec<JoinHandle<()>>,
shutdown_tx: Option<broadcast::Sender<()>>,
}
impl ScraperService {
/// Creates a new `ScraperService`.
pub fn new(db_pool: PgPool, banner_api: Arc<BannerApi>) -> Self {
Self {
db_pool,
banner_api,
scheduler_handle: None,
worker_handles: Vec::new(),
shutdown_tx: None,
}
}
/// Starts the scheduler and a pool of workers.
pub fn start(&mut self) {
info!("ScraperService starting");
// Create shutdown channel
let (shutdown_tx, _) = broadcast::channel(1);
self.shutdown_tx = Some(shutdown_tx.clone());
let scheduler = Scheduler::new(self.db_pool.clone(), self.banner_api.clone());
let shutdown_rx = shutdown_tx.subscribe();
let scheduler_handle = tokio::spawn(async move {
scheduler.run(shutdown_rx).await;
});
self.scheduler_handle = Some(scheduler_handle);
info!("Scheduler task spawned");
let worker_count = 4; // This could be configurable
for i in 0..worker_count {
let worker = Worker::new(i, self.db_pool.clone(), self.banner_api.clone());
let shutdown_rx = shutdown_tx.subscribe();
let worker_handle = tokio::spawn(async move {
worker.run(shutdown_rx).await;
});
self.worker_handles.push(worker_handle);
}
info!(
worker_count = self.worker_handles.len(),
"Spawned worker tasks"
);
}
}
#[async_trait::async_trait]
impl Service for ScraperService {
fn name(&self) -> &'static str {
"scraper"
}
async fn run(&mut self) -> Result<(), anyhow::Error> {
self.start();
std::future::pending::<()>().await;
Ok(())
}
async fn shutdown(&mut self) -> Result<(), anyhow::Error> {
info!("Shutting down scraper service");
// Send shutdown signal to all tasks
if let Some(shutdown_tx) = self.shutdown_tx.take() {
let _ = shutdown_tx.send(());
} else {
warn!("No shutdown channel found for scraper service");
return Err(anyhow::anyhow!("No shutdown channel available"));
}
// Collect all handles
let mut all_handles = Vec::new();
if let Some(handle) = self.scheduler_handle.take() {
all_handles.push(handle);
}
all_handles.append(&mut self.worker_handles);
// Wait for all tasks to complete (no internal timeout - let ServiceManager handle it)
let results = futures::future::join_all(all_handles).await;
let failed = results.iter().filter(|r| r.is_err()).count();
if failed > 0 {
warn!(
failed_count = failed,
"Some scraper tasks panicked during shutdown"
);
return Err(anyhow::anyhow!("{} task(s) panicked", failed));
}
info!("All scraper tasks shutdown gracefully");
Ok(())
}
}

188
src/scraper/scheduler.rs Normal file
View File

@@ -0,0 +1,188 @@
use crate::banner::{BannerApi, Term};
use crate::data::models::{ScrapePriority, TargetType};
use crate::error::Result;
use crate::scraper::jobs::subject::SubjectJob;
use serde_json::json;
use sqlx::PgPool;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::broadcast;
use tokio::time;
use tokio_util::sync::CancellationToken;
use tracing::{debug, error, info, warn};
/// Periodically analyzes data and enqueues prioritized scrape jobs.
pub struct Scheduler {
db_pool: PgPool,
banner_api: Arc<BannerApi>,
}
impl Scheduler {
pub fn new(db_pool: PgPool, banner_api: Arc<BannerApi>) -> Self {
Self {
db_pool,
banner_api,
}
}
/// Runs the scheduler's main loop with graceful shutdown support.
///
/// The scheduler wakes up every 60 seconds to analyze data and enqueue jobs.
/// When a shutdown signal is received:
/// 1. Any in-progress scheduling work is gracefully cancelled via CancellationToken
/// 2. The scheduler waits up to 5 seconds for work to complete
/// 3. If timeout occurs, the task is abandoned (it will be aborted when dropped)
///
/// This ensures that shutdown is responsive even if scheduling work is blocked.
pub async fn run(&self, mut shutdown_rx: broadcast::Receiver<()>) {
info!("Scheduler service started");
let work_interval = Duration::from_secs(60);
let mut next_run = time::Instant::now();
let mut current_work: Option<(tokio::task::JoinHandle<()>, CancellationToken)> = None;
loop {
tokio::select! {
_ = time::sleep_until(next_run) => {
let cancel_token = CancellationToken::new();
// Spawn work in separate task to allow graceful cancellation during shutdown.
// Without this, shutdown would have to wait for the full scheduling cycle.
let work_handle = tokio::spawn({
let db_pool = self.db_pool.clone();
let banner_api = self.banner_api.clone();
let cancel_token = cancel_token.clone();
async move {
tokio::select! {
result = Self::schedule_jobs_impl(&db_pool, &banner_api) => {
if let Err(e) = result {
error!(error = ?e, "Failed to schedule jobs");
}
}
_ = cancel_token.cancelled() => {
debug!("Scheduling work cancelled gracefully");
}
}
}
});
current_work = Some((work_handle, cancel_token));
next_run = time::Instant::now() + work_interval;
}
_ = shutdown_rx.recv() => {
info!("Scheduler received shutdown signal");
if let Some((handle, cancel_token)) = current_work.take() {
cancel_token.cancel();
// Wait briefly for graceful completion
if tokio::time::timeout(Duration::from_secs(5), handle).await.is_err() {
warn!("Scheduling work did not complete within 5s, abandoning");
} else {
debug!("Scheduling work completed gracefully");
}
}
info!("Scheduler exiting gracefully");
break;
}
}
}
}
/// Core scheduling logic that analyzes data and creates scrape jobs.
///
/// Strategy:
/// 1. Fetch all subjects for the current term from Banner API
/// 2. Query existing jobs in a single batch query
/// 3. Create jobs only for subjects that don't have pending jobs
///
/// This is a static method (not &self) to allow it to be called from spawned tasks.
#[tracing::instrument(skip_all, fields(term))]
async fn schedule_jobs_impl(db_pool: &PgPool, banner_api: &BannerApi) -> Result<()> {
// For now, we will implement a simple baseline scheduling strategy:
// 1. Get a list of all subjects from the Banner API.
// 2. Query existing jobs for all subjects in a single query.
// 3. Create new jobs only for subjects that don't have existing jobs.
let term = Term::get_current().inner().to_string();
tracing::Span::current().record("term", term.as_str());
debug!(term = term, "Enqueuing subject jobs");
let subjects = banner_api.get_subjects("", &term, 1, 500).await?;
debug!(
subject_count = subjects.len(),
"Retrieved subjects from API"
);
// Create payloads for all subjects
let subject_payloads: Vec<_> = subjects
.iter()
.map(|subject| json!({ "subject": subject.code }))
.collect();
// Query existing jobs for all subjects in a single query
let existing_jobs: Vec<(serde_json::Value,)> = sqlx::query_as(
"SELECT target_payload FROM scrape_jobs
WHERE target_type = $1 AND target_payload = ANY($2) AND locked_at IS NULL",
)
.bind(TargetType::Subject)
.bind(&subject_payloads)
.fetch_all(db_pool)
.await?;
// Convert to a HashSet for efficient lookup
let existing_payloads: std::collections::HashSet<String> = existing_jobs
.into_iter()
.map(|(payload,)| payload.to_string())
.collect();
// Filter out subjects that already have jobs and prepare new jobs
let mut skipped_count = 0;
let new_jobs: Vec<_> = subjects
.into_iter()
.filter_map(|subject| {
let job = SubjectJob::new(subject.code.clone());
let payload = serde_json::to_value(&job).unwrap();
let payload_str = payload.to_string();
if existing_payloads.contains(&payload_str) {
skipped_count += 1;
None
} else {
Some((payload, subject.code))
}
})
.collect();
if skipped_count > 0 {
debug!(count = skipped_count, "Skipped subjects with existing jobs");
}
// Insert all new jobs in a single batch
if !new_jobs.is_empty() {
let now = chrono::Utc::now();
let mut tx = db_pool.begin().await?;
for (payload, subject_code) in new_jobs {
sqlx::query(
"INSERT INTO scrape_jobs (target_type, target_payload, priority, execute_at) VALUES ($1, $2, $3, $4)"
)
.bind(TargetType::Subject)
.bind(&payload)
.bind(ScrapePriority::Low)
.bind(now)
.execute(&mut *tx)
.await?;
debug!(subject = subject_code, "New job enqueued for subject");
}
tx.commit().await?;
}
debug!("Job scheduling complete");
Ok(())
}
}

300
src/scraper/worker.rs Normal file
View File

@@ -0,0 +1,300 @@
use crate::banner::{BannerApi, BannerApiError};
use crate::data::models::ScrapeJob;
use crate::error::Result;
use crate::scraper::jobs::{JobError, JobType};
use sqlx::PgPool;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::broadcast;
use tokio::time;
use tracing::{Instrument, debug, error, info, trace, warn};
/// A single worker instance.
///
/// Each worker runs in its own asynchronous task and continuously polls the
/// database for scrape jobs to execute.
pub struct Worker {
id: usize, // For logging purposes
db_pool: PgPool,
banner_api: Arc<BannerApi>,
}
impl Worker {
pub fn new(id: usize, db_pool: PgPool, banner_api: Arc<BannerApi>) -> Self {
Self {
id,
db_pool,
banner_api,
}
}
/// Runs the worker's main loop.
pub async fn run(&self, mut shutdown_rx: broadcast::Receiver<()>) {
info!(worker_id = self.id, "Worker started");
loop {
// Fetch and lock a job, racing against shutdown signal
let job = tokio::select! {
_ = shutdown_rx.recv() => {
info!(worker_id = self.id, "Worker received shutdown signal, exiting gracefully");
break;
}
result = self.fetch_and_lock_job() => {
match result {
Ok(Some(job)) => job,
Ok(None) => {
trace!(worker_id = self.id, "No jobs available, waiting");
time::sleep(Duration::from_secs(5)).await;
continue;
}
Err(e) => {
warn!(worker_id = self.id, error = ?e, "Failed to fetch job, waiting");
time::sleep(Duration::from_secs(10)).await;
continue;
}
}
}
};
let job_id = job.id;
let retry_count = job.retry_count;
let max_retries = job.max_retries;
let start = std::time::Instant::now();
// Process the job, racing against shutdown signal
let process_result = tokio::select! {
_ = shutdown_rx.recv() => {
self.handle_shutdown_during_processing(job_id).await;
break;
}
result = self.process_job(job) => result
};
let duration = start.elapsed();
// Handle the job processing result
self.handle_job_result(job_id, retry_count, max_retries, process_result, duration)
.await;
}
}
/// Atomically fetches a job from the queue, locking it for processing.
///
/// This uses a `FOR UPDATE SKIP LOCKED` query to ensure that multiple
/// workers can poll the queue concurrently without conflicts.
async fn fetch_and_lock_job(&self) -> Result<Option<ScrapeJob>> {
let mut tx = self.db_pool.begin().await?;
let job = sqlx::query_as::<_, ScrapeJob>(
"SELECT * FROM scrape_jobs WHERE locked_at IS NULL AND execute_at <= NOW() ORDER BY priority DESC, execute_at ASC LIMIT 1 FOR UPDATE SKIP LOCKED"
)
.fetch_optional(&mut *tx)
.await?;
if let Some(ref job) = job {
sqlx::query("UPDATE scrape_jobs SET locked_at = NOW() WHERE id = $1")
.bind(job.id)
.execute(&mut *tx)
.await?;
}
tx.commit().await?;
Ok(job)
}
async fn process_job(&self, job: ScrapeJob) -> Result<(), JobError> {
// Convert the database job to our job type
let job_type = JobType::from_target_type_and_payload(job.target_type, job.target_payload)
.map_err(|e| JobError::Unrecoverable(anyhow::anyhow!(e)))?; // Parse errors are unrecoverable
// Get the job implementation
let job_impl = job_type.boxed();
// Create span with job context
let span = tracing::debug_span!(
"process_job",
job_id = job.id,
job_type = job_impl.description()
);
async move {
debug!(
worker_id = self.id,
job_id = job.id,
description = job_impl.description(),
"Processing job"
);
// Process the job - API errors are recoverable
job_impl
.process(&self.banner_api, &self.db_pool)
.await
.map_err(JobError::Recoverable)?;
Ok(())
}
.instrument(span)
.await
}
async fn delete_job(&self, job_id: i32) -> Result<()> {
sqlx::query("DELETE FROM scrape_jobs WHERE id = $1")
.bind(job_id)
.execute(&self.db_pool)
.await?;
Ok(())
}
async fn unlock_job(&self, job_id: i32) -> Result<()> {
sqlx::query("UPDATE scrape_jobs SET locked_at = NULL WHERE id = $1")
.bind(job_id)
.execute(&self.db_pool)
.await?;
Ok(())
}
async fn unlock_and_increment_retry(&self, job_id: i32, max_retries: i32) -> Result<bool> {
let result = sqlx::query_scalar::<_, Option<i32>>(
"UPDATE scrape_jobs
SET locked_at = NULL, retry_count = retry_count + 1
WHERE id = $1
RETURNING CASE WHEN retry_count + 1 < $2 THEN retry_count + 1 ELSE NULL END",
)
.bind(job_id)
.bind(max_retries)
.fetch_one(&self.db_pool)
.await?;
Ok(result.is_some())
}
/// Handle shutdown signal received during job processing
async fn handle_shutdown_during_processing(&self, job_id: i32) {
info!(
worker_id = self.id,
job_id, "Shutdown received during job processing"
);
if let Err(e) = self.unlock_job(job_id).await {
warn!(
worker_id = self.id,
job_id,
error = ?e,
"Failed to unlock job during shutdown"
);
} else {
debug!(worker_id = self.id, job_id, "Job unlocked during shutdown");
}
info!(worker_id = self.id, "Worker exiting gracefully");
}
/// Handle the result of job processing
async fn handle_job_result(
&self,
job_id: i32,
retry_count: i32,
max_retries: i32,
result: Result<(), JobError>,
duration: std::time::Duration,
) {
match result {
Ok(()) => {
debug!(
worker_id = self.id,
job_id,
duration_ms = duration.as_millis(),
"Job completed successfully"
);
if let Err(e) = self.delete_job(job_id).await {
error!(worker_id = self.id, job_id, error = ?e, "Failed to delete completed job");
}
}
Err(JobError::Recoverable(e)) => {
self.handle_recoverable_error(job_id, retry_count, max_retries, e, duration)
.await;
}
Err(JobError::Unrecoverable(e)) => {
error!(
worker_id = self.id,
job_id,
duration_ms = duration.as_millis(),
error = ?e,
"Job corrupted, deleting"
);
if let Err(e) = self.delete_job(job_id).await {
error!(worker_id = self.id, job_id, error = ?e, "Failed to delete corrupted job");
}
}
}
}
/// Handle recoverable errors by logging appropriately and unlocking the job
async fn handle_recoverable_error(
&self,
job_id: i32,
retry_count: i32,
max_retries: i32,
e: anyhow::Error,
duration: std::time::Duration,
) {
let next_attempt = retry_count.saturating_add(1);
let remaining_retries = max_retries.saturating_sub(next_attempt);
// Log the error appropriately based on type
if let Some(BannerApiError::InvalidSession(_)) = e.downcast_ref::<BannerApiError>() {
warn!(
worker_id = self.id,
job_id,
duration_ms = duration.as_millis(),
retry_attempt = next_attempt,
max_retries = max_retries,
remaining_retries = remaining_retries,
"Invalid session detected, will retry"
);
} else {
error!(
worker_id = self.id,
job_id,
duration_ms = duration.as_millis(),
retry_attempt = next_attempt,
max_retries = max_retries,
remaining_retries = remaining_retries,
error = ?e,
"Failed to process job, will retry"
);
}
// Atomically unlock and increment retry count, checking if retry is allowed
match self.unlock_and_increment_retry(job_id, max_retries).await {
Ok(can_retry) if can_retry => {
info!(
worker_id = self.id,
job_id,
retry_attempt = next_attempt,
remaining_retries = remaining_retries,
"Job unlocked for retry"
);
}
Ok(_) => {
// Max retries exceeded (detected atomically)
error!(
worker_id = self.id,
job_id,
duration_ms = duration.as_millis(),
retry_count = next_attempt,
max_retries = max_retries,
error = ?e,
"Job failed permanently (max retries exceeded), deleting"
);
if let Err(e) = self.delete_job(job_id).await {
error!(worker_id = self.id, job_id, error = ?e, "Failed to delete failed job");
}
}
Err(e) => {
error!(worker_id = self.id, job_id, error = ?e, "Failed to unlock and increment retry count");
}
}
}
}

View File

@@ -1,20 +1,193 @@
use super::Service;
use crate::bot::{Data, get_commands};
use crate::config::Config;
use crate::state::AppState;
use num_format::{Locale, ToFormattedString};
use serenity::Client;
use serenity::all::{ActivityData, ClientBuilder, GatewayIntents};
use std::sync::Arc;
use tracing::{error, warn};
use std::time::Duration;
use tokio::sync::{Mutex, broadcast};
use tokio::task::JoinHandle;
use tracing::{debug, error, info, warn};
/// Discord bot service implementation
pub struct BotService {
client: Client,
shard_manager: Arc<serenity::gateway::ShardManager>,
status_task_handle: Arc<Mutex<Option<JoinHandle<()>>>>,
status_shutdown_tx: Option<broadcast::Sender<()>>,
}
impl BotService {
pub fn new(client: Client) -> Self {
/// Create a new Discord bot client with full configuration
pub async fn create_client(
config: &Config,
app_state: AppState,
status_task_handle: Arc<Mutex<Option<JoinHandle<()>>>>,
status_shutdown_rx: broadcast::Receiver<()>,
) -> Result<Client, anyhow::Error> {
let intents = GatewayIntents::non_privileged();
let bot_target_guild = config.bot_target_guild;
let framework = poise::Framework::builder()
.options(poise::FrameworkOptions {
commands: get_commands(),
pre_command: |ctx| {
Box::pin(async move {
let content = match ctx {
poise::Context::Application(_) => ctx.invocation_string(),
poise::Context::Prefix(prefix) => prefix.msg.content.to_string(),
};
let channel_name = ctx
.channel_id()
.name(ctx.http())
.await
.unwrap_or("unknown".to_string());
let span = tracing::Span::current();
span.record("command_name", ctx.command().qualified_name.as_str());
span.record("invocation", ctx.invocation_string());
span.record("msg.content", content.as_str());
span.record("msg.author", ctx.author().tag().as_str());
span.record("msg.id", ctx.id());
span.record("msg.channel_id", ctx.channel_id().get());
span.record("msg.channel", channel_name.as_str());
tracing::info!(
command_name = ctx.command().qualified_name.as_str(),
invocation = ctx.invocation_string(),
msg.content = %content,
msg.author = %ctx.author().tag(),
msg.author_id = %ctx.author().id,
msg.id = %ctx.id(),
msg.channel = %channel_name.as_str(),
msg.channel_id = %ctx.channel_id(),
"{} invoked by {}",
ctx.command().name,
ctx.author().tag()
);
})
},
on_error: |error| {
Box::pin(async move {
if let Err(e) = poise::builtins::on_error(error).await {
tracing::error!(error = %e, "Fatal error while sending error message");
}
})
},
..Default::default()
})
.setup(move |ctx, _ready, framework| {
let app_state = app_state.clone();
let status_task_handle = status_task_handle.clone();
Box::pin(async move {
poise::builtins::register_in_guild(
ctx,
&framework.options().commands,
bot_target_guild.into(),
)
.await?;
poise::builtins::register_globally(ctx, &framework.options().commands).await?;
// Start status update task with shutdown support
let handle = Self::start_status_update_task(
ctx.clone(),
app_state.clone(),
status_shutdown_rx,
);
*status_task_handle.lock().await = Some(handle);
Ok(Data { app_state })
})
})
.build();
Ok(ClientBuilder::new(config.bot_token.clone(), intents)
.framework(framework)
.await?)
}
/// Start the status update task for the Discord bot with graceful shutdown support
fn start_status_update_task(
ctx: serenity::client::Context,
app_state: AppState,
mut shutdown_rx: broadcast::Receiver<()>,
) -> JoinHandle<()> {
tokio::spawn(async move {
let max_interval = Duration::from_secs(300); // 5 minutes
let base_interval = Duration::from_secs(30);
let mut interval = tokio::time::interval(base_interval);
let mut previous_course_count: Option<i64> = None;
// This runs once immediately on startup, then with adaptive intervals
loop {
tokio::select! {
_ = interval.tick() => {
// Get the course count, update the activity if it has changed/hasn't been set this session
let course_count = app_state.get_course_count().await.unwrap();
if previous_course_count.is_none() || previous_course_count != Some(course_count) {
ctx.set_activity(Some(ActivityData::playing(format!(
"Querying {:} classes",
course_count.to_formatted_string(&Locale::en)
))));
}
// Increase or reset the interval
interval = tokio::time::interval(
// Avoid logging the first 'change'
if course_count != previous_course_count.unwrap_or(0) {
if previous_course_count.is_some() {
debug!(
new_course_count = course_count,
last_interval = interval.period().as_secs(),
"Course count changed, resetting interval"
);
}
// Record the new course count
previous_course_count = Some(course_count);
// Reset to base interval
base_interval
} else {
// Increase interval by 10% (up to maximum)
let new_interval = interval.period().mul_f32(1.1).min(max_interval);
debug!(
current_course_count = course_count,
last_interval = interval.period().as_secs(),
new_interval = new_interval.as_secs(),
"Course count unchanged, increasing interval"
);
new_interval
},
);
// Reset the interval, otherwise it will tick again immediately
interval.reset();
}
_ = shutdown_rx.recv() => {
info!("Status update task received shutdown signal");
break;
}
}
}
})
}
pub fn new(
client: Client,
status_task_handle: Arc<Mutex<Option<JoinHandle<()>>>>,
status_shutdown_tx: broadcast::Sender<()>,
) -> Self {
let shard_manager = client.shard_manager.clone();
Self {
client,
shard_manager,
status_task_handle,
status_shutdown_tx: Some(status_shutdown_tx),
}
}
}
@@ -28,17 +201,39 @@ impl Service for BotService {
async fn run(&mut self) -> Result<(), anyhow::Error> {
match self.client.start().await {
Ok(()) => {
warn!(service = "bot", "Stopped early.");
warn!(service = "bot", "stopped early");
Err(anyhow::anyhow!("bot stopped early"))
}
Err(e) => {
error!(service = "bot", "Error: {e:?}");
error!(service = "bot", "error: {e:?}");
Err(e.into())
}
}
}
async fn shutdown(&mut self) -> Result<(), anyhow::Error> {
// Signal status update task to stop
if let Some(status_shutdown_tx) = self.status_shutdown_tx.take() {
let _ = status_shutdown_tx.send(());
}
// Wait for status update task to complete (with timeout)
let handle = self.status_task_handle.lock().await.take();
if let Some(handle) = handle {
match tokio::time::timeout(Duration::from_secs(2), handle).await {
Ok(Ok(())) => {
debug!("Status update task completed gracefully");
}
Ok(Err(e)) => {
warn!(error = ?e, "Status update task panicked");
}
Err(_) => {
warn!("Status update task did not complete within 2s timeout");
}
}
}
// Shutdown Discord shards
self.shard_manager.shutdown_all().await;
Ok(())
}

View File

@@ -1,155 +1,220 @@
use std::collections::HashMap;
use std::time::Duration;
use tokio::sync::broadcast;
use tokio::task::JoinHandle;
use tracing::{error, info, warn};
use tokio::sync::{broadcast, mpsc};
use tracing::{debug, info, trace, warn};
use crate::services::ServiceResult;
use crate::services::{Service, ServiceResult, run_service};
/// Manages multiple services and their lifecycle
pub struct ServiceManager {
services: HashMap<String, JoinHandle<ServiceResult>>,
registered_services: HashMap<String, Box<dyn Service>>,
service_handles: HashMap<String, tokio::task::AbortHandle>,
completion_rx: Option<mpsc::UnboundedReceiver<(String, ServiceResult)>>,
completion_tx: mpsc::UnboundedSender<(String, ServiceResult)>,
shutdown_tx: broadcast::Sender<()>,
}
impl Default for ServiceManager {
fn default() -> Self {
Self::new()
}
}
impl ServiceManager {
pub fn new() -> Self {
let (shutdown_tx, _) = broadcast::channel(1);
let (completion_tx, completion_rx) = mpsc::unbounded_channel();
Self {
services: HashMap::new(),
registered_services: HashMap::new(),
service_handles: HashMap::new(),
completion_rx: Some(completion_rx),
completion_tx,
shutdown_tx,
}
}
/// Add a service to be managed
pub fn add_service(&mut self, name: String, handle: JoinHandle<ServiceResult>) {
self.services.insert(name, handle);
/// Register a service to be managed (not yet spawned)
pub fn register_service(&mut self, name: &str, service: Box<dyn Service>) {
self.registered_services.insert(name.to_string(), service);
}
/// Get a shutdown receiver for services to subscribe to
pub fn subscribe(&self) -> broadcast::Receiver<()> {
self.shutdown_tx.subscribe()
/// Check if there are any registered services
pub fn has_services(&self) -> bool {
!self.registered_services.is_empty()
}
/// Spawn all registered services
pub fn spawn_all(&mut self) {
let service_count = self.registered_services.len();
let service_names: Vec<_> = self.registered_services.keys().cloned().collect();
for (name, service) in self.registered_services.drain() {
let shutdown_rx = self.shutdown_tx.subscribe();
let completion_tx = self.completion_tx.clone();
let name_clone = name.clone();
// Spawn service task
let handle = tokio::spawn(async move {
let result = run_service(service, shutdown_rx).await;
// Send completion notification
let _ = completion_tx.send((name_clone, result));
});
// Store abort handle for shutdown control
self.service_handles
.insert(name.clone(), handle.abort_handle());
debug!(service = name, id = ?handle.id(), "service spawned");
}
info!(
service_count,
services = ?service_names,
"spawned {} services",
service_count
);
}
/// Run all services until one completes or fails
/// Returns the first service that completes and its result
pub async fn run(&mut self) -> (String, ServiceResult) {
if self.services.is_empty() {
if self.service_handles.is_empty() {
return (
"none".to_string(),
ServiceResult::Error(anyhow::anyhow!("No services to run")),
);
}
info!("ServiceManager running {} services", self.services.len());
// Wait for any service to complete
loop {
let mut completed_services = Vec::new();
for (name, handle) in &mut self.services {
if handle.is_finished() {
completed_services.push(name.clone());
}
}
if let Some(completed_name) = completed_services.first() {
let handle = self.services.remove(completed_name).unwrap();
match handle.await {
Ok(result) => {
return (completed_name.clone(), result);
}
Err(e) => {
error!(service = completed_name, "Service task panicked: {e}");
return (
completed_name.clone(),
ServiceResult::Error(anyhow::anyhow!("Task panic: {e}")),
);
}
}
}
// Small delay to prevent busy-waiting
tokio::time::sleep(Duration::from_millis(10)).await;
}
}
/// Shutdown all services gracefully with a timeout
/// Returns Ok(()) if all services shut down, or Err(Vec<String>) with names of services that timed out
pub async fn shutdown(mut self, timeout: Duration) -> Result<(), Vec<String>> {
if self.services.is_empty() {
info!("No services to shutdown");
return Ok(());
}
info!(
"Shutting down {} services with {}s timeout",
self.services.len(),
timeout.as_secs()
"servicemanager running {} services",
self.service_handles.len()
);
// Signal all services to shutdown
// Wait for any service to complete via the channel
let completion_rx = self
.completion_rx
.as_mut()
.expect("completion_rx should be available");
completion_rx
.recv()
.await
.map(|(name, result)| {
self.service_handles.remove(&name);
(name, result)
})
.unwrap_or_else(|| {
(
"channel_closed".to_string(),
ServiceResult::Error(anyhow::anyhow!("Completion channel closed")),
)
})
}
/// Shutdown all services gracefully with a timeout.
///
/// All services receive the shutdown signal simultaneously and shut down in parallel.
/// Each service gets the full timeout duration (they don't share/consume from a budget).
/// If any service fails to shutdown within the timeout, it will be aborted.
///
/// Returns the elapsed time if all succeed, or a list of failed service names.
pub async fn shutdown(&mut self, timeout: Duration) -> Result<Duration, Vec<String>> {
let service_count = self.service_handles.len();
let service_names: Vec<_> = self.service_handles.keys().cloned().collect();
info!(
service_count,
services = ?service_names,
timeout = format!("{:.2?}", timeout),
"shutting down {} services in parallel with {:?} timeout each",
service_count,
timeout
);
if service_count == 0 {
return Ok(Duration::ZERO);
}
// Send shutdown signal to all services simultaneously
let _ = self.shutdown_tx.send(());
// Wait for all services to complete with timeout
let shutdown_result = tokio::time::timeout(timeout, async {
let mut completed = Vec::new();
let mut failed = Vec::new();
let start_time = std::time::Instant::now();
while !self.services.is_empty() {
let mut to_remove = Vec::new();
// Collect results from all services with timeout
let completion_rx = self
.completion_rx
.as_mut()
.expect("completion_rx should be available");
for (name, handle) in &mut self.services {
if handle.is_finished() {
to_remove.push(name.clone());
}
}
for name in to_remove {
let handle = self.services.remove(&name).unwrap();
match handle.await {
Ok(ServiceResult::GracefulShutdown) => {
completed.push(name);
}
Ok(ServiceResult::NormalCompletion) => {
warn!(service = name, "Service completed normally during shutdown");
completed.push(name);
}
Ok(ServiceResult::Error(e)) => {
error!(service = name, "Service error during shutdown: {e}");
failed.push(name);
}
Err(e) => {
error!(service = name, "Service panic during shutdown: {e}");
failed.push(name);
}
}
}
if !self.services.is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
// Collect all completion results with a single timeout
let collect_future = async {
let mut collected: Vec<Option<(String, ServiceResult)>> = Vec::new();
for _ in 0..service_count {
if let Some(result) = completion_rx.recv().await {
collected.push(Some(result));
} else {
collected.push(None);
}
}
collected
};
(completed, failed)
})
.await;
match shutdown_result {
Ok((completed, failed)) => {
if !completed.is_empty() {
info!("Services shutdown completed: {}", completed.join(", "));
}
if !failed.is_empty() {
warn!("Services had errors during shutdown: {}", failed.join(", "));
}
Ok(())
}
let results = match tokio::time::timeout(timeout, collect_future).await {
Ok(results) => results,
Err(_) => {
// Timeout occurred - return names of services that didn't complete
let pending_services: Vec<String> = self.services.keys().cloned().collect();
Err(pending_services)
// Timeout exceeded - abort all remaining services
warn!(
timeout = format!("{:.2?}", timeout),
"shutdown timeout exceeded - aborting all remaining services"
);
let failed: Vec<String> = self.service_handles.keys().cloned().collect();
for handle in self.service_handles.values() {
handle.abort();
}
self.service_handles.clear();
return Err(failed);
}
};
// Process results and identify failures
let mut failed_services = Vec::new();
for (name, service_result) in results.into_iter().flatten() {
self.service_handles.remove(&name);
if matches!(service_result, ServiceResult::GracefulShutdown) {
trace!(service = name, "service shutdown completed");
} else {
warn!(
service = name,
result = ?service_result,
"service shutdown with non-graceful result"
);
failed_services.push(name);
}
}
let elapsed = start_time.elapsed();
if failed_services.is_empty() {
info!(
service_count,
elapsed = format!("{:.2?}", elapsed),
"all services shutdown successfully: {}",
service_names.join(", ")
);
Ok(elapsed)
} else {
warn!(
failed_count = failed_services.len(),
failed_services = ?failed_services,
elapsed = format!("{:.2?}", elapsed),
"{} service(s) failed to shutdown gracefully: {}",
failed_services.len(),
failed_services.join(", ")
);
Err(failed_services)
}
}
}

View File

@@ -3,6 +3,7 @@ use tracing::{error, info, warn};
pub mod bot;
pub mod manager;
pub mod web;
#[derive(Debug)]
pub enum ServiceResult {
@@ -21,6 +22,12 @@ pub trait Service: Send + Sync {
async fn run(&mut self) -> Result<(), anyhow::Error>;
/// Gracefully shutdown the service
///
/// Implementations should initiate shutdown and MAY wait for completion.
/// Services are expected to respond to this call and begin cleanup promptly.
/// When managed by ServiceManager, the configured timeout (default 8s) applies to
/// ALL services combined, not per-service. Services should complete shutdown as
/// quickly as possible to avoid timeout.
async fn shutdown(&mut self) -> Result<(), anyhow::Error>;
}
@@ -30,16 +37,16 @@ pub async fn run_service(
mut shutdown_rx: broadcast::Receiver<()>,
) -> ServiceResult {
let name = service.name();
info!(service = name, "Service started");
info!(service = name, "service started");
let work = async {
match service.run().await {
Ok(()) => {
warn!(service = name, "Service completed unexpectedly");
warn!(service = name, "service completed unexpectedly");
ServiceResult::NormalCompletion
}
Err(e) => {
error!(service = name, "Service failed: {e}");
error!(service = name, "service failed: {e}");
ServiceResult::Error(e)
}
}
@@ -48,18 +55,18 @@ pub async fn run_service(
tokio::select! {
result = work => result,
_ = shutdown_rx.recv() => {
info!(service = name, "Shutting down...");
info!(service = name, "shutting down...");
let start_time = std::time::Instant::now();
match service.shutdown().await {
Ok(()) => {
let elapsed = start_time.elapsed();
info!(service = name, "Shutdown completed in {elapsed:.2?}");
info!(service = name, "shutdown completed in {elapsed:.2?}");
ServiceResult::GracefulShutdown
}
Err(e) => {
let elapsed = start_time.elapsed();
error!(service = name, "Shutdown failed after {elapsed:.2?}: {e}");
error!(service = name, "shutdown failed after {elapsed:.2?}: {e}");
ServiceResult::Error(e)
}
}

78
src/services/web.rs Normal file
View File

@@ -0,0 +1,78 @@
use super::Service;
use crate::web::{BannerState, create_router};
use std::net::SocketAddr;
use tokio::net::TcpListener;
use tokio::sync::broadcast;
use tracing::{info, trace, warn};
/// Web server service implementation
pub struct WebService {
port: u16,
banner_state: BannerState,
shutdown_tx: Option<broadcast::Sender<()>>,
}
impl WebService {
pub fn new(port: u16, banner_state: BannerState) -> Self {
Self {
port,
banner_state,
shutdown_tx: None,
}
}
}
#[async_trait::async_trait]
impl Service for WebService {
fn name(&self) -> &'static str {
"web"
}
async fn run(&mut self) -> Result<(), anyhow::Error> {
// Create the main router with Banner API routes
let app = create_router(self.banner_state.clone());
let addr = SocketAddr::from(([0, 0, 0, 0], self.port));
let listener = TcpListener::bind(addr).await?;
info!(
service = "web",
address = %addr,
link = format!("http://localhost:{}", addr.port()),
"web server listening"
);
// Create internal shutdown channel for axum graceful shutdown
let (shutdown_tx, mut shutdown_rx) = broadcast::channel(1);
self.shutdown_tx = Some(shutdown_tx);
// Use axum's graceful shutdown with the internal shutdown signal
axum::serve(listener, app)
.with_graceful_shutdown(async move {
let _ = shutdown_rx.recv().await;
trace!(
service = "web",
"received shutdown signal, starting graceful shutdown"
);
})
.await?;
trace!(service = "web", "graceful shutdown completed");
info!(service = "web", "web server stopped");
Ok(())
}
async fn shutdown(&mut self) -> Result<(), anyhow::Error> {
if let Some(shutdown_tx) = self.shutdown_tx.take() {
let _ = shutdown_tx.send(());
trace!(service = "web", "sent shutdown signal to axum");
} else {
warn!(
service = "web",
"no shutdown channel found, cannot trigger graceful shutdown"
);
}
Ok(())
}
}

106
src/signals.rs Normal file
View File

@@ -0,0 +1,106 @@
use crate::services::ServiceResult;
use crate::services::manager::ServiceManager;
use std::process::ExitCode;
use std::time::Duration;
use tokio::signal;
use tracing::{error, info, warn};
/// Handle application shutdown signals and graceful shutdown
pub async fn handle_shutdown_signals(
mut service_manager: ServiceManager,
shutdown_timeout: Duration,
) -> ExitCode {
// Set up signal handling for both SIGINT (Ctrl+C) and SIGTERM
let ctrl_c = async {
signal::ctrl_c()
.await
.expect("Failed to install CTRL+C signal handler");
info!("received ctrl+c, gracefully shutting down...");
};
#[cfg(unix)]
let sigterm = async {
use tokio::signal::unix::{SignalKind, signal};
let mut sigterm_stream =
signal(SignalKind::terminate()).expect("Failed to install SIGTERM signal handler");
sigterm_stream.recv().await;
info!("received SIGTERM, gracefully shutting down...");
};
#[cfg(not(unix))]
let sigterm = async {
// On non-Unix systems, create a future that never completes
// This ensures the select! macro works correctly
std::future::pending::<()>().await;
};
// Main application loop - wait for services or signals
let mut exit_code = ExitCode::SUCCESS;
tokio::select! {
(service_name, result) = service_manager.run() => {
// A service completed unexpectedly
match result {
ServiceResult::GracefulShutdown => {
info!(service = service_name, "service completed gracefully");
}
ServiceResult::NormalCompletion => {
warn!(service = service_name, "service completed unexpectedly");
exit_code = ExitCode::FAILURE;
}
ServiceResult::Error(e) => {
error!(service = service_name, error = ?e, "service failed");
exit_code = ExitCode::FAILURE;
}
}
// Shutdown remaining services
exit_code = handle_graceful_shutdown(service_manager, shutdown_timeout, exit_code).await;
}
_ = ctrl_c => {
// User requested shutdown via Ctrl+C
info!("user requested shutdown via ctrl+c");
exit_code = handle_graceful_shutdown(service_manager, shutdown_timeout, ExitCode::SUCCESS).await;
}
_ = sigterm => {
// System requested shutdown via SIGTERM
info!("system requested shutdown via SIGTERM");
exit_code = handle_graceful_shutdown(service_manager, shutdown_timeout, ExitCode::SUCCESS).await;
}
}
info!(exit_code = ?exit_code, "application shutdown complete");
exit_code
}
/// Handle graceful shutdown of remaining services
async fn handle_graceful_shutdown(
mut service_manager: ServiceManager,
shutdown_timeout: Duration,
current_exit_code: ExitCode,
) -> ExitCode {
match service_manager.shutdown(shutdown_timeout).await {
Ok(elapsed) => {
info!(
remaining = format!("{:.2?}", shutdown_timeout - elapsed),
"graceful shutdown complete"
);
current_exit_code
}
Err(pending_services) => {
warn!(
pending_count = pending_services.len(),
pending_services = ?pending_services,
"graceful shutdown elapsed - {} service(s) did not complete",
pending_services.len()
);
// Non-zero exit code, default to FAILURE if not set
if current_exit_code == ExitCode::SUCCESS {
ExitCode::FAILURE
} else {
current_exit_code
}
}
}
}

38
src/state.rs Normal file
View File

@@ -0,0 +1,38 @@
//! Application state shared across components (bot, web, scheduler).
use crate::banner::BannerApi;
use crate::banner::Course;
use anyhow::Result;
use sqlx::PgPool;
use std::sync::Arc;
#[derive(Clone)]
pub struct AppState {
pub banner_api: Arc<BannerApi>,
pub db_pool: PgPool,
}
impl AppState {
pub fn new(banner_api: Arc<BannerApi>, db_pool: PgPool) -> Self {
Self {
banner_api,
db_pool,
}
}
/// Get a course by CRN directly from Banner API
pub async fn get_course_or_fetch(&self, term: &str, crn: &str) -> Result<Course> {
self.banner_api
.get_course_by_crn(term, crn)
.await?
.ok_or_else(|| anyhow::anyhow!("Course not found for CRN {crn}"))
}
/// Get the total number of courses in the database
pub async fn get_course_count(&self) -> Result<i64> {
let count: (i64,) = sqlx::query_as("SELECT COUNT(*) FROM courses")
.fetch_one(&self.db_pool)
.await?;
Ok(count.0)
}
}

1
src/utils/mod.rs Normal file
View File

@@ -0,0 +1 @@
pub mod shutdown;

32
src/utils/shutdown.rs Normal file
View File

@@ -0,0 +1,32 @@
use tokio::task::JoinHandle;
use tracing::warn;
/// Helper for joining multiple task handles with proper error handling.
///
/// This function waits for all tasks to complete and reports any that panicked.
/// Returns an error if any task panicked, otherwise returns Ok.
pub async fn join_tasks(handles: Vec<JoinHandle<()>>) -> Result<(), anyhow::Error> {
let results = futures::future::join_all(handles).await;
let failed = results.iter().filter(|r| r.is_err()).count();
if failed > 0 {
warn!(failed_count = failed, "Some tasks panicked during shutdown");
Err(anyhow::anyhow!("{} task(s) panicked", failed))
} else {
Ok(())
}
}
/// Helper for joining multiple task handles with a timeout.
///
/// Waits for all tasks to complete within the specified timeout.
/// If timeout occurs, remaining tasks are aborted.
pub async fn join_tasks_with_timeout(
handles: Vec<JoinHandle<()>>,
timeout: std::time::Duration,
) -> Result<(), anyhow::Error> {
match tokio::time::timeout(timeout, join_tasks(handles)).await {
Ok(result) => result,
Err(_) => Err(anyhow::anyhow!("Task join timed out after {:?}", timeout)),
}
}

96
src/web/assets.rs Normal file
View File

@@ -0,0 +1,96 @@
//! Embedded assets for the web frontend
//!
//! This module handles serving static assets that are embedded into the binary
//! at compile time using rust-embed.
use dashmap::DashMap;
use once_cell::sync::Lazy;
use rapidhash::v3::rapidhash_v3;
use rust_embed::RustEmbed;
use std::fmt;
/// Embedded web assets from the dist directory
#[derive(RustEmbed)]
#[folder = "web/dist/"]
#[include = "*"]
#[exclude = "*.map"]
pub struct WebAssets;
/// RapidHash hash type for asset content (u64 native output size)
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct AssetHash(u64);
impl AssetHash {
/// Create a new AssetHash from u64 value
pub fn new(hash: u64) -> Self {
Self(hash)
}
/// Get the hash as a hex string
pub fn to_hex(&self) -> String {
format!("{:016x}", self.0)
}
/// Get the hash as a quoted hex string
pub fn quoted(&self) -> String {
format!("\"{}\"", self.to_hex())
}
}
impl fmt::Display for AssetHash {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.to_hex())
}
}
/// Metadata for an asset including MIME type and RapidHash hash
#[derive(Debug, Clone)]
pub struct AssetMetadata {
pub mime_type: Option<String>,
pub hash: AssetHash,
}
impl AssetMetadata {
/// Check if the etag matches the asset hash
pub fn etag_matches(&self, etag: &str) -> bool {
// Remove quotes if present (ETags are typically quoted)
let etag = etag.trim_matches('"');
// ETags generated from u64 hex should be 16 characters
etag.len() == 16
// Parse the hexadecimal, compare if it matches
&& etag.parse::<u64>()
.map(|parsed| parsed == self.hash.0)
.unwrap_or(false)
}
}
/// Global cache for asset metadata to avoid repeated calculations
static ASSET_CACHE: Lazy<DashMap<String, AssetMetadata>> = Lazy::new(DashMap::new);
/// Get cached asset metadata for a file path, caching on-demand
/// Returns AssetMetadata containing MIME type and RapidHash hash
pub fn get_asset_metadata_cached(path: &str, content: &[u8]) -> AssetMetadata {
// Check cache first
if let Some(cached) = ASSET_CACHE.get(path) {
return cached.value().clone();
}
// Calculate MIME type
let mime_type = mime_guess::from_path(path)
.first()
.map(|mime| mime.to_string());
// Calculate RapidHash hash (using u64 native output size)
let hash_value = rapidhash_v3(content);
let hash = AssetHash::new(hash_value);
let metadata = AssetMetadata { mime_type, hash };
// Only cache if we haven't exceeded the limit
if ASSET_CACHE.len() < 1000 {
ASSET_CACHE.insert(path.to_string(), metadata.clone());
}
metadata
}

6
src/web/mod.rs Normal file
View File

@@ -0,0 +1,6 @@
//! Web API module for the banner application.
pub mod assets;
pub mod routes;
pub use routes::*;

296
src/web/routes.rs Normal file
View File

@@ -0,0 +1,296 @@
//! Web API endpoints for Banner bot monitoring and metrics.
use axum::{
Router,
body::Body,
extract::{Request, State},
http::{HeaderMap, HeaderValue, StatusCode, Uri},
response::{Html, IntoResponse, Json, Response},
routing::get,
};
use http::header;
use serde::Serialize;
use serde_json::{Value, json};
use std::{collections::BTreeMap, time::Duration};
use tower_http::timeout::TimeoutLayer;
use tower_http::{
classify::ServerErrorsFailureClass,
cors::{Any, CorsLayer},
trace::TraceLayer,
};
use tracing::{Span, debug, info, warn};
use crate::web::assets::{WebAssets, get_asset_metadata_cached};
/// Set appropriate caching headers based on asset type
fn set_caching_headers(response: &mut Response, path: &str, etag: &str) {
let headers = response.headers_mut();
// Set ETag
if let Ok(etag_value) = HeaderValue::from_str(etag) {
headers.insert(header::ETAG, etag_value);
}
// Set Cache-Control based on asset type
let cache_control = if path.starts_with("assets/") {
// Static assets with hashed filenames - long-term cache
"public, max-age=31536000, immutable"
} else if path == "index.html" {
// HTML files - short-term cache
"public, max-age=300"
} else {
match path.split_once('.').map(|(_, extension)| extension) {
Some(ext) => match ext {
// CSS/JS files - medium-term cache
"css" | "js" => "public, max-age=86400",
// Images - long-term cache
"png" | "jpg" | "jpeg" | "gif" | "svg" | "ico" => "public, max-age=2592000",
// Default for other files
_ => "public, max-age=3600",
},
// Default for files without an extension
None => "public, max-age=3600",
}
};
if let Ok(cache_control_value) = HeaderValue::from_str(cache_control) {
headers.insert(header::CACHE_CONTROL, cache_control_value);
}
}
/// Shared application state for web server
#[derive(Clone)]
pub struct BannerState {}
/// Creates the web server router
pub fn create_router(state: BannerState) -> Router {
let api_router = Router::new()
.route("/health", get(health))
.route("/status", get(status))
.route("/metrics", get(metrics))
.with_state(state);
let mut router = Router::new().nest("/api", api_router);
if cfg!(debug_assertions) {
router = router.layer(
CorsLayer::new()
.allow_origin(Any)
.allow_methods(Any)
.allow_headers(Any),
)
} else {
router = router.fallback(fallback);
}
router.layer((
TraceLayer::new_for_http()
.make_span_with(|request: &Request<Body>| {
tracing::debug_span!("request", path = request.uri().path())
})
.on_request(())
.on_body_chunk(())
.on_eos(())
.on_response(
|response: &Response<Body>, latency: Duration, _span: &Span| {
let latency_threshold = if cfg!(debug_assertions) {
Duration::from_millis(100)
} else {
Duration::from_millis(1000)
};
// Format latency, status, and code
let (latency_str, status) = (
format!("{latency:.2?}"),
format!(
"{} {}",
response.status().as_u16(),
response.status().canonical_reason().unwrap_or("??")
),
);
// Log in warn if latency is above threshold, otherwise debug
if latency > latency_threshold {
warn!(latency = latency_str, status = status, "Response");
} else {
debug!(latency = latency_str, status = status, "Response");
}
},
)
.on_failure(
|error: ServerErrorsFailureClass, latency: Duration, _span: &Span| {
warn!(
error = ?error,
latency = format!("{latency:.2?}"),
"Request failed"
);
},
),
TimeoutLayer::new(Duration::from_secs(10)),
))
}
/// Handler that extracts request information for caching
async fn fallback(request: Request) -> Response {
let uri = request.uri().clone();
let headers = request.headers().clone();
handle_spa_fallback_with_headers(uri, headers).await
}
/// Handles SPA routing by serving index.html for non-API, non-asset requests
/// This version includes HTTP caching headers and ETag support
async fn handle_spa_fallback_with_headers(uri: Uri, request_headers: HeaderMap) -> Response {
let path = uri.path().trim_start_matches('/');
if let Some(content) = WebAssets::get(path) {
// Get asset metadata (MIME type and hash) with caching
let metadata = get_asset_metadata_cached(path, &content.data);
// Check if client has a matching ETag (conditional request)
if let Some(etag) = request_headers.get(header::IF_NONE_MATCH)
&& metadata.etag_matches(etag.to_str().unwrap())
{
return StatusCode::NOT_MODIFIED.into_response();
}
// Use cached MIME type, only set Content-Type if we have a valid MIME type
let mut response = (
[(
header::CONTENT_TYPE,
// For unknown types, set to application/octet-stream
metadata
.mime_type
.unwrap_or("application/octet-stream".to_string()),
)],
content.data,
)
.into_response();
// Set caching headers
set_caching_headers(&mut response, path, &metadata.hash.quoted());
return response;
} else {
// Any assets that are not found should be treated as a 404, not falling back to the SPA index.html
if path.starts_with("assets/") {
return (StatusCode::NOT_FOUND, "Asset not found").into_response();
}
}
// Fall back to the SPA index.html
match WebAssets::get("index.html") {
Some(content) => {
let metadata = get_asset_metadata_cached("index.html", &content.data);
// Check if client has a matching ETag for index.html
if let Some(etag) = request_headers.get(header::IF_NONE_MATCH)
&& metadata.etag_matches(etag.to_str().unwrap())
{
return StatusCode::NOT_MODIFIED.into_response();
}
let mut response = Html(content.data).into_response();
set_caching_headers(&mut response, "index.html", &metadata.hash.quoted());
response
}
None => (
StatusCode::INTERNAL_SERVER_ERROR,
"Failed to load index.html",
)
.into_response(),
}
}
/// Health check endpoint
async fn health() -> Json<Value> {
info!("health check requested");
Json(json!({
"status": "healthy",
"timestamp": chrono::Utc::now().to_rfc3339()
}))
}
#[derive(Serialize)]
enum Status {
Disabled,
Connected,
Active,
Healthy,
Error,
}
#[derive(Serialize)]
struct ServiceInfo {
name: String,
status: Status,
}
#[derive(Serialize)]
struct StatusResponse {
status: Status,
version: String,
commit: String,
services: BTreeMap<String, ServiceInfo>,
}
/// Status endpoint showing bot and system status
async fn status(State(_state): State<BannerState>) -> Json<StatusResponse> {
let mut services = BTreeMap::new();
// Bot service status - hardcoded as disabled for now
services.insert(
"bot".to_string(),
ServiceInfo {
name: "Bot".to_string(),
status: Status::Disabled,
},
);
// Banner API status - always connected for now
services.insert(
"banner".to_string(),
ServiceInfo {
name: "Banner".to_string(),
status: Status::Connected,
},
);
// Discord status - hardcoded as disabled for now
services.insert(
"discord".to_string(),
ServiceInfo {
name: "Discord".to_string(),
status: Status::Disabled,
},
);
let overall_status = if services.values().any(|s| matches!(s.status, Status::Error)) {
Status::Error
} else if services
.values()
.all(|s| matches!(s.status, Status::Active | Status::Connected))
{
Status::Active
} else {
// If we have any Disabled services but no errors, show as Healthy
Status::Healthy
};
Json(StatusResponse {
status: overall_status,
version: env!("CARGO_PKG_VERSION").to_string(),
commit: env!("GIT_COMMIT_HASH").to_string(),
services,
})
}
/// Metrics endpoint for monitoring
async fn metrics(State(_state): State<BannerState>) -> Json<Value> {
// For now, return basic metrics structure
Json(json!({
"banner_api": {
"status": "connected"
},
"timestamp": chrono::Utc::now().to_rfc3339()
}))
}

39
tests/basic_test.rs Normal file
View File

@@ -0,0 +1,39 @@
use banner::utils::shutdown::join_tasks;
use tokio::task::JoinHandle;
#[tokio::test]
async fn test_join_tasks_success() {
// Create some tasks that complete successfully
let handles: Vec<JoinHandle<()>> = vec![
tokio::spawn(async { tokio::time::sleep(tokio::time::Duration::from_millis(10)).await }),
tokio::spawn(async { tokio::time::sleep(tokio::time::Duration::from_millis(20)).await }),
tokio::spawn(async { /* immediate completion */ }),
];
// All tasks should complete successfully
let result = join_tasks(handles).await;
assert!(
result.is_ok(),
"Expected all tasks to complete successfully"
);
}
#[tokio::test]
async fn test_join_tasks_with_panic() {
// Create some tasks, including one that panics
let handles: Vec<JoinHandle<()>> = vec![
tokio::spawn(async { tokio::time::sleep(tokio::time::Duration::from_millis(10)).await }),
tokio::spawn(async { panic!("intentional test panic") }),
tokio::spawn(async { /* immediate completion */ }),
];
// Should return an error because one task panicked
let result = join_tasks(handles).await;
assert!(result.is_err(), "Expected an error when a task panics");
let error_msg = result.unwrap_err().to_string();
assert!(
error_msg.contains("1 task(s) panicked"),
"Error message should mention panicked tasks"
);
}

9
web/.gitignore vendored Normal file
View File

@@ -0,0 +1,9 @@
node_modules
.DS_Store
dist
dist-ssr
*.local
count.txt
.env
.nitro
.tanstack

11
web/.vscode/settings.json vendored Normal file
View File

@@ -0,0 +1,11 @@
{
"files.watcherExclude": {
"**/routeTree.gen.ts": true
},
"search.exclude": {
"**/routeTree.gen.ts": true
},
"files.readonlyInclude": {
"**/routeTree.gen.ts": true
}
}

30
web/biome.json Normal file
View File

@@ -0,0 +1,30 @@
{
"$schema": "https://biomejs.dev/schemas/1.9.4/schema.json",
"vcs": {
"enabled": true,
"clientKind": "git",
"useIgnoreFile": true
},
"files": {
"ignoreUnknown": false,
"ignore": ["dist/", "node_modules/", ".tanstack/"]
},
"formatter": {
"enabled": true,
"indentStyle": "space",
"indentWidth": 2,
"lineWidth": 100,
"lineEnding": "lf"
},
"javascript": {
"formatter": {
"quoteStyle": "double",
"trailingCommas": "es5",
"semicolons": "always",
"arrowParentheses": "always"
}
},
"linter": {
"enabled": false
}
}

1297
web/bun.lock Normal file
View File

File diff suppressed because it is too large Load Diff

60
web/eslint.config.js Normal file
View File

@@ -0,0 +1,60 @@
import js from "@eslint/js";
import tseslint from "typescript-eslint";
import react from "eslint-plugin-react";
import reactHooks from "eslint-plugin-react-hooks";
import reactRefresh from "eslint-plugin-react-refresh";
export default tseslint.config(
// Ignore generated files and build outputs
{
ignores: ["dist", "node_modules", "src/routeTree.gen.ts", "*.config.js"],
},
// Base configs
js.configs.recommended,
...tseslint.configs.recommendedTypeChecked,
// React plugin configuration
{
files: ["**/*.{ts,tsx}"],
plugins: {
react,
"react-hooks": reactHooks,
"react-refresh": reactRefresh,
},
languageOptions: {
parserOptions: {
project: true,
tsconfigRootDir: import.meta.dirname,
ecmaFeatures: {
jsx: true,
},
},
},
settings: {
react: {
version: "19.0",
},
},
rules: {
// React rules
...react.configs.recommended.rules,
...react.configs["jsx-runtime"].rules,
...reactHooks.configs.recommended.rules,
// React Refresh
"react-refresh/only-export-components": ["warn", { allowConstantExport: true }],
// TypeScript overrides
"@typescript-eslint/no-unused-vars": [
"error",
{
argsIgnorePattern: "^_",
varsIgnorePattern: "^_",
},
],
"@typescript-eslint/no-explicit-any": "warn",
// Disable prop-types since we're using TypeScript
"react/prop-types": "off",
},
}
);

20
web/index.html Normal file
View File

@@ -0,0 +1,20 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<link rel="icon" href="/favicon.ico" />
<meta name="theme-color" content="#000000" />
<meta
name="description"
content="Banner, a Discord bot and web interface for UTSA Course Monitoring"
/>
<link rel="apple-touch-icon" href="/logo192.png" />
<link rel="manifest" href="/manifest.json" />
<title>Banner</title>
</head>
<body>
<div id="app"></div>
<script type="module" src="/src/main.tsx"></script>
</body>
</html>

49
web/package.json Normal file
View File

@@ -0,0 +1,49 @@
{
"name": "banner-web",
"private": true,
"type": "module",
"scripts": {
"dev": "vite --port 3000",
"start": "vite --port 3000",
"build": "vite build && tsc",
"serve": "vite preview",
"test": "vitest run",
"lint": "tsc && eslint . --ext .ts,.tsx",
"typecheck": "tsc --noEmit",
"format": "biome format --write .",
"format:check": "biome format ."
},
"dependencies": {
"@radix-ui/themes": "^3.2.1",
"@tanstack/react-devtools": "^0.2.2",
"@tanstack/react-router": "^1.130.2",
"@tanstack/react-router-devtools": "^1.131.5",
"@tanstack/router-plugin": "^1.121.2",
"lucide-react": "^0.544.0",
"next-themes": "^0.4.6",
"react": "^19.0.0",
"react-dom": "^19.0.0",
"react-timeago": "^8.3.0",
"recharts": "^3.2.0"
},
"devDependencies": {
"@biomejs/biome": "^1.9.4",
"@eslint/js": "^9.39.0",
"@testing-library/dom": "^10.4.0",
"@testing-library/react": "^16.2.0",
"@types/node": "^24.3.3",
"@types/react": "^19.0.8",
"@types/react-dom": "^19.0.3",
"@vitejs/plugin-react": "^4.3.4",
"eslint": "^9.39.0",
"eslint-plugin-react": "^7.37.5",
"eslint-plugin-react-hooks": "^7.0.1",
"eslint-plugin-react-refresh": "^0.4.24",
"jsdom": "^26.0.0",
"typescript": "^5.7.2",
"typescript-eslint": "^8.46.2",
"vite": "^6.3.5",
"vitest": "^3.0.5",
"web-vitals": "^4.2.4"
}
}

BIN
web/public/favicon.ico Normal file
View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.8 KiB

BIN
web/public/logo192.png Normal file
View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.2 KiB

BIN
web/public/logo512.png Normal file
View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.4 KiB

25
web/public/manifest.json Normal file
View File

@@ -0,0 +1,25 @@
{
"short_name": "Banner",
"name": "Banner, a Discord bot and web interface for UTSA Course Monitoring",
"icons": [
{
"src": "favicon.ico",
"sizes": "64x64 32x32 24x24 16x16",
"type": "image/x-icon"
},
{
"src": "logo192.png",
"type": "image/png",
"sizes": "192x192"
},
{
"src": "logo512.png",
"type": "image/png",
"sizes": "512x512"
}
],
"start_url": ".",
"display": "standalone",
"theme_color": "#ffffff",
"background_color": "#ffffff"
}

3
web/public/robots.txt Normal file
View File

@@ -0,0 +1,3 @@
# https://www.robotstxt.org/robotstxt.html
User-agent: *
Disallow:

34
web/src/App.css Normal file
View File

@@ -0,0 +1,34 @@
.App {
min-height: 100vh;
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Roboto", "Oxygen", "Ubuntu",
"Cantarell", "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;
background-color: var(--color-background);
color: var(--color-text);
}
@keyframes pulse {
0%,
100% {
opacity: 0.2;
}
50% {
opacity: 0.4;
}
}
.animate-pulse {
animation: pulse 2s ease-in-out infinite;
}
/* Screen reader only text */
.sr-only {
position: absolute;
width: 1px;
height: 1px;
padding: 0;
margin: -1px;
overflow: hidden;
clip: rect(0, 0, 0, 0);
white-space: nowrap;
border: 0;
}

View File

@@ -0,0 +1,60 @@
import { useTheme } from "next-themes";
import { Button } from "@radix-ui/themes";
import { Sun, Moon, Monitor } from "lucide-react";
import { useMemo } from "react";
export function ThemeToggle() {
const { theme, setTheme } = useTheme();
const nextTheme = useMemo(() => {
switch (theme) {
case "light":
return "dark";
case "dark":
return "system";
case "system":
return "light";
default:
console.error(`Invalid theme: ${theme}`);
return "system";
}
}, [theme]);
const icon = useMemo(() => {
if (nextTheme === "system") {
return <Monitor size={18} />;
}
return nextTheme === "dark" ? <Moon size={18} /> : <Sun size={18} />;
}, [nextTheme]);
return (
<Button
variant="ghost"
size="3"
onClick={() => setTheme(nextTheme)}
style={{
cursor: "pointer",
backgroundColor: "transparent",
border: "none",
margin: "4px",
padding: "7px",
borderRadius: "6px",
display: "flex",
alignItems: "center",
justifyContent: "center",
color: "var(--gray-11)",
transition: "background-color 0.2s, color 0.2s",
transform: "scale(1.25)",
}}
onMouseEnter={(e) => {
e.currentTarget.style.backgroundColor = "var(--gray-4)";
}}
onMouseLeave={(e) => {
e.currentTarget.style.backgroundColor = "transparent";
}}
>
{icon}
<span className="sr-only">Toggle theme</span>
</Button>
);
}

63
web/src/lib/api.test.ts Normal file
View File

@@ -0,0 +1,63 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { BannerApiClient } from "./api";
// Mock fetch
global.fetch = vi.fn();
describe("BannerApiClient", () => {
let apiClient: BannerApiClient;
beforeEach(() => {
apiClient = new BannerApiClient();
vi.clearAllMocks();
});
it("should fetch health data", async () => {
const mockHealth = {
status: "healthy",
timestamp: "2024-01-01T00:00:00Z",
};
vi.mocked(fetch).mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve(mockHealth),
} as Response);
const result = await apiClient.getHealth();
expect(fetch).toHaveBeenCalledWith("/api/health");
expect(result).toEqual(mockHealth);
});
it("should fetch status data", async () => {
const mockStatus = {
status: "operational",
bot: { status: "running", uptime: "1h" },
cache: { status: "connected", courses: "100", subjects: "50" },
banner_api: { status: "connected" },
timestamp: "2024-01-01T00:00:00Z",
};
vi.mocked(fetch).mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve(mockStatus),
} as Response);
const result = await apiClient.getStatus();
expect(fetch).toHaveBeenCalledWith("/api/status");
expect(result).toEqual(mockStatus);
});
it("should handle API errors", async () => {
vi.mocked(fetch).mockResolvedValueOnce({
ok: false,
status: 500,
statusText: "Internal Server Error",
} as Response);
await expect(apiClient.getHealth()).rejects.toThrow(
"API request failed: 500 Internal Server Error"
);
});
});

61
web/src/lib/api.ts Normal file
View File

@@ -0,0 +1,61 @@
// API client for Banner backend
const API_BASE_URL = "/api";
export interface HealthResponse {
status: string;
timestamp: string;
}
export type Status = "Disabled" | "Connected" | "Active" | "Healthy" | "Error";
export interface ServiceInfo {
name: string;
status: Status;
}
export interface StatusResponse {
status: Status;
version: string;
commit: string;
services: Record<string, ServiceInfo>;
}
export interface MetricsResponse {
banner_api: {
status: string;
};
timestamp: string;
}
export class BannerApiClient {
private baseUrl: string;
constructor(baseUrl: string = API_BASE_URL) {
this.baseUrl = baseUrl;
}
private async request<T>(endpoint: string): Promise<T> {
const response = await fetch(`${this.baseUrl}${endpoint}`);
if (!response.ok) {
throw new Error(`API request failed: ${response.status} ${response.statusText}`);
}
return (await response.json()) as T;
}
async getHealth(): Promise<HealthResponse> {
return this.request<HealthResponse>("/health");
}
async getStatus(): Promise<StatusResponse> {
return this.request<StatusResponse>("/status");
}
async getMetrics(): Promise<MetricsResponse> {
return this.request<MetricsResponse>("/metrics");
}
}
// Export a default instance
export const client = new BannerApiClient();

44
web/src/logo.svg Normal file
View File

@@ -0,0 +1,44 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg id="Layer_1"
xmlns="http://www.w3.org/2000/svg" version="1.1" viewBox="0 0 841.9 595.3">
<!-- Generator: Adobe Illustrator 29.3.0, SVG Export Plug-In . SVG Version: 2.1.0 Build 146) -->
<defs>
<style>
.st0 {
fill: #9ae7fc;
}
.st1 {
fill: #61dafb;
}
</style>
</defs>
<g>
<path class="st1" d="M666.3,296.5c0-32.5-40.7-63.3-103.1-82.4,14.4-63.6,8-114.2-20.2-130.4-6.5-3.8-14.1-5.6-22.4-5.6v22.3c4.6,0,8.3.9,11.4,2.6,13.6,7.8,19.5,37.5,14.9,75.7-1.1,9.4-2.9,19.3-5.1,29.4-19.6-4.8-41-8.5-63.5-10.9-13.5-18.5-27.5-35.3-41.6-50,32.6-30.3,63.2-46.9,84-46.9v-22.3c-27.5,0-63.5,19.6-99.9,53.6-36.4-33.8-72.4-53.2-99.9-53.2v22.3c20.7,0,51.4,16.5,84,46.6-14,14.7-28,31.4-41.3,49.9-22.6,2.4-44,6.1-63.6,11-2.3-10-4-19.7-5.2-29-4.7-38.2,1.1-67.9,14.6-75.8,3-1.8,6.9-2.6,11.5-2.6v-22.3c-8.4,0-16,1.8-22.6,5.6-28.1,16.2-34.4,66.7-19.9,130.1-62.2,19.2-102.7,49.9-102.7,82.3s40.7,63.3,103.1,82.4c-14.4,63.6-8,114.2,20.2,130.4,6.5,3.8,14.1,5.6,22.5,5.6,27.5,0,63.5-19.6,99.9-53.6,36.4,33.8,72.4,53.2,99.9,53.2,8.4,0,16-1.8,22.6-5.6,28.1-16.2,34.4-66.7,19.9-130.1,62-19.1,102.5-49.9,102.5-82.3zm-130.2-66.7c-3.7,12.9-8.3,26.2-13.5,39.5-4.1-8-8.4-16-13.1-24-4.6-8-9.5-15.8-14.4-23.4,14.2,2.1,27.9,4.7,41,7.9zm-45.8,106.5c-7.8,13.5-15.8,26.3-24.1,38.2-14.9,1.3-30,2-45.2,2s-30.2-.7-45-1.9c-8.3-11.9-16.4-24.6-24.2-38-7.6-13.1-14.5-26.4-20.8-39.8,6.2-13.4,13.2-26.8,20.7-39.9,7.8-13.5,15.8-26.3,24.1-38.2,14.9-1.3,30-2,45.2-2s30.2.7,45,1.9c8.3,11.9,16.4,24.6,24.2,38,7.6,13.1,14.5,26.4,20.8,39.8-6.3,13.4-13.2,26.8-20.7,39.9zm32.3-13c5.4,13.4,10,26.8,13.8,39.8-13.1,3.2-26.9,5.9-41.2,8,4.9-7.7,9.8-15.6,14.4-23.7,4.6-8,8.9-16.1,13-24.1zm-101.4,106.7c-9.3-9.6-18.6-20.3-27.8-32,9,.4,18.2.7,27.5.7s18.7-.2,27.8-.7c-9,11.7-18.3,22.4-27.5,32zm-74.4-58.9c-14.2-2.1-27.9-4.7-41-7.9,3.7-12.9,8.3-26.2,13.5-39.5,4.1,8,8.4,16,13.1,24s9.5,15.8,14.4,23.4zm73.9-208.1c9.3,9.6,18.6,20.3,27.8,32-9-.4-18.2-.7-27.5-.7s-18.7.2-27.8.7c9-11.7,18.3-22.4,27.5-32zm-74,58.9c-4.9,7.7-9.8,15.6-14.4,23.7-4.6,8-8.9,16-13,24-5.4-13.4-10-26.8-13.8-39.8,13.1-3.1,26.9-5.8,41.2-7.9zm-90.5,125.2c-35.4-15.1-58.3-34.9-58.3-50.6s22.9-35.6,58.3-50.6c8.6-3.7,18-7,27.7-10.1,5.7,19.6,13.2,40,22.5,60.9-9.2,20.8-16.6,41.1-22.2,60.6-9.9-3.1-19.3-6.5-28-10.2zm53.8,142.9c-13.6-7.8-19.5-37.5-14.9-75.7,1.1-9.4,2.9-19.3,5.1-29.4,19.6,4.8,41,8.5,63.5,10.9,13.5,18.5,27.5,35.3,41.6,50-32.6,30.3-63.2,46.9-84,46.9-4.5-.1-8.3-1-11.3-2.7zm237.2-76.2c4.7,38.2-1.1,67.9-14.6,75.8-3,1.8-6.9,2.6-11.5,2.6-20.7,0-51.4-16.5-84-46.6,14-14.7,28-31.4,41.3-49.9,22.6-2.4,44-6.1,63.6-11,2.3,10.1,4.1,19.8,5.2,29.1zm38.5-66.7c-8.6,3.7-18,7-27.7,10.1-5.7-19.6-13.2-40-22.5-60.9,9.2-20.8,16.6-41.1,22.2-60.6,9.9,3.1,19.3,6.5,28.1,10.2,35.4,15.1,58.3,34.9,58.3,50.6,0,15.7-23,35.6-58.4,50.6zm-264.9-268.7z"/>
<circle class="st1" cx="420.9" cy="296.5" r="45.7"/>
<path class="st1" d="M520.5,78.1"/>
</g>
<circle class="st0" cx="420.8" cy="296.6" r="43"/>
<path class="st1" d="M466.1,296.6c0,25-20.2,45.2-45.2,45.2s-45.2-20.2-45.2-45.2,20.2-45.2,45.2-45.2,45.2,20.2,45.2,45.2ZM386,295.6v-6.3c0-1.1,1.2-5.1,1.8-6.2,1-1.9,2.9-3.5,4.6-4.7l-3.4-3.4c4-3.6,9.4-3.7,13.7-.7,1.9-4.7,6.6-7.1,11.6-6.7l-.8,4.2c5.9.2,13.1,4.1,13.1,10.8s0,.5-.7.7c-1.7.3-3.4-.4-5-.6s-1.2-.4-1.2.3,2.5,4.1,3,5.5,1,3.5.8,5.3c-5.6-.8-10.5-3.2-14.8-6.7.3,2.6,4.1,21.7,5.3,21.9s.8-.6,1-1.1,1.3-6.3,1.3-6.7c0-1-1.7-1.8-2.2-2.8-1.2-2.7,1.3-4.7,3.7-3.3s5.2,6.2,7.5,7.3,13,1.4,14.8,3.3-2.9,4.6-1.5,7.6c6.7-2.6,13.5-3.3,20.6-2.5,3.1-9.7,3.1-20.3-.9-29.8-7.3,0-14.7-3.6-17.2-10.8-2.5-7.2-.7-8.6-1.3-9.3-.8-1-6.3.6-7.4-1.5s.3-1.1-.2-1.4-1.9-.6-2.6-.8c-26-6.4-51.3,15.7-49.7,42.1,0,1.6,1.6,10.3,2.4,11.1s4.8,0,6.3,0,3.7.3,5,.5c2.9.4,7.2,2.4,9.4,2.5s2.4-.8,2.7-2.4c.4-2.6.5-7.4.5-10.1s-1-7.8-1.3-11.6c-.9-.2-.7,0-.9.5-.7,1.3-1.1,3.2-1.9,4.8s-5.2,8.7-5.7,9-.7-.5-.8-.8c-1.6-3.5-2-7.9-1.9-11.8-.9-1-5.4,4.9-6.7,5.3l-.8-.4v-.3h-.2ZM455.6,276.4c1.1-1.2-6-8.9-7.2-10-3-2.7-5.4-4.5-3.5,1.4s5.7,7.8,10.6,8.5h.1ZM410.9,270.1c-.4-.5-6.1,2.9-5.5,4.6,1.9-1.3,5.9-1.7,5.5-4.6ZM400.4,276.4c-.3-2.4-6.3-2.7-7.2-1s1.6,1.4,1.9,1.4c1.8.3,3.5-.6,5.2-.4h.1ZM411.3,276.8c3.8,1.3,6.6,3.6,10.9,3.7s0-3-1.2-3.9c-2.2-1.7-5.1-2.4-7.8-2.4s-1.6-.3-1.4.4c2.8.6,7.3.7,8.4,3.8-2.3-.3-3.9-1.6-6.2-2s-2.5-.5-2.6.3h0ZM420.6,290.3c-.8-5.1-5.7-10.8-10.9-11.6s-1.3-.4-.8.5,4.7,3.2,5.7,4,4.5,4.2,2.1,3.8-8.4-7.8-9.4-6.7c.2.9,1.1,1.9,1.7,2.7,3,3.8,6.9,6.8,11.8,7.4h-.2ZM395.3,279.8c-5,1.1-6.9,6.3-6.7,11,.7.8,5-3.8,5.4-4.5s2.7-4.6,1.1-4-2.9,4.4-4.2,4.6.2-2.1.4-2.5c1.1-1.6,2.9-3.1,4-4.6h0ZM400.4,281.5c-.4-.5-2,1.3-2.3,1.7-2.9,3.9-2.6,10.2-1.5,14.8.8.2.8-.3,1.2-.7,3-3.8,5.5-10.5,4.5-15.4-2.1,3.1-3.1,7.3-3.6,11h-1.3c0-4,1.9-7.7,3-11.4h0ZM426.9,305.9c0-1.7-1.7-1.4-2.5-1.9s-1.3-1.9-3-1.4c1.3,2.1,3,3.2,5.5,3.4h0ZM417.2,308.5c7.6.7,5.5-1.9,1.4-5.5-1.3-.3-1.5,4.5-1.4,5.5ZM437,309.7c-3.5-.3-7.8-2-11.2-2.1s-1.3,0-1.9.7c4,1.3,8.4,1.7,12.1,4l1-2.5h0ZM420.5,312.8c-7.3,0-15.1,3.7-20.4,8.8s-4.8,5.3-4.8,6.2c0,1.8,8.6,6.2,10.5,6.8,12.1,4.8,27.5,3.5,38.2-4.2s3.1-2.7,0-6.2c-5.7-6.6-14.7-11.4-23.4-11.3h-.1ZM398.7,316.9c-1.4-1.4-5-1.9-7-2.1s-5.3-.3-6.9.6l13.9,1.4h0ZM456.9,314.8h-7.4c-.9,0-4.9,1.1-6,1.6s-.8.6,0,.5c2.4,0,5.1-1,7.6-1.3s3.5.2,5.1,0,1.3-.3.6-.8h0Z"/>
<path class="st0" d="M386,295.6l.8.4c1.3-.3,5.8-6.2,6.7-5.3,0,3.9.3,8.3,1.9,11.8s0,1.2.8.8,5.1-7.8,5.7-9,1.3-3.5,1.9-4.8,0-.7.9-.5c.3,3.8,1.2,7.8,1.3,11.6s0,7.5-.5,10.1-1.1,2.4-2.7,2.4-6.5-2.1-9.4-2.5-3.7-.5-5-.5-5.4,1.1-6.3,0-2.2-9.5-2.4-11.1c-1.5-26.4,23.7-48.5,49.7-42.1s2.2.4,2.6.8,0,1,.2,1.4c1.1,2,6.5.5,7.4,1.5s.4,6.9,1.3,9.3c2.5,7.2,10,10.9,17.2,10.8,4,9.4,4,20.1.9,29.8-7.2-.7-13.9,0-20.6,2.5-1.3-3.1,4.1-5.1,1.5-7.6s-11.8-1.9-14.8-3.3-5.4-6.1-7.5-7.3-4.9.6-3.7,3.3,2.1,1.8,2.2,2.8-1,6.2-1.3,6.7-.3,1.3-1,1.1c-1.1-.3-5-19.3-5.3-21.9,4.3,3.5,9.2,5.9,14.8,6.7.2-1.9-.3-3.5-.8-5.3s-3-5.1-3-5.5c0-.8.9-.3,1.2-.3,1.6,0,3.3.8,5,.6s.7.3.7-.7c0-6.6-7.2-10.6-13.1-10.8l.8-4.2c-5.1-.3-9.6,2-11.6,6.7-4.3-3-9.8-3-13.7.7l3.4,3.4c-1.8,1.3-3.5,2.8-4.6,4.7s-1.8,5.1-1.8,6.2v6.6h.2ZM431.6,265c7.8,2.1,8.7-3.5.2-1.3l-.2,1.3ZM432.4,270.9c.3.6,6.4-.4,5.8-2.3s-4.6.6-5.7.6l-.2,1.7h.1ZM434.5,276c.8,1.2,5.7-1.8,5.5-2.7-.4-1.9-6.6,1.2-5.5,2.7ZM442.9,276.4c-.9-.9-5,2.8-4.6,4,.6,2.4,5.7-3,4.6-4ZM445.1,279.9c-.3.2-3.1,4.6-1.5,5s3.5-3.4,3.5-4-1.3-1.3-2-.9h0ZM448.9,287.4c2.1.8,3.8-5.1,2.3-5.5-1.9-.6-2.6,5.1-2.3,5.5ZM457.3,288.6c.5-1.7,1.1-4.7-1-5.5-1,.3-.6,3.9-.6,4.8l.3.5,1.3.2h0Z"/>
<path class="st0" d="M455.6,276.4c-5-.8-9.1-3.6-10.6-8.5s.5-4,3.5-1.4,8.3,8.7,7.2,10h-.1Z"/>
<path class="st0" d="M420.6,290.3c-4.9-.6-8.9-3.6-11.8-7.4s-1.5-1.8-1.7-2.7c1-1,8.5,6.6,9.4,6.7,2.4.4-1.8-3.5-2.1-3.8-1-.8-5.4-3.5-5.7-4-.4-.8.5-.5.8-.5,5.2.8,10.1,6.6,10.9,11.6h.2Z"/>
<path class="st0" d="M400.4,281.5c-1.1,3.7-3,7.3-3,11.4h1.3c.5-3.7,1.5-7.8,3.6-11,1,4.8-1.5,11.6-4.5,15.4s-.4.8-1.2.7c-1.1-4.5-1.3-10.8,1.5-14.8s1.9-2.2,2.3-1.7h0Z"/>
<path class="st0" d="M411.3,276.8c0-.8,2.1-.4,2.6-.3,2.4.4,4,1.7,6.2,2-1.2-3.1-5.7-3.2-8.4-3.8,0-.8.9-.4,1.4-.4,2.8,0,5.6.7,7.8,2.4,2.2,1.7,4,4,1.2,3.9-4.3,0-7.1-2.4-10.9-3.7h0Z"/>
<path class="st0" d="M395.3,279.8c-1.1,1.6-3,3-4,4.6s-1.9,2.8-.4,2.5,2.8-4,4.2-4.6-.9,3.6-1.1,4c-.4.7-4.7,5.2-5.4,4.5-.2-4.6,1.8-9.9,6.7-11h0Z"/>
<path class="st0" d="M437,309.7l-1,2.5c-3.6-2.3-8-2.8-12.1-4,.5-.7,1.1-.7,1.9-.7,3.4,0,7.8,1.8,11.2,2.1h0Z"/>
<path class="st0" d="M417.2,308.5c0-1,0-5.8,1.4-5.5,4,3.5,6.1,6.2-1.4,5.5Z"/>
<path class="st0" d="M400.4,276.4c-1.8-.3-3.5.7-5.2.4s-2.3-.8-1.9-1.4c.8-1.6,6.9-1.4,7.2,1h-.1Z"/>
<path class="st0" d="M410.9,270.1c.4,3-3.6,3.3-5.5,4.6-.6-1.8,5-5.1,5.5-4.6Z"/>
<path class="st0" d="M426.9,305.9c-2.5-.2-4.1-1.3-5.5-3.4,1.7-.4,2,.8,3,1.4s2.6.3,2.5,1.9h0Z"/>
<path class="st1" d="M432.4,270.9l.2-1.7c1.1,0,5.1-2.2,5.7-.6s-5.5,2.9-5.8,2.3h-.1Z"/>
<path class="st1" d="M431.6,265l.2-1.3c8.4-2.1,7.7,3.4-.2,1.3Z"/>
<path class="st1" d="M434.5,276c-1.1-1.5,5.1-4.6,5.5-2.7s-4.6,4-5.5,2.7Z"/>
<path class="st1" d="M442.9,276.4c1.1,1.1-4,6.4-4.6,4s3.7-4.9,4.6-4Z"/>
<path class="st1" d="M445.1,279.9c.7-.4,2.1,0,2,.9s-2.4,4.4-3.5,4,1.3-4.8,1.5-5h0Z"/>
<path class="st1" d="M448.9,287.4c-.3-.3.4-6.1,2.3-5.5,1.4.4-.2,6.2-2.3,5.5Z"/>
<path class="st1" d="M457.3,288.6l-1.3-.2-.3-.5c0-.9-.4-4.6.6-4.8,2.1.8,1.5,3.8,1,5.5h0Z"/>
<path class="st0" d="M420.5,312.8c8.9,0,17.9,4.7,23.4,11.3,5.6,6.6,3.8,3.5,0,6.2-10.7,7.7-26.1,9-38.2,4.2-1.9-.8-10.5-5.1-10.5-6.8s4-5.3,4.8-6.2c5.3-5,13.1-8.6,20.4-8.8h.1Z"/>
<path class="st0" d="M398.7,316.9l-13.9-1.4c1.7-1,5-.8,6.9-.6s5.6.7,7,2.1h0Z"/>
<path class="st0" d="M456.9,314.8c.7.5,0,.8-.6.8-1.6.2-3.5-.2-5.1,0-2.4.3-5.2,1.2-7.6,1.3s-1.1,0,0-.5,5.1-1.6,6-1.6h7.4,0Z"/>
</svg>

After

Width:  |  Height:  |  Size: 8.4 KiB

53
web/src/main.tsx Normal file
View File

@@ -0,0 +1,53 @@
import { StrictMode } from "react";
import ReactDOM from "react-dom/client";
import { RouterProvider, createRouter } from "@tanstack/react-router";
import { ThemeProvider } from "next-themes";
import { Theme } from "@radix-ui/themes";
// Import the generated route tree
import { routeTree } from "./routeTree.gen";
import "./styles.css";
import reportWebVitals from "./reportWebVitals.ts";
// Create a new router instance
const router = createRouter({
routeTree,
context: {},
defaultPreload: "intent",
scrollRestoration: true,
defaultStructuralSharing: true,
defaultPreloadStaleTime: 0,
});
// Register the router instance for type safety
declare module "@tanstack/react-router" {
interface Register {
router: typeof router;
}
}
// Render the app
const rootElement = document.getElementById("app");
if (rootElement && !rootElement.innerHTML) {
const root = ReactDOM.createRoot(rootElement);
root.render(
<StrictMode>
<ThemeProvider
attribute="class"
defaultTheme="system"
enableSystem
disableTransitionOnChange={false}
>
<Theme>
<RouterProvider router={router} />
</Theme>
</ThemeProvider>
</StrictMode>
);
}
// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();

View File

@@ -0,0 +1,13 @@
const reportWebVitals = (onPerfEntry?: () => void) => {
if (onPerfEntry && onPerfEntry instanceof Function) {
void import("web-vitals").then(({ onCLS, onINP, onFCP, onLCP, onTTFB }) => {
onCLS(onPerfEntry);
onINP(onPerfEntry);
onFCP(onPerfEntry);
onLCP(onPerfEntry);
onTTFB(onPerfEntry);
});
}
};
export default reportWebVitals;

59
web/src/routeTree.gen.ts Normal file
View File

@@ -0,0 +1,59 @@
/* eslint-disable */
// @ts-nocheck
// noinspection JSUnusedGlobalSymbols
// This file was automatically generated by TanStack Router.
// You should NOT make any changes in this file as it will be overwritten.
// Additionally, you should also exclude this file from your linter and/or formatter to prevent it from being checked or modified.
import { Route as rootRouteImport } from "./routes/__root";
import { Route as IndexRouteImport } from "./routes/index";
const IndexRoute = IndexRouteImport.update({
id: "/",
path: "/",
getParentRoute: () => rootRouteImport,
} as any);
export interface FileRoutesByFullPath {
"/": typeof IndexRoute;
}
export interface FileRoutesByTo {
"/": typeof IndexRoute;
}
export interface FileRoutesById {
__root__: typeof rootRouteImport;
"/": typeof IndexRoute;
}
export interface FileRouteTypes {
fileRoutesByFullPath: FileRoutesByFullPath;
fullPaths: "/";
fileRoutesByTo: FileRoutesByTo;
to: "/";
id: "__root__" | "/";
fileRoutesById: FileRoutesById;
}
export interface RootRouteChildren {
IndexRoute: typeof IndexRoute;
}
declare module "@tanstack/react-router" {
interface FileRoutesByPath {
"/": {
id: "/";
path: "/";
fullPath: "/";
preLoaderRoute: typeof IndexRouteImport;
parentRoute: typeof rootRouteImport;
};
}
}
const rootRouteChildren: RootRouteChildren = {
IndexRoute: IndexRoute,
};
export const routeTree = rootRouteImport
._addFileChildren(rootRouteChildren)
._addFileTypes<FileRouteTypes>();

34
web/src/routes/__root.tsx Normal file
View File

@@ -0,0 +1,34 @@
import { Outlet, createRootRoute } from "@tanstack/react-router";
import { TanStackRouterDevtoolsPanel } from "@tanstack/react-router-devtools";
import { TanstackDevtools } from "@tanstack/react-devtools";
import { Theme } from "@radix-ui/themes";
import "@radix-ui/themes/styles.css";
import { ThemeProvider } from "next-themes";
export const Route = createRootRoute({
component: () => (
<ThemeProvider
attribute="class"
defaultTheme="system"
enableSystem
disableTransitionOnChange={false}
>
<Theme accentColor="blue" grayColor="gray">
<Outlet />
{import.meta.env.DEV ? (
<TanstackDevtools
config={{
position: "bottom-left",
}}
plugins={[
{
name: "Tanstack Router",
render: <TanStackRouterDevtoolsPanel />,
},
]}
/>
) : null}
</Theme>
</ThemeProvider>
),
});

399
web/src/routes/index.tsx Normal file
View File

@@ -0,0 +1,399 @@
import { createFileRoute } from "@tanstack/react-router";
import { useState, useEffect } from "react";
import { client, type StatusResponse, type Status } from "../lib/api";
import { Card, Flex, Text, Tooltip, Skeleton } from "@radix-ui/themes";
import {
CheckCircle,
XCircle,
Clock,
Bot,
Globe,
Hourglass,
Activity,
MessageCircle,
Circle,
WifiOff,
} from "lucide-react";
import TimeAgo from "react-timeago";
import { ThemeToggle } from "../components/ThemeToggle";
import "../App.css";
const REFRESH_INTERVAL = import.meta.env.DEV ? 3000 : 30000;
const REQUEST_TIMEOUT = 10000; // 10 seconds
const CARD_STYLES = {
padding: "24px",
maxWidth: "400px",
width: "100%",
} as const;
const BORDER_STYLES = {
marginTop: "16px",
paddingTop: "16px",
borderTop: "1px solid var(--gray-7)",
} as const;
const SERVICE_ICONS: Record<string, typeof Bot> = {
bot: Bot,
banner: Globe,
discord: MessageCircle,
};
interface ResponseTiming {
health: number | null;
status: number | null;
}
interface StatusIcon {
icon: typeof CheckCircle;
color: string;
}
interface Service {
name: string;
status: Status;
icon: typeof Bot;
}
type StatusState =
| {
mode: "loading";
}
| {
mode: "response";
timing: ResponseTiming;
lastFetch: Date;
status: StatusResponse;
}
| {
mode: "error";
lastFetch: Date;
}
| {
mode: "timeout";
lastFetch: Date;
};
const formatNumber = (num: number): string => {
return num.toLocaleString();
};
const getStatusIcon = (status: Status | "Unreachable"): StatusIcon => {
const statusMap: Record<Status | "Unreachable", StatusIcon> = {
Active: { icon: CheckCircle, color: "green" },
Connected: { icon: CheckCircle, color: "green" },
Healthy: { icon: CheckCircle, color: "green" },
Disabled: { icon: Circle, color: "gray" },
Error: { icon: XCircle, color: "red" },
Unreachable: { icon: WifiOff, color: "red" },
};
return statusMap[status];
};
const getOverallHealth = (state: StatusState): Status | "Unreachable" => {
if (state.mode === "timeout") return "Unreachable";
if (state.mode === "error") return "Error";
if (state.mode === "response") return state.status.status;
return "Error";
};
const getServices = (state: StatusState): Service[] => {
if (state.mode !== "response") return [];
return Object.entries(state.status.services).map(([serviceId, serviceInfo]) => ({
name: serviceInfo.name,
status: serviceInfo.status,
icon: SERVICE_ICONS[serviceId] || SERVICE_ICONS.default,
}));
};
const StatusDisplay = ({ status }: { status: Status | "Unreachable" }) => {
const { icon: Icon, color } = getStatusIcon(status);
return (
<Flex align="center" gap="2">
<Text
size="2"
style={{
color: status === "Disabled" ? "var(--gray-11)" : undefined,
opacity: status === "Disabled" ? 0.7 : undefined,
}}
>
{status}
</Text>
<Icon color={color} size={16} />
</Flex>
);
};
const ServiceStatus = ({ service }: { service: Service }) => {
return (
<Flex align="center" justify="between">
<Flex align="center" gap="2">
<service.icon size={18} />
<Text style={{ color: "var(--gray-11)" }}>{service.name}</Text>
</Flex>
<StatusDisplay status={service.status} />
</Flex>
);
};
const SkeletonService = () => {
return (
<Flex align="center" justify="between">
<Flex align="center" gap="2">
<Skeleton height="24px" width="18px" />
<Skeleton height="24px" width="60px" />
</Flex>
<Flex align="center" gap="2">
<Skeleton height="20px" width="50px" />
<Skeleton height="20px" width="16px" />
</Flex>
</Flex>
);
};
const TimingRow = ({
icon: Icon,
name,
children,
}: {
icon: React.ComponentType<{ size?: number }>;
name: string;
children: React.ReactNode;
}) => (
<Flex align="center" justify="between">
<Flex align="center" gap="2">
<Icon size={13} />
<Text size="2" color="gray">
{name}
</Text>
</Flex>
{children}
</Flex>
);
function App() {
const [state, setState] = useState<StatusState>({ mode: "loading" });
// State helpers
const isLoading = state.mode === "loading";
const hasError = state.mode === "error";
const hasTimeout = state.mode === "timeout";
const hasResponse = state.mode === "response";
const shouldShowSkeleton = isLoading || hasError;
const shouldShowTiming = hasResponse && state.timing.health !== null;
const shouldShowLastFetch = hasResponse || hasError || hasTimeout;
useEffect(() => {
let timeoutId: NodeJS.Timeout;
const fetchData = async () => {
try {
const startTime = Date.now();
// Create a timeout promise
const timeoutPromise = new Promise<never>((_, reject) => {
setTimeout(() => reject(new Error("Request timeout")), REQUEST_TIMEOUT);
});
// Race between the API call and timeout
const statusData = await Promise.race([client.getStatus(), timeoutPromise]);
const endTime = Date.now();
const responseTime = endTime - startTime;
setState({
mode: "response",
status: statusData,
timing: { health: responseTime, status: responseTime },
lastFetch: new Date(),
});
} catch (err) {
const errorMessage = err instanceof Error ? err.message : "Failed to fetch data";
// Check if it's a timeout error
if (errorMessage === "Request timeout") {
setState({
mode: "timeout",
lastFetch: new Date(),
});
} else {
setState({
mode: "error",
lastFetch: new Date(),
});
}
}
// Schedule the next request after the current one completes
timeoutId = setTimeout(() => void fetchData(), REFRESH_INTERVAL);
};
// Start the first request immediately
void fetchData();
return () => {
if (timeoutId) {
clearTimeout(timeoutId);
}
};
}, []);
const overallHealth = getOverallHealth(state);
const { color: overallColor } = getStatusIcon(overallHealth);
const services = getServices(state);
return (
<div className="App">
<div
style={{
position: "fixed",
top: "20px",
right: "20px",
zIndex: 1000,
}}
>
<ThemeToggle />
</div>
<Flex
direction="column"
align="center"
justify="center"
style={{ minHeight: "100vh", padding: "20px" }}
>
<Card style={CARD_STYLES}>
<Flex direction="column" gap="4">
{/* Overall Status */}
<Flex align="center" justify="between">
<Flex align="center" gap="2">
<Activity
color={isLoading ? undefined : overallColor}
size={18}
className={isLoading ? "animate-pulse" : ""}
style={{
opacity: isLoading ? 0.3 : 1,
transition: "opacity 2s ease-in-out, color 2s ease-in-out",
}}
/>
<Text size="4" style={{ color: "var(--gray-12)" }}>
System Status
</Text>
</Flex>
{isLoading ? (
<Skeleton height="20px" width="80px" />
) : (
<StatusDisplay status={overallHealth} />
)}
</Flex>
{/* Individual Services */}
<Flex direction="column" gap="3" style={{ marginTop: "16px" }}>
{shouldShowSkeleton
? // Show skeleton for 3 services during initial loading only
Array.from({ length: 3 }).map((_, index) => <SkeletonService key={index} />)
: services.map((service) => <ServiceStatus key={service.name} service={service} />)}
</Flex>
<Flex direction="column" gap="2" style={BORDER_STYLES}>
{isLoading ? (
<TimingRow icon={Hourglass} name="Response Time">
<Skeleton height="18px" width="50px" />
</TimingRow>
) : shouldShowTiming ? (
<TimingRow icon={Hourglass} name="Response Time">
<Text size="2" style={{ color: "var(--gray-11)" }}>
{formatNumber(state.timing.health!)}ms
</Text>
</TimingRow>
) : null}
{shouldShowLastFetch ? (
<TimingRow icon={Clock} name="Last Updated">
{isLoading ? (
<Text size="2" style={{ paddingBottom: "2px" }} color="gray">
Loading...
</Text>
) : (
<Tooltip content={`as of ${state.lastFetch.toLocaleTimeString()}`}>
<abbr
style={{
cursor: "pointer",
textDecoration: "underline",
textDecorationStyle: "dotted",
textDecorationColor: "var(--gray-6)",
textUnderlineOffset: "6px",
}}
>
<Text size="2" style={{ color: "var(--gray-11)" }}>
<TimeAgo date={state.lastFetch} />
</Text>
</abbr>
</Tooltip>
)}
</TimingRow>
) : isLoading ? (
<TimingRow icon={Clock} name="Last Updated">
<Text size="2" color="gray">
Loading...
</Text>
</TimingRow>
) : null}
</Flex>
</Flex>
</Card>
<Flex justify="center" style={{ marginTop: "12px" }} gap="2" align="center">
{__APP_VERSION__ && (
<Text
size="1"
style={{
color: "var(--gray-11)",
}}
>
v{__APP_VERSION__}
</Text>
)}
{__APP_VERSION__ && (
<div
style={{
width: "1px",
height: "12px",
backgroundColor: "var(--gray-10)",
opacity: 0.3,
}}
/>
)}
<Text
size="1"
style={{
color: "var(--gray-11)",
textDecoration: "none",
}}
>
<a
href={
hasResponse && state.status.commit
? `https://github.com/Xevion/banner/commit/${state.status.commit}`
: "https://github.com/Xevion/banner"
}
target="_blank"
rel="noopener noreferrer"
style={{
color: "inherit",
textDecoration: "none",
}}
>
GitHub
</a>
</Text>
</Flex>
</Flex>
</div>
);
}
export const Route = createFileRoute("/")({
component: App,
});

13
web/src/styles.css Normal file
View File

@@ -0,0 +1,13 @@
@import "@radix-ui/themes/styles.css";
body {
margin: 0;
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Roboto", "Oxygen", "Ubuntu",
"Cantarell", "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
}
code {
font-family: source-code-pro, Menlo, Monaco, Consolas, "Courier New", monospace;
}

3
web/src/vite-env.d.ts vendored Normal file
View File

@@ -0,0 +1,3 @@
/// <reference types="vite/client" />
declare const __APP_VERSION__: string;

29
web/tsconfig.json Normal file
View File

@@ -0,0 +1,29 @@
{
"include": ["**/*.ts", "**/*.tsx"],
"compilerOptions": {
"target": "ES2022",
"jsx": "react-jsx",
"module": "ESNext",
"lib": ["ES2022", "DOM", "DOM.Iterable"],
"types": ["vite/client"],
/* Bundler mode */
"moduleResolution": "bundler",
"allowImportingTsExtensions": true,
"verbatimModuleSyntax": true,
"isolatedModules": true,
"noEmit": true,
/* Linting */
"skipLibCheck": true,
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true,
"noUncheckedSideEffectImports": true,
"baseUrl": ".",
"paths": {
"@/*": ["./src/*"]
}
}
}

67
web/vite.config.ts Normal file
View File

@@ -0,0 +1,67 @@
import { defineConfig } from "vite";
import viteReact from "@vitejs/plugin-react";
import tanstackRouter from "@tanstack/router-plugin/vite";
import { resolve } from "node:path";
import { readFileSync, existsSync } from "node:fs";
// Extract version from Cargo.toml
function getVersion() {
const filename = "Cargo.toml";
const paths = [resolve(__dirname, filename), resolve(__dirname, "..", filename)];
for (const path of paths) {
try {
// Check if file exists before reading
if (!existsSync(path)) {
console.log("Skipping ", path, " because it does not exist");
continue;
}
const cargoTomlContent = readFileSync(path, "utf8");
const versionMatch = cargoTomlContent.match(/^version\s*=\s*"([^"]+)"/m);
if (versionMatch) {
console.log("Found version in ", path, ": ", versionMatch[1]);
return versionMatch[1];
}
} catch (error) {
console.warn("Failed to read Cargo.toml at path: ", path, error);
// Continue to next path
}
}
console.warn("Could not read version from Cargo.toml in any location");
return "unknown";
}
const version = getVersion();
// https://vitejs.dev/config/
export default defineConfig({
plugins: [tanstackRouter({ autoCodeSplitting: true }), viteReact()],
test: {
globals: true,
environment: "jsdom",
},
resolve: {
alias: {
"@": resolve(__dirname, "./src"),
},
},
server: {
port: 3000,
proxy: {
"/api": {
target: "http://localhost:8080",
changeOrigin: true,
secure: false,
},
},
},
build: {
outDir: "dist",
sourcemap: true,
},
define: {
__APP_VERSION__: JSON.stringify(version),
},
});