mirror of
https://github.com/Xevion/banner.git
synced 2025-12-09 16:06:32 -06:00
Rate limiting/costs/bursting section in README
This commit is contained in:
13
README.md
13
README.md
@@ -87,4 +87,15 @@ For now, all majors will be scraped fully every 4 hours with at least 5 minutes
|
||||
- Other majors will be scraped in arbitrary order (if required).
|
||||
- Scrape timing will be stored in Redis.
|
||||
- CRNs will be the Primary Key within SQLite
|
||||
- If CRNs are duplicated between terms, then the primary key will be (CRN, Term)
|
||||
- If CRNs are duplicated between terms, then the primary key will be (CRN, Term)
|
||||
|
||||
## Rate Limiting, Costs & Bursting
|
||||
|
||||
Ideally, this application would implement dynamic rate limiting to ensure overload on the server does not occur.
|
||||
Better, it would also ensure that priority requests (commands) are dispatched faster than background processes (scraping), while making sure different requests are weighted differently.
|
||||
For example, a recent scrape of 350 classes should be weighted 5x more than a search for 8 classes by a user.
|
||||
Still, even if the cap does not normally allow for this request to be processed immediately, the small user search should procede with a small bursting cap.
|
||||
|
||||
The requirements to this hypothetical system would be:
|
||||
- Conditional Bursting: background processes or other requests deemed "low priority" are not allowed to use bursting.
|
||||
- Arbitrary Costs: rate limiting is considered in the form of the request size/speed more or less, such that small simple requests can be made more frequently, unlike large requests.
|
||||
Reference in New Issue
Block a user