xevion/undefined.behavio.rs
1
0
Fork 0
mirror of https://github.com/Xevion/undefined.behavio.rs.git synced 2025-12-07 16:09:03 -06:00
Code Issues Packages Projects Releases Wiki Activity

Transfer all posts

Browse Source
This commit is contained in:
Xevion
2023-11-24 17:36:08 -06:00
parent 790d6c7156
commit 872e7cc5f1
7 changed files with 728 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
src/pages/posts/jekyll-github-pages-and-azabani.md Normal file
View File

@@ -0,0 +1,42 @@
---
title: "Jekyll, GitHub Pages, and Azabani"
pubDate: 2020-12-04 13:16:22 -0600
layout: "@layouts/Post.astro"
tags: ["jekyll", "github-pages"]
description: "This is my first time trying to create a proper blog, and my second time using GitHub pages. How did it turn out?"
---
This is my first time trying to create a proper blog, and my second time using GitHub pages, [the first][github-pages-resume] less than 2 weeks earlier.
As a cheap high school student with zero income and expensive subscriptions, it's obvious that I wouldn't try to brunt
the cost of any sort of subscription for a site I <span style="white-space: nowrap;">barely use - well</span>, not anymore, at least.
Jekyll is pretty cool - I'm really having fun learning new web tech (like Vue, BrowserSync, Sass). The syntax of the header is something new, but the templating language's filters are something I'm already familiar with,
using Jinja with Flask and Django already (great, a third dialect of both similar and completely different filters to get used to!).
GitHub pages is interesting too. The main feature is not in its performance, features or ease of use
(the documentation is sort of hard to read), but in the fact that's free. Free static site hosting with Jekyll,
custom domains and HTTPS supported. That's perfect for me, and I wish I had tried using it before now.
I'm planning to use Jekyll for my future projects now, creating a site for my school's FTC Robotics team, and hopefully, for all of my projects deserving it.
---
Now, I know this may come as a surprise, but: I'm not very good at making good websites. This entire blog as of December 4th, 2020, is of
[Delan Azabani's][delan-azabani] creation, with just a few edits. I hope to change that by designing my own site or changing
enough to say that I've made it my own, but I haven't yet had the chance to do that.
Delan, if you're reading this before I get to improve it - I really admire your work. Your blog, your interests, your skills,
everything I've read on around you is super cool. Even you website has it's own unique LaTeX-like style.
I have no idea if you had inspiration or came up with everything from the font to the [fleurons][fleurons] to the super cool date icons on the index,
but this is one of the most memorable and true developer sites I've come across.
I've cleaned up most of the pages and made the repositories structure a little easier to look through, and I'm going to continue doing that,
but overall, this is Azabani's work. I hope me copying your site comes across as admiration more than thievery.
Additionally, the [ISC license][repo-license] has been obeyed, and [you are linked][about-attribution] as the clear creator of the site.
[github-pages-resume]: https://xevion.github.io/
[delan-azabani]: https://www.azabani.com/
[fleurons]: https://en.wikipedia.org/wiki/Fleuron_(typography)
[repo-license]: https://github.com/Xevion/v2.xevion.dev/blob/fc1761725eb5d4f21bbf778ee504c1e89f9ee9dd/LICENSE#L1
[about-attribution]: https://github.com/Xevion/v2.xevion.dev/blame/fc1761725eb5d4f21bbf778ee504c1e89f9ee9dd/about.md#L13

178
src/pages/posts/painting-images-with-ipv6.md Normal file
View File

@@ -0,0 +1,178 @@
---
layout: "@layouts/Post.astro"
title: "Painting Images with IPv6"
date: 2023-04-14 13:07:43 -0500
tags: ["ipv6", "python", "asyncio", "websocket", "PIL"]
description: "Have you ever painted images with IPv6? I found out how in 30 minutes."
---
Despite how it sounds, this is not a joke. Well, maybe it is, but it's a fantastic demonstration of IPv6 addressing,
and you can join in on the fun right now too!
Some months ago, I found an interesting little site: [v6.sys42.net][place-v6]*.
> **Warning**: This site contains image curated by an unrestricted community: It is entirely out of my control what you
> will see on it, but _usually_, it's safe for work.
To give a short explanation of this site, it's similar to [/r/place][place-reddit], an online canvas where users can
place a single
pixel on a canvas every 5 minutes.
The difference between [/r/place][place-reddit] and [Place: IPv6][place-v6] is that the latter uses a human-accessible
user interface for
selecting colors and placing pixels, while the other uses IPv6 addresses.
Hold on - you might be thinking, _"Don't you mean a REST API? Or at least an HTTP webserver? Or even a TCP/UDP
socket?"_.
None of that, it's not just a normal server accessible by IPv6 exclusively - it's an IPv6 address range booked
specifically for the purpose of receiving R, G, B, X and Y arguments.
## How does it work?
To use the service, you send an ICMP packet (a `ping`) to a specific IPv6 address dictated by your arguments.
The arguments are encoded into the IPv6 address, and the service will receive and parse whatever you put into it.
> **Note**: The service has since been updated to use a different IPv6 address range, but the concept is the same.
Originally, the IPv6 address was {% ihighlight digdag %}2a06:a003:d040:SXXX:YYY:RR:GG:BB{% endihighlight %} where `XXX`
and `YYY` are the X and Y coordinates, and `RR`, `GG` and `BB` are the R, G and B values of the pixel. By substituting
you arguments into these positions, you can paint a pixel on the canvas. Lastly, the `S` represents the size of the
pixel.
Only `1` or `2` is accepted, representing a 1x1 or 2x2 pixel.
On top of this, the values are encoded in hexadecimal, so you can use the full range of 0-255 for each color without
worry.
As an example, painting the color {% ihighlight dart %}#008080{% endihighlight %} (teal) at the position `45, 445` would
be encoded as
{% ihighlight digdag %}2a06:a003:a040:102d:01bd:00:80:80{% endihighlight %}. To help you pick out the X and Y
coordinates, {% ihighlight java %}45{% endihighlight %} is {% ihighlight java %}0x2D{% endihighlight %} in hexadecimal,
and {% ihighlight java %}445{% endihighlight %}
is {% ihighlight java %}0x1BD{% endihighlight %}. The color is simply placing in the last 6 bytes of the address, no
hexadecimal conversion needed.
By now, the basic concept should be clear. You can paint a pixel by sending a `ping` to a specific IPv6 address with the
arguments encoded into it.
In Python, the encoding can be done like so:
```python
def get_ip(x: int, y: int, rgb: Tuple[int, int, int], large: bool = False):
return f"2a06:a003:d040:{'2' if large else '1'}{x:03X}:{y:03X}:{rgb[0]:02X}:{rgb[1]:02X}:{rgb[2]:02X}"
```
> I've been interested in ways to format this dynamically with arbitrary base IPv6 addresses, but I haven't found
> a performant way of calculating it. It seems like it would require lots of bit shifting, precalculated constants,
> and a C extension to maximize performance. A future endeavour, perhaps.
> Additionally, the base IP does not change often (I have only observed it changing once).
## The Rest of Place: IPv6
IPv6 place includes a proper canvas to boot, and you can view it at [v6.sys42.net][place-v6], along with the basic
instructions on how to use it.
The way you view (and technically, receive information), is through a WebSocket connection. In the browser, you'll
receive updates that are combined into a Javascript-based canvas element.
On the initial page load, you'll receive a one-time message containing a PNG image of the current canvas. After that,
you'll receive partial updates whenever updates occur. These updates are also PNG images, but they are transparent
except for the pixels that have changed.
The WebSocket also contains information about PPS, or Pixels Per Second. This is the rate at which the canvas is
updated globally by all users. PPS messages are drawn onto a historical graph displayed below the canvas.
## 30 Minute Implementation
I had a lab due the day I decided to implement this, so I only had 30 minutes to spare. I decided to use Python, as it
offers a lot of flexibility and is my go-to language for anything quick.
I realized quite early on that any normal ping operations would be far too slow, and looked into batch pinging
implementations.
I found the [`multiping`][pypi-multiping] package first, and stuck with it. As the name implies, it specializes in
sending multiple pings at once, and is quite fast for Python. It also supports IPv6 without complaint.
The great part of this package is that it allows you to set timeouts - this is key in performance, as we don't care
about the response, only that the packet was sent. This allows us to send a large number of pings at once, and not
have to wait for a response.
Here was my first implementation:
```python
def upload_pixels(pixels: List[Pixel], chunk_size: int = None):
"""
Given a list of pixels, upload them with the given chunk size.
"""
ips = [get_ip(x, y, rgb) for x, y, rgb in pixels]
return upload(ips, chunk_size)
def upload(ips: List[str], chunk_size: int = None):
# Default to maximum chunk size
if chunk_size is None: chunk_size = maximum_chunk
chunked = list(chunkify(ips, min(maximum_chunk, chunk_size)))
for i, chunk in enumerate(chunked, start=1):
print(f'Chunk {i}/{len(chunked)}')
multi_ping(chunk, timeout=0.2, retry=0)
```
`multiping` only supports sending a maximum of 10,000 pings at once, so I chunked the list of IPs into groups of 10,000
before letting `multiping` handle the rest.
Receiving data from the Websocket required a bit more guesswork, but the ultimate implementation is quite
straightforward:
```python
async def get_image(websocket):
while True:
data = await websocket.recv()
if type(data) == bytes:
return Image.open(io.BytesIO(data))
```
I decided to use [Pillow][pypi-pillow] for image manipulation, as it's a well-known and well-supported library. I
went with [`websockets`][pypi-websockets] for the WebSocket implementation.
Since the Websocket is used for both image updates and PPS data, a type check is required. PPS data is completely
ignored.
## The Final Implementation
While my first implementation was just fine for 30 minutes, and I could have left off there, I wanted to see if I could
do better. Initial usage demonstrated that any given pixel could fail to be placed - at high PPS, it seems some pixels
don't reach the server. Either the packets are dropped, they aren't ever sent, or the server simply doesn't process
them.
Whatever the case, a 'repainting' mechanism had to be implemented, and the Canvas had to be kept in memory.
I manage this with an async-based `PlaceClient` class that provides methods for receiving WebSocket updates, storing
the canvas image, identifying differences between the canvas and the local image, and sending pixels synchronously.
Here's what my final API looks like:
```python
client = await PlaceClient.connect(os.getenv(Environment.WEBSOCKET_ADDRESS))
# Set the current target to the image we want on the canvas
client.current_target = original_image
# Launch a background task to receive updates from the WebSocket and update our local 'canvas'
asyncio.create_task(client.receive())
# Paint the image with up to 5% error (difference tolerance)
await client.complete(0.05)
```
You can check out the repository at [github.com/Xevion/v6-place][github-v6-place].
[place-v6]: https://v6.sys42.net/
[place-reddit]: https://www.reddit.com/r/place/
[pypi-multiping]: https://pypi.org/project/multiping/
[pypi-pillow]: https://pypi.org/project/Pillow/
[pypi-websockets]: https://pypi.org/project/websockets/
[github-v6-place]: https://github.com/Xevion/v6-place

81
src/pages/posts/project-facelift-new-and-old.md Normal file
View File

@@ -0,0 +1,81 @@
---
title: "Project Facelift, New and Old"
layout: "@layouts/Post.astro"
date: 2021-02-25 16:41:41 -0600
tags: ["projects", "photography", "update"]
preview_image: "https://raw.githubusercontent.com/Xevion/Paths/master/.media/banner.png"
description: "Through December, I decided to make heavy visual changes to many of my biggest projects, creating custom banners, of which I am quite proud..."
---
Through December, I decided to make heavy visual changes to many of my biggest projects, creating custom banners, of which I am quite proud...
[![/Paths/ Banner][paths-banner]][paths]
The most advanced of them, the cells in the background are designed to mimic the real Paths project as closely as possible.
The angled gradient stroke on the text helps break up the text on the background with all
If you want to see [the SVG form][paths-svg], Adobe Illustrator exports it perfectly!
It's a shame though, [not all browsers support it][paths-svg-glitch], as I figured out on my laptop shortly after using a SVG.
[![/phototag/ Banner][phototag-banner]][phototag]
Besides being my first major project, this banner means something personal to me as every one of those photos in the background are *mine*.
I took them with my own camera. You can find every single one on my [photography portfolio][photography-portfolio].
[![/Boids/ Banner][boids-banner]][boids]
The first banner I designed, I'm still not exactly happy with how it came out, but regardless, it works well enough I guess.
[![/power-math/ Banner][power-math-banner]][power-math]
A very simple banner indeed.
[![/contest/ Banner][contest-banner]][contest]
Yeah, I'm not super proud of this one either.
To be clear, I am not normally a graphic designer - HTML & CSS is something I struggle with for hours and hours trying to get right
(I hope my efforts have paid off in all that you see), so making designs like this was both a challenge and a learning experience like no other.
I'm so happy I did it, and I hope those who see it see the work I put in to make my projects look good. Fingers crossed I return to Illustrator for every major project.
---
I created [**power-math**][power-math] in order to create a method for me to practice simple and fast arithmetic problems among others.
I designed it with LaTeX rendering in mind, to give it that feel of 'reality' other fonts can't give.
Once again, another dark theme app since I'm lazy and don't know how to do light themes.
Using Vue sped this process up *a lot*, and very quickly I had a working application with awesome visuals and animations.
Additionally, since the project is completely client-side, so you can [practice with it yourself][power-math-pages]!
<video autoplay muted>
<source src="/assets/img/power-math-demonstration.mp4" type="video/mp4">
</video>
I'm probably embarrassing myself quite a bit here with how slow I am - I wasn't kidding when I said I made this project so I could get better!
![power-math settings][power-math-settings]
The project is setup to support many different kinds of problems, each with their own difficulty settings, as well as the ability to completely disable them.
On top of that, by hovering over each level of difficulty, you can preview what a problem might look like, latex supported!
---
This post was not posted on time correctly, so if anything seems out of place - that's why. I attempted to refurbish it
so things read cleaner and aren't out of place, but I'm not perfect working at 1:40AM in the morning.
[paths]: https://github.com/Xevion/Paths
[paths-banner]: https://raw.githubusercontent.com/Xevion/Paths/master/.media/banner.png
[paths-svg]: https://raw.githubusercontent.com/Xevion/Paths/master/.media/banner.svg
[paths-svg-glitch]: https://i.imgur.com/ynZ5vqy.png
[phototag-banner]: https://raw.githubusercontent.com/Xevion/phototag/master/.media/banner.png
[phototag]: https://github.com/Xevion/phototag
[boids]: https://github.com/Xevion/Boids
[boids-banner]: https://raw.githubusercontent.com/Xevion/Boids/master/.media/banner.png
[power-math]: https://github.com/Xevion/power-math/
[power-math-banner]: https://raw.githubusercontent.com/Xevion/power-math/master/.media/banner.png
[contest]: https://github.com/Xevion/contest
[contest-banner]: https://raw.githubusercontent.com/Xevion/contest/master/.media/banner.png
[photography-portfolio]: https://www.rcw.photos/
[power-math-pages]: https://xevion.github.io/power-math/
[power-math-video]: /assets/img/power-math-demonstration.mp4
[power-math-settings]: /assets/img/power-math-settings.png

1
src/pages/posts/race-conditions-in-signal-handlers.md
View File

@@ -3,6 +3,7 @@ title: "Race Conditions in Signal Handlers"
pubDate: 2023-07-26 16:08:12 -0500
description: "Signals offer a unique, low-level way of communicating with processes. But under certain circumstances, they can kill processes, even when they shouldn't."
layout: "@layouts/Post.astro"
tags: ["tar", "signals", "interrupt", "handler", "process", "unix", "race-condition"]
---
Signals offer a unique, low-level way of communicating with processes. But under certain circumstances, they can kill

195
src/pages/posts/restricted-memory-and-data-framing-tricks.md Normal file
View File

@@ -0,0 +1,195 @@
---
title: "Restricted Memory & Data Framing Tricks"
layout: "@layouts/Post.astro"
date: 2022-07-16 13:51:00 -0500
tags: ["c", "memory", "embedded", "ti", "msp430", "union"]
description: "Tips and tricks I learned about handling restricted memory while working on microcontrollers at my first internship"
---
Working on microcontrollers is a far cry from the essentially unlimited memory and floating point operations available
in _Python_ and other high level languages. Here's what I have been learning at my _first_ internship...
## Restricted Memory
While working there, I was assigned to a project that required usage of a Texas Instruments MSP430 microcontroller.
While there were many quirks with working on MCUs like this (like having to ditch JetBrains & VSCode altogether!), the
biggest quirk isn't working with C: it's working without `malloc` altogether.
On low memory devices like this, memory is extremely limited and there is no certainty that your code will not leak memory.
Usage of `malloc` and other dynamic memory allocation methods are considered innately dangerous - while there is a chance
you will write perfect code that will properly allocate/deallocate, you can't be certain that your complex program
won't run out of memory on such a small footprint to work with.
Instead, variables are assigned either inside methods for short periods (and passed around), or they are assigned statically and globally.
It appears that the libraries I use personally prefer globally accessible variables, which 99% of the time, is very wrong - but in
Microcontroller land, global variables are your friend.
```c
#include <stdint.h>
uint8_t uid[4]; // A unique identifier
int main(void) {
UID_generate(uid, 4); // Pass the pointer, the function writes to it (and does not return it)
UART_putUid(uid, 4); // Write the UID to UART
}
```
```c
void UID_generate(uint8_t uid, int length) {
uint8_t i = 0;
while(i < length)
uid[i++] = RANDOM_char();
}
void UART_putUid(uint8_t* uid, int length) {
uint8_t i = 0;
while(i < length)
UART_putChar(uid[i++]);
}
void UART_putChar(uint8_t value) {
while(!(UCB0IFG & 0x1));
UCB0TXBUF = value;
}
```
<center>
<i>
<small>
UART is a serial communication technology we use to send characters & text to the COM terminal.
For more information, click <a href="https://www.youtube.com/watch?v=VBRUyLcqXV4">here</a>.
</small>
</i>
</center>
There's not much more to this - don't use `malloc`, stick to the _stack_ for actively executing methods and use _global variables_
when you need to go into low power mode while maintaining state.
Overall, this doesn't hinder ones ability to write working code - the features are still there, but the way you access
methods, store data & manipulate is re-organized - sometimes at the detriment to quality & refactoring efforts.
## Data Framing Tricks
While at my internship, I used my MSP430 microcontroller to communicate with various devices over UART and SPI. I also sent
commands to a ISO15693 NFC wafers. All of these interfaces are extremely low level and the best documentation I have
is often just a PDF and some random code scattered across the internet. There is no library to speak of, usually.
Communicating at a low level like this requires reading and writing individual bytes of data into _frames_, or arrays
of bytes with a well-defined structure.
<center>
<b>ISO15693 Write Single Block (Addressed)</b> <a href="http://www.ti.com/lit/an/sloa141/sloa141.pdf" style="color: #90bcff">source</a>
</center>
[![ISO15693 Write Single Block Diagram][iso15693-diagram]][iso15693-diagram-edn]
Traditionally, commands are built statically all at once in a mostly hardcoded manner:
```c
uint8_t offset = 0;
ui8TRFBuffer[offset++] = 0x61;
ui8TRFBuffer[offset++] = 0x21;
ui8TRFBuffer[offset++] = 0xA7;
ui8TRFBuffer[offset++] = 0x3E;
ui8TRFBuffer[offset++] = 0xFF;
ui8TRFBuffer[offset++] = 0x58;
// ... You get the idea
```
Instead, what if we could format this into a `struct` that we could pass around on the stack with a pointer?
```c
struct AddressedSingleBlockWrite {
uint8_t Flag;
uint8_t Command;
uint8_t Address[8];
uint8_t Block;
uint8_t Data[4];
};
int main() {
struct AddressedSingleBlockWrite command;
command.Flag = 0x20 | 0x40;
command.Command = 0x21;
uint8_t address[8] = {0xA7, 0x3E, 0xFF, 0x58, 0x21, 0x32, 0x10, 0xFE};
memcpy(&command.Address, &address, sizeof(command.Address));
command.Block = 0x05;
uint8_t data[4] = {0x11, 0x22, 0x33, 0x44};
memcpy(&command.Data, &data, sizeof(command.Data));
}
```
Now we have a defined structure in our source code and we can move and manipulate various parts of our command
structures without having to deal with hardcoded offsets. Still though, if we want to copy this command structure into
the buffer, we have to individually copy each part of the command - which will break the second we modify its structure.
There's a fantastic solution for it: **Unions**.
```c
union ASBWUnion {
uint8_t data[15];
struct AddressedSingleBlockWrite marshalled;
};
```
```c
union ASBWUnion demarshalled;
demarshalled.marshalled = command;
for (int i = 0; i < 15; i++)
printf("%x ", demarshalled.data[i]);
```
```sass
60 21 a7 3e ff 58 21 32 10 fe 5 11 22 33 44
```
`union`s are special datatypes that share a single memory footprint (equal to it's largest member) starting at the exact same point memory.
They combine neatly with `struct`s to allow us to represent the `AddressedSingleBlockWrite` as a single byte array.
<center>
<b>Note</b>: When implementing this, I do recommend that you create macro definitions for the length of the final command structure.
This will help greatly when it comes to refactoring or making adjustments to your command structure.
</center>
#### Reversing Endianness
If you check out TI's [sloa141][sloa141] PDF on ISO 15693 commands, you'll notice that many of the examples have
sections of their bytes reversed - sections like the _Address_ and _Data_ sections, but not the entire compiled command.
One such example, `60``21``9080C2E5D2C407E0``06``55443322`, has the _Flag_, _Command_,
_Address_, _Block_ and _Data_ stored in that order. But for this particular example, how could the address be `E007C4D2E5C28090`?
How could the data be `0x22334455`? This odd ordering has a name - Endianness - the order of bytes as the underlying architecture understands it.
While for my particular usage, reversing endianness was not needed, it's an interesting problem that can be solved
quite easily with our new data structure.
```c
void ReverseEndianness(struct AddressedSingleBlockWrite asbw) {
uint8_t temp;
int i = 0;
for (; i < 4; i++) {
temp = command.Address[i];
command.Address[i] = command.Address[8 - i - 1];
command.Address[8 - i - 1] = temp;
}
for (i = 0; i < 2; i++) {
uint8_t temp = command.Data[4 - i - 1];
command.Data[4 - i - 1] = command.Data[i];
command.Data[i] = temp;
}
}
```
### Conclusion
Working in a restricted memory space is not that hard once you get used to the lack of normal functions such as `malloc`
and `printf`, but balancing performance, power consumption, memory allocation and code quality gets harder the more
complex your program gets.
[iso15693-diagram]: /assets/img/iso15693_diagram.png
[iso15693-diagram-edn]: /assets/iso15693_diagram.edn
[sloa141]: http://www.ti.com/lit/an/sloa141/sloa141.pdf

231
src/pages/posts/runnerspace-built-in-under-30-hours.md Normal file
View File

@@ -0,0 +1,231 @@
---
layout: "@layouts/Post.astro"
title: "Runnerspace, Built in Under 30 Hours"
date: 2022-03-29 13:56:22 -0600
tags: ["flask", "hackathon", "utsa", "rowdyhacks", "projects"]
preview_image: https://raw.githubusercontent.com/Xevion/runnerspace/master/static/embed-banner.png
description: "I attended Rowdy Hacks 2022 at UTSA, and while I can't say I left smiling, I did create a pretty cool project that I'd like to talk about."
---
I attended Rowdy Hacks 2022 at UTSA, and while I can't say I left smiling, I did create a pretty cool project that I'd like to talk about.
## Rowdy Hacks 2022
At Rowdy Hacks 2022, there were a couple tracks and challenges available to those attending; I'll skip telling you what
was available and tell you what we took - 'General' (the experienced developers) track and the 'Retro' challenge.
For our project, we initially started about making a tile-memorization game that would have a fast-paced MMO aspect
where players would compete to push each other out of the round by 'memorizing' sequences of information (like colored tiles).
Players would have health and successfully completing a sequence would lower other's health while raising your own,
almost like stealing health, but inefficiently (lossy vampire?).
We planned to do all of this with Web Sockets ran on FastAPI, although we pivoted to FastAPI for the backend when we
couldn't get web sockets to communicate properly. After 4 hours of fooling around and slow progress, we realized that
neither of us were game developers and that this project would not suffice.
## Runnerspace
[![/runnerspace/ Banner][runnerspace-banner]][runnerspace-github]
Our new plan was to create a retro social media site based on MySpace - it would use Flask for the frontend,
Sass (to make frontend development a little less painful), and SQLAlchemy as a database ORM. Furthermore, with Flask,
Jinja2 is the standardized templating engine.
In the spirit of the Retro challenge, we made sure to style it like MySpace and other social media sites of the era.
We started at a slow pace, but eventually, we got on decently and were quickly beginning to make progress on the
frontend - which happened to be the major bottleneck. Given that our development time was limited and both of us were not
database modeling experts, we had to design out database around whatever the frontend ended up looking like.
Eventually though, after dozens of hours, we were able to come to a stopping point where the site was presentable to
judges. We submitted, got judged, and went through the rest of the Hackathon without too much sleep.
By the time I got home, I ended up going without sleep for 35 hours straight (from 8AM Saturday until 7PM Sunday) through
the whole hackathon. Never have I ever been so tired in my life, but my peak exhaustion was not right before I went to
sleep - in fact, it was in the hours preceding our project submission.
After waking up the next day, I began working on finishing the project in a way more suitable for a resume submission
and portfolio project; the project in its current state was not deployable, it had a couple major issues and simply put -
it needed a lot more work before it was truly ready for presentation.
<center>The rest of this post will document Runnerspace's development.</center>
### Form Validation
Introducing *real* form validation into the project exposed a LOT of major issues with login, signup and other forms
that I had never thought of before; it's hard to say if some changes that it wanted were just paradigms I decided to
follow or not, but there were definite issues.
For example, passwords and usernames had absolutely no requirements on them!
A empty password was valid, spaces, unicode (emojis) and more were completely acceptable. when you're designing a webapp,
you never think about the millions of things users could do to mess with it, whether stupid or malicious.
In the process of form validation, I also ended up combining separate POST and GET routes into one - allowing GET and
POST methods at the same route. I had never thought of taking them at the same route for some reason, as if Flask
wouldn't be able to handle it or something. I didn't do this for ALL my routes - I later ended up creating a `/post/{id}/like` route
to allow users to *like* a certain post; this request was sent through AJAX with jQuery; only POST requests were allowed
on this route.
Combining POST and GET routes into the same `@route` function had more than a refactoring purpose; it allowed rejected forms
to be returned to the user properly! Displaying errors could now target specific fields!
At the same time, rendering these fields properly became rather complex; a macro function needed to be built in order to
make sure fields were consistent across the site.
{% raw %}
```jinja
{% macro render_field(field, show_label=True) %}
{% if show_label %}
{{ field.label }}
{% endif %}
{{ field(placeholder=field.description, **kwargs)|safe }}
{% if field.errors %}
<ul class=errors>
{% for error in field.errors %}
<li>{{ error }}</li>
{% endfor %}
</ul>
{% endif %}
<br>
{% endmacro %}
```
{% endraw %}
This code is a modified version from the WTF-Flask docs.
### Heroku Deployment
Heroku is a fantastic tool for cheap students or developers like me who don't wanna pay, but do want the project running.
Employers are not going to care that much about a branded domain, and they probably won't care about it taking 7-12 seconds to boot up,
but the fact that you could deploy it all and keep it running long term? That speaks volumes. Or at least, I hope it does. :P
And speaking of volumes, integrating a project with Heroku generates volumes of issues that one would never find in
development locally.
#### Flask, Postgres and Heroku
One of the first things you'll find with Heroku is that the drives your application runs on are
<abbr title="Short lived, temporary">ephemeral</abbr> and anything written in run-time will not exist afterwards.
Some applications wouldn't mind this, but ours would, given that during development we ran our database ORM, SQLAlchemy,
off a local `.sqlite` file - a local file. This works great for our local developmnet purposes, but given that our
database should persist between dynos (effectively, different workers that work the Flask application) and restarts, it
would result in chaos and confusion if we actually ran with that.
So, my solution was to simply boot up a Heroku Postgres database, which luckily is completely *free* and can be setup
within seconds; additionally, SQLAlchemy supports many, many database types, including Postgres.
Additionally, Heroku provides the private URL for connecting to this database through environment variables, so
accessing it and integrating it's usage is quite easy; check the application's environment type variable to see if it's
running in `development` or `production`, and set the database URL accordingly.
Normally, that's it, but when I ran it for the first time, it errored: Heroku actually provides a different URL than
SQLAlchemy wants; SQLAlchemy wants `postgresql` for the protocol, but Heroku provides a URL with `postgres` instead. While
it once did support the latter, it no longer does. The solution? Simply using `.replace()` on the string to swap out the two.
Do make sure though to include the FULL protocol as the chance that some other part of the URL could contain `postgres` or
`postgresql` are slim, but not zero, and could result in a very, very confusing error if not. I also took advantage of the
`limit` parameter for `.replace()` as *extra* insurance.
```python
# Heroku deployment
if app.config['ENV'] == 'production':
app.config['SECRET_KEY'] = os.getenv('SECRET_KEY')
app.config['SQLALCHEMY_DATABASE_URI'] = os.getenv('DATABASE_URL', '').replace('postgres://', 'postgresql://', 1)
```
Side-note: Do make sure to provide some kind of null-check or fallback for environment variable retrieval; it is possible the environment
variable could be un-configured and the `.replace` will fail on non-string values like `None`.
Lastly, with the protocol discrepancy solved, one would believe everything to be sorted out? Not quite; there is one last piece to go.
To connect to PostgreSQL databases, one actually needs to install a Pypi module; `psycopg2`, the most weirdly named module
I've ever seen that needed to be explicitly named and installed for a functioning application.
It seems the logic in including this was that if they included adapters for every type of database SQLAlchemy could work with, the dependency graph
would be huge and installing SQLAlchemy would add tons of modules and code that would *never* end up being ran, and worse,
could become failure points for applications which might only want to work with SQLite databases, like mine in the beginning.
#### spaCy Models
In the pursuit of making sure my application was safe to keep up indefinitely on Heroku and accessible to anyone, I wanted
to make it at least a little hard for people to post profanity on it; I wanted to spend as little time on this part of
the app as possible while getting the most out of it. It turned out to actually be quite a searching frenzy to find a
fully functional module on Pypi that would do this for me, but I eventually found something suitable.
Adding this to my application initially was quite easy; `pipenv install profanity-filter`
and then `pipenv exec python -m spacy download en`, and the module was ready to go. The module in total was surprisingly easy to use;
instantiate an object, then run text into a method and respond to the web request accordingly.
Later, I would end up building a WTForms validator to encapsulate profanity checks into a single function, which is another fantastic
reason for using WTForms (I am very happy with how WTForms works in Runnerspace). See below:
```python
class NoProfanity(object):
def __init__(self, message: Optional[str] = None):
if not message:
message = 'Profanity is not acceptable on Runnerspace'
self.message = message
def __call__(self, form, field):
if profanity.contains_profanity(field.data):
raise ValidationError(self.message)
```
Based on how this works with `__call__`, you'd note that a function (or *lambda*) is a perfectly fine substitute,
but using an object lets one provide a custom message with your validators. I really like the pattern the documentation
showed me and would love to use it sometime again soon; perhaps other languages and projects could use something similar.
So, what went wrong? Disk persistence. In the process of getting Heroku to boot, it would completely fail to load the
`profanity_filter` module! The `spaCy` models it needed did not exist. So, I found out that `spacy` provided pragmatic
methods to download models on the fly...
```python
if app.config['ENV'] == 'production':
# Ensure spaCy model is downloaded
spacy_model = 'en_core_web_sm'
try:
spacy.load(spacy_model)
except: # If not present, we download it, then load.
spacy.cli.download(spacy_model)
spacy.load(spacy_model)
```
But again - it didn't work. I submitted a discussion post asking for help, and they told me you need to restart the process
in order to use newly downloaded models.
So, I had to pivot to a different way to download the model. You may be wondering: why can't I download the model
through requirements.txt or something? Well, the thing is, I use `pipenv` *and* the model it needs is **not** hosted on
PyPi!
So, I started looking into it - how do I get `pipenv`, which works with Heroku right out of the box, to download the
spaCy model for me? It turns out, it's quite straight-forward. `pip` can install packages directly from third-party URLs
directly using the same `install` command you'd use with `PyPi` packages.
Furthermore, `pipenv` does indeed support URLs [since late 2018][pipenv-direct-url-github-issue] and adding it
is simple as `pipenv install {url}`. So, that should work, right? It'll download the module, and it's even the smallest model version,
so there's not going to be any issues with [maxing out the tiny RAM][heroku-and-spacy-model-so] the hobby dyno is provided with, right?
Haha, no, it still doesn't work, and I never found out why. Additionally, the `profanity-filter` project is
[abandoned and archived][profanity-filter-github] by its author.
So, to replace it, I simply started looking for a new package that had the same functionality without requiring some kind of
NLP text-processing module, and I eventually found [`better-profanity`][better-profanity-pypi]{: .no-underline}. It ended up being a
drop-in replacement, although it seems to have fewer features *and* holes in its profanity detection. But, for now,
it's good enough.
## Conclusion
This project is hard to categorize as strictly either a waste of time, practice of skills I still have (kinda), or a
genuine project for my resume. It's hard to say that I learned a bunch of new things, but I also didn't know or remember
half the issues I ran into with it. At the very least though, it has restarted my urge to improve my resume and continue
programming for the first time in months. I ended up putting down 120 commits in less than a week, and I'm still going.
If you'd like, [check out my project][runnerspace-heroku] and [leave a star for it on GitHub][runnerspace-github].
Bye.
[runnerspace-banner]: https://raw.githubusercontent.com/Xevion/runnerspace/master/static/runnerspace-banner-slim.png
[runnerspace-heroku]: https://runnerspace-utsa.herokuapp.com/
[runnerspace-github]: https://github.com/Xevion/runnerspace/
[better-profanity-pypi]: https://pypi.org/project/better-profanity/
[profanity-filter-github]: https://github.com/rominf/profanity-filter/
[pipenv-direct-url-github-issue]: https://github.com/pypa/pipenv/issues/3058
[heroku-and-spacy-model-so]: https://stackoverflow.com/a/70432019/6912830

16
src/pages/posts/test-post.md
View File

@@ -1,16 +0,0 @@
---
title: "Test Post Here"
pubDate: 2023-11-24
description: "This is a test post"
layout: "@layouts/Post.astro"
---
# My First Blog Post
This is just a test to test markdown formatting features
```python
a = None
for i in range(3, a or 93 // 7):
print(list(range(1, i * 2, i / 3)))
```