diff --git a/src/pages/posts/jekyll-github-pages-and-azabani.md b/src/pages/posts/jekyll-github-pages-and-azabani.md new file mode 100644 index 0000000..d7bb62b --- /dev/null +++ b/src/pages/posts/jekyll-github-pages-and-azabani.md @@ -0,0 +1,42 @@ +--- +title: "Jekyll, GitHub Pages, and Azabani" +pubDate: 2020-12-04 13:16:22 -0600 +layout: "@layouts/Post.astro" +tags: ["jekyll", "github-pages"] +description: "This is my first time trying to create a proper blog, and my second time using GitHub pages. How did it turn out?" +--- + +This is my first time trying to create a proper blog, and my second time using GitHub pages, [the first][github-pages-resume] less than 2 weeks earlier. + +As a cheap high school student with zero income and expensive subscriptions, it's obvious that I wouldn't try to brunt +the cost of any sort of subscription for a site I barely use - well, not anymore, at least. + +Jekyll is pretty cool - I'm really having fun learning new web tech (like Vue, BrowserSync, Sass). The syntax of the header is something new, but the templating language's filters are something I'm already familiar with, +using Jinja with Flask and Django already (great, a third dialect of both similar and completely different filters to get used to!). + +GitHub pages is interesting too. The main feature is not in its performance, features or ease of use +(the documentation is sort of hard to read), but in the fact that's free. Free static site hosting with Jekyll, +custom domains and HTTPS supported. That's perfect for me, and I wish I had tried using it before now. + +I'm planning to use Jekyll for my future projects now, creating a site for my school's FTC Robotics team, and hopefully, for all of my projects deserving it. + +--- + +Now, I know this may come as a surprise, but: I'm not very good at making good websites. This entire blog as of December 4th, 2020, is of +[Delan Azabani's][delan-azabani] creation, with just a few edits. I hope to change that by designing my own site or changing +enough to say that I've made it my own, but I haven't yet had the chance to do that. + +Delan, if you're reading this before I get to improve it - I really admire your work. Your blog, your interests, your skills, +everything I've read on around you is super cool. Even you website has it's own unique LaTeX-like style. +I have no idea if you had inspiration or came up with everything from the font to the [fleurons][fleurons] to the super cool date icons on the index, +but this is one of the most memorable and true developer sites I've come across. + +I've cleaned up most of the pages and made the repositories structure a little easier to look through, and I'm going to continue doing that, +but overall, this is Azabani's work. I hope me copying your site comes across as admiration more than thievery. +Additionally, the [ISC license][repo-license] has been obeyed, and [you are linked][about-attribution] as the clear creator of the site. + +[github-pages-resume]: https://xevion.github.io/ +[delan-azabani]: https://www.azabani.com/ +[fleurons]: https://en.wikipedia.org/wiki/Fleuron_(typography) +[repo-license]: https://github.com/Xevion/v2.xevion.dev/blob/fc1761725eb5d4f21bbf778ee504c1e89f9ee9dd/LICENSE#L1 +[about-attribution]: https://github.com/Xevion/v2.xevion.dev/blame/fc1761725eb5d4f21bbf778ee504c1e89f9ee9dd/about.md#L13 \ No newline at end of file diff --git a/src/pages/posts/painting-images-with-ipv6.md b/src/pages/posts/painting-images-with-ipv6.md new file mode 100644 index 0000000..c924baf --- /dev/null +++ b/src/pages/posts/painting-images-with-ipv6.md @@ -0,0 +1,178 @@ +--- +layout: "@layouts/Post.astro" +title: "Painting Images with IPv6" +date: 2023-04-14 13:07:43 -0500 +tags: ["ipv6", "python", "asyncio", "websocket", "PIL"] +description: "Have you ever painted images with IPv6? I found out how in 30 minutes." +--- + +Despite how it sounds, this is not a joke. Well, maybe it is, but it's a fantastic demonstration of IPv6 addressing, +and you can join in on the fun right now too! + +Some months ago, I found an interesting little site: [v6.sys42.net][place-v6]*. + +> **Warning**: This site contains image curated by an unrestricted community: It is entirely out of my control what you +> will see on it, but _usually_, it's safe for work. + +To give a short explanation of this site, it's similar to [/r/place][place-reddit], an online canvas where users can +place a single +pixel on a canvas every 5 minutes. + +The difference between [/r/place][place-reddit] and [Place: IPv6][place-v6] is that the latter uses a human-accessible +user interface for +selecting colors and placing pixels, while the other uses IPv6 addresses. + +Hold on - you might be thinking, _"Don't you mean a REST API? Or at least an HTTP webserver? Or even a TCP/UDP +socket?"_. + +None of that, it's not just a normal server accessible by IPv6 exclusively - it's an IPv6 address range booked +specifically for the purpose of receiving R, G, B, X and Y arguments. + +## How does it work? + +To use the service, you send an ICMP packet (a `ping`) to a specific IPv6 address dictated by your arguments. + +The arguments are encoded into the IPv6 address, and the service will receive and parse whatever you put into it. + +> **Note**: The service has since been updated to use a different IPv6 address range, but the concept is the same. + +Originally, the IPv6 address was {% ihighlight digdag %}2a06:a003:d040:SXXX:YYY:RR:GG:BB{% endihighlight %} where `XXX` +and `YYY` are the X and Y coordinates, and `RR`, `GG` and `BB` are the R, G and B values of the pixel. By substituting +you arguments into these positions, you can paint a pixel on the canvas. Lastly, the `S` represents the size of the +pixel. +Only `1` or `2` is accepted, representing a 1x1 or 2x2 pixel. + +On top of this, the values are encoded in hexadecimal, so you can use the full range of 0-255 for each color without +worry. + +As an example, painting the color {% ihighlight dart %}#008080{% endihighlight %} (teal) at the position `45, 445` would +be encoded as +{% ihighlight digdag %}2a06:a003:a040:102d:01bd:00:80:80{% endihighlight %}. To help you pick out the X and Y +coordinates, {% ihighlight java %}45{% endihighlight %} is {% ihighlight java %}0x2D{% endihighlight %} in hexadecimal, +and {% ihighlight java %}445{% endihighlight %} +is {% ihighlight java %}0x1BD{% endihighlight %}. The color is simply placing in the last 6 bytes of the address, no +hexadecimal conversion needed. + +By now, the basic concept should be clear. You can paint a pixel by sending a `ping` to a specific IPv6 address with the +arguments encoded into it. + +In Python, the encoding can be done like so: + +```python +def get_ip(x: int, y: int, rgb: Tuple[int, int, int], large: bool = False): + return f"2a06:a003:d040:{'2' if large else '1'}{x:03X}:{y:03X}:{rgb[0]:02X}:{rgb[1]:02X}:{rgb[2]:02X}" +``` + +> I've been interested in ways to format this dynamically with arbitrary base IPv6 addresses, but I haven't found +> a performant way of calculating it. It seems like it would require lots of bit shifting, precalculated constants, +> and a C extension to maximize performance. A future endeavour, perhaps. +> Additionally, the base IP does not change often (I have only observed it changing once). + +## The Rest of Place: IPv6 + +IPv6 place includes a proper canvas to boot, and you can view it at [v6.sys42.net][place-v6], along with the basic +instructions on how to use it. + +The way you view (and technically, receive information), is through a WebSocket connection. In the browser, you'll +receive updates that are combined into a Javascript-based canvas element. + +On the initial page load, you'll receive a one-time message containing a PNG image of the current canvas. After that, +you'll receive partial updates whenever updates occur. These updates are also PNG images, but they are transparent +except for the pixels that have changed. + +The WebSocket also contains information about PPS, or Pixels Per Second. This is the rate at which the canvas is +updated globally by all users. PPS messages are drawn onto a historical graph displayed below the canvas. + +## 30 Minute Implementation + +I had a lab due the day I decided to implement this, so I only had 30 minutes to spare. I decided to use Python, as it +offers a lot of flexibility and is my go-to language for anything quick. + +I realized quite early on that any normal ping operations would be far too slow, and looked into batch pinging +implementations. + +I found the [`multiping`][pypi-multiping] package first, and stuck with it. As the name implies, it specializes in +sending multiple pings at once, and is quite fast for Python. It also supports IPv6 without complaint. + +The great part of this package is that it allows you to set timeouts - this is key in performance, as we don't care +about the response, only that the packet was sent. This allows us to send a large number of pings at once, and not +have to wait for a response. + +Here was my first implementation: + +```python +def upload_pixels(pixels: List[Pixel], chunk_size: int = None): + """ + Given a list of pixels, upload them with the given chunk size. + """ + ips = [get_ip(x, y, rgb) for x, y, rgb in pixels] + return upload(ips, chunk_size) + + +def upload(ips: List[str], chunk_size: int = None): + # Default to maximum chunk size + if chunk_size is None: chunk_size = maximum_chunk + + chunked = list(chunkify(ips, min(maximum_chunk, chunk_size))) + for i, chunk in enumerate(chunked, start=1): + print(f'Chunk {i}/{len(chunked)}') + multi_ping(chunk, timeout=0.2, retry=0) +``` + +`multiping` only supports sending a maximum of 10,000 pings at once, so I chunked the list of IPs into groups of 10,000 +before letting `multiping` handle the rest. + +Receiving data from the Websocket required a bit more guesswork, but the ultimate implementation is quite +straightforward: + +```python +async def get_image(websocket): + while True: + data = await websocket.recv() + if type(data) == bytes: + return Image.open(io.BytesIO(data)) +``` + +I decided to use [Pillow][pypi-pillow] for image manipulation, as it's a well-known and well-supported library. I +went with [`websockets`][pypi-websockets] for the WebSocket implementation. + +Since the Websocket is used for both image updates and PPS data, a type check is required. PPS data is completely +ignored. + +## The Final Implementation + +While my first implementation was just fine for 30 minutes, and I could have left off there, I wanted to see if I could +do better. Initial usage demonstrated that any given pixel could fail to be placed - at high PPS, it seems some pixels +don't reach the server. Either the packets are dropped, they aren't ever sent, or the server simply doesn't process +them. + +Whatever the case, a 'repainting' mechanism had to be implemented, and the Canvas had to be kept in memory. + +I manage this with an async-based `PlaceClient` class that provides methods for receiving WebSocket updates, storing +the canvas image, identifying differences between the canvas and the local image, and sending pixels synchronously. + +Here's what my final API looks like: + +```python +client = await PlaceClient.connect(os.getenv(Environment.WEBSOCKET_ADDRESS)) +# Set the current target to the image we want on the canvas +client.current_target = original_image +# Launch a background task to receive updates from the WebSocket and update our local 'canvas' +asyncio.create_task(client.receive()) +# Paint the image with up to 5% error (difference tolerance) +await client.complete(0.05) +``` + +You can check out the repository at [github.com/Xevion/v6-place][github-v6-place]. + +[place-v6]: https://v6.sys42.net/ + +[place-reddit]: https://www.reddit.com/r/place/ + +[pypi-multiping]: https://pypi.org/project/multiping/ + +[pypi-pillow]: https://pypi.org/project/Pillow/ + +[pypi-websockets]: https://pypi.org/project/websockets/ + +[github-v6-place]: https://github.com/Xevion/v6-place \ No newline at end of file diff --git a/src/pages/posts/project-facelift-new-and-old.md b/src/pages/posts/project-facelift-new-and-old.md new file mode 100644 index 0000000..0dda3cb --- /dev/null +++ b/src/pages/posts/project-facelift-new-and-old.md @@ -0,0 +1,81 @@ +--- +title: "Project Facelift, New and Old" +layout: "@layouts/Post.astro" +date: 2021-02-25 16:41:41 -0600 +tags: ["projects", "photography", "update"] +preview_image: "https://raw.githubusercontent.com/Xevion/Paths/master/.media/banner.png" +description: "Through December, I decided to make heavy visual changes to many of my biggest projects, creating custom banners, of which I am quite proud..." +--- + +Through December, I decided to make heavy visual changes to many of my biggest projects, creating custom banners, of which I am quite proud... + +[![/Paths/ Banner][paths-banner]][paths] + +The most advanced of them, the cells in the background are designed to mimic the real Paths project as closely as possible. +The angled gradient stroke on the text helps break up the text on the background with all + +If you want to see [the SVG form][paths-svg], Adobe Illustrator exports it perfectly! +It's a shame though, [not all browsers support it][paths-svg-glitch], as I figured out on my laptop shortly after using a SVG. + +[![/phototag/ Banner][phototag-banner]][phototag] + +Besides being my first major project, this banner means something personal to me as every one of those photos in the background are *mine*. +I took them with my own camera. You can find every single one on my [photography portfolio][photography-portfolio]. + +[![/Boids/ Banner][boids-banner]][boids] + +The first banner I designed, I'm still not exactly happy with how it came out, but regardless, it works well enough I guess. + +[![/power-math/ Banner][power-math-banner]][power-math] + +A very simple banner indeed. + +[![/contest/ Banner][contest-banner]][contest] + +Yeah, I'm not super proud of this one either. + +To be clear, I am not normally a graphic designer - HTML & CSS is something I struggle with for hours and hours trying to get right +(I hope my efforts have paid off in all that you see), so making designs like this was both a challenge and a learning experience like no other. + +I'm so happy I did it, and I hope those who see it see the work I put in to make my projects look good. Fingers crossed I return to Illustrator for every major project. + +--- + +I created [**power-math**][power-math] in order to create a method for me to practice simple and fast arithmetic problems among others. +I designed it with LaTeX rendering in mind, to give it that feel of 'reality' other fonts can't give. +Once again, another dark theme app since I'm lazy and don't know how to do light themes. + +Using Vue sped this process up *a lot*, and very quickly I had a working application with awesome visuals and animations. +Additionally, since the project is completely client-side, so you can [practice with it yourself][power-math-pages]! + + +I'm probably embarrassing myself quite a bit here with how slow I am - I wasn't kidding when I said I made this project so I could get better! + +![power-math settings][power-math-settings] +The project is setup to support many different kinds of problems, each with their own difficulty settings, as well as the ability to completely disable them. +On top of that, by hovering over each level of difficulty, you can preview what a problem might look like, latex supported! + +--- + +This post was not posted on time correctly, so if anything seems out of place - that's why. I attempted to refurbish it +so things read cleaner and aren't out of place, but I'm not perfect working at 1:40AM in the morning. + +[paths]: https://github.com/Xevion/Paths +[paths-banner]: https://raw.githubusercontent.com/Xevion/Paths/master/.media/banner.png +[paths-svg]: https://raw.githubusercontent.com/Xevion/Paths/master/.media/banner.svg +[paths-svg-glitch]: https://i.imgur.com/ynZ5vqy.png +[phototag-banner]: https://raw.githubusercontent.com/Xevion/phototag/master/.media/banner.png +[phototag]: https://github.com/Xevion/phototag +[boids]: https://github.com/Xevion/Boids +[boids-banner]: https://raw.githubusercontent.com/Xevion/Boids/master/.media/banner.png +[power-math]: https://github.com/Xevion/power-math/ +[power-math-banner]: https://raw.githubusercontent.com/Xevion/power-math/master/.media/banner.png +[contest]: https://github.com/Xevion/contest +[contest-banner]: https://raw.githubusercontent.com/Xevion/contest/master/.media/banner.png + +[photography-portfolio]: https://www.rcw.photos/ +[power-math-pages]: https://xevion.github.io/power-math/ +[power-math-video]: /assets/img/power-math-demonstration.mp4 +[power-math-settings]: /assets/img/power-math-settings.png diff --git a/src/pages/posts/race-conditions-in-signal-handlers.md b/src/pages/posts/race-conditions-in-signal-handlers.md index 2bb4583..befcb57 100644 --- a/src/pages/posts/race-conditions-in-signal-handlers.md +++ b/src/pages/posts/race-conditions-in-signal-handlers.md @@ -3,6 +3,7 @@ title: "Race Conditions in Signal Handlers" pubDate: 2023-07-26 16:08:12 -0500 description: "Signals offer a unique, low-level way of communicating with processes. But under certain circumstances, they can kill processes, even when they shouldn't." layout: "@layouts/Post.astro" +tags: ["tar", "signals", "interrupt", "handler", "process", "unix", "race-condition"] --- Signals offer a unique, low-level way of communicating with processes. But under certain circumstances, they can kill diff --git a/src/pages/posts/restricted-memory-and-data-framing-tricks.md b/src/pages/posts/restricted-memory-and-data-framing-tricks.md new file mode 100644 index 0000000..95a4999 --- /dev/null +++ b/src/pages/posts/restricted-memory-and-data-framing-tricks.md @@ -0,0 +1,195 @@ +--- +title: "Restricted Memory & Data Framing Tricks" +layout: "@layouts/Post.astro" +date: 2022-07-16 13:51:00 -0500 +tags: ["c", "memory", "embedded", "ti", "msp430", "union"] +description: "Tips and tricks I learned about handling restricted memory while working on microcontrollers at my first internship" +--- + +Working on microcontrollers is a far cry from the essentially unlimited memory and floating point operations available +in _Python_ and other high level languages. Here's what I have been learning at my _first_ internship... + +## Restricted Memory + +While working there, I was assigned to a project that required usage of a Texas Instruments MSP430 microcontroller. +While there were many quirks with working on MCUs like this (like having to ditch JetBrains & VSCode altogether!), the +biggest quirk isn't working with C: it's working without `malloc` altogether. + +On low memory devices like this, memory is extremely limited and there is no certainty that your code will not leak memory. +Usage of `malloc` and other dynamic memory allocation methods are considered innately dangerous - while there is a chance +you will write perfect code that will properly allocate/deallocate, you can't be certain that your complex program +won't run out of memory on such a small footprint to work with. + +Instead, variables are assigned either inside methods for short periods (and passed around), or they are assigned statically and globally. +It appears that the libraries I use personally prefer globally accessible variables, which 99% of the time, is very wrong - but in +Microcontroller land, global variables are your friend. + +```c +#include + +uint8_t uid[4]; // A unique identifier + +int main(void) { + UID_generate(uid, 4); // Pass the pointer, the function writes to it (and does not return it) + UART_putUid(uid, 4); // Write the UID to UART +} +``` +```c +void UID_generate(uint8_t uid, int length) { + uint8_t i = 0; + while(i < length) + uid[i++] = RANDOM_char(); +} + +void UART_putUid(uint8_t* uid, int length) { + uint8_t i = 0; + while(i < length) + UART_putChar(uid[i++]); +} + +void UART_putChar(uint8_t value) { + while(!(UCB0IFG & 0x1)); + UCB0TXBUF = value; +} +``` + +
+ + + UART is a serial communication technology we use to send characters & text to the COM terminal. + For more information, click here. + + +
+ +There's not much more to this - don't use `malloc`, stick to the _stack_ for actively executing methods and use _global variables_ +when you need to go into low power mode while maintaining state. + +Overall, this doesn't hinder ones ability to write working code - the features are still there, but the way you access +methods, store data & manipulate is re-organized - sometimes at the detriment to quality & refactoring efforts. + +## Data Framing Tricks + +While at my internship, I used my MSP430 microcontroller to communicate with various devices over UART and SPI. I also sent +commands to a ISO15693 NFC wafers. All of these interfaces are extremely low level and the best documentation I have +is often just a PDF and some random code scattered across the internet. There is no library to speak of, usually. + +Communicating at a low level like this requires reading and writing individual bytes of data into _frames_, or arrays +of bytes with a well-defined structure. + +
+ ISO15693 Write Single Block (Addressed) source +
+ +[![ISO15693 Write Single Block Diagram][iso15693-diagram]][iso15693-diagram-edn] + +Traditionally, commands are built statically all at once in a mostly hardcoded manner: + +```c +uint8_t offset = 0; +ui8TRFBuffer[offset++] = 0x61; +ui8TRFBuffer[offset++] = 0x21; +ui8TRFBuffer[offset++] = 0xA7; +ui8TRFBuffer[offset++] = 0x3E; +ui8TRFBuffer[offset++] = 0xFF; +ui8TRFBuffer[offset++] = 0x58; +// ... You get the idea +``` + +Instead, what if we could format this into a `struct` that we could pass around on the stack with a pointer? + +```c +struct AddressedSingleBlockWrite { + uint8_t Flag; + uint8_t Command; + uint8_t Address[8]; + uint8_t Block; + uint8_t Data[4]; +}; + +int main() { + struct AddressedSingleBlockWrite command; + command.Flag = 0x20 | 0x40; + command.Command = 0x21; + uint8_t address[8] = {0xA7, 0x3E, 0xFF, 0x58, 0x21, 0x32, 0x10, 0xFE}; + memcpy(&command.Address, &address, sizeof(command.Address)); + command.Block = 0x05; + uint8_t data[4] = {0x11, 0x22, 0x33, 0x44}; + memcpy(&command.Data, &data, sizeof(command.Data)); +} +``` + + +Now we have a defined structure in our source code and we can move and manipulate various parts of our command +structures without having to deal with hardcoded offsets. Still though, if we want to copy this command structure into +the buffer, we have to individually copy each part of the command - which will break the second we modify its structure. + +There's a fantastic solution for it: **Unions**. + +```c +union ASBWUnion { + uint8_t data[15]; + struct AddressedSingleBlockWrite marshalled; +}; +``` + +```c +union ASBWUnion demarshalled; +demarshalled.marshalled = command; + +for (int i = 0; i < 15; i++) + printf("%x ", demarshalled.data[i]); +``` + +```sass +60 21 a7 3e ff 58 21 32 10 fe 5 11 22 33 44 +``` + +`union`s are special datatypes that share a single memory footprint (equal to it's largest member) starting at the exact same point memory. +They combine neatly with `struct`s to allow us to represent the `AddressedSingleBlockWrite` as a single byte array. + +
+Note: When implementing this, I do recommend that you create macro definitions for the length of the final command structure. +This will help greatly when it comes to refactoring or making adjustments to your command structure. +
+ +#### Reversing Endianness + +If you check out TI's [sloa141][sloa141] PDF on ISO 15693 commands, you'll notice that many of the examples have +sections of their bytes reversed - sections like the _Address_ and _Data_ sections, but not the entire compiled command. + +One such example, `60``21``9080C2E5D2C407E0``06``55443322`, has the _Flag_, _Command_, +_Address_, _Block_ and _Data_ stored in that order. But for this particular example, how could the address be `E007C4D2E5C28090`? +How could the data be `0x22334455`? This odd ordering has a name - Endianness - the order of bytes as the underlying architecture understands it. + +While for my particular usage, reversing endianness was not needed, it's an interesting problem that can be solved +quite easily with our new data structure. + +```c +void ReverseEndianness(struct AddressedSingleBlockWrite asbw) { + uint8_t temp; + int i = 0; + + for (; i < 4; i++) { + temp = command.Address[i]; + command.Address[i] = command.Address[8 - i - 1]; + command.Address[8 - i - 1] = temp; + } + + for (i = 0; i < 2; i++) { + uint8_t temp = command.Data[4 - i - 1]; + command.Data[4 - i - 1] = command.Data[i]; + command.Data[i] = temp; + } +} +``` + +### Conclusion + +Working in a restricted memory space is not that hard once you get used to the lack of normal functions such as `malloc` +and `printf`, but balancing performance, power consumption, memory allocation and code quality gets harder the more +complex your program gets. + +[iso15693-diagram]: /assets/img/iso15693_diagram.png +[iso15693-diagram-edn]: /assets/iso15693_diagram.edn +[sloa141]: http://www.ti.com/lit/an/sloa141/sloa141.pdf \ No newline at end of file diff --git a/src/pages/posts/runnerspace-built-in-under-30-hours.md b/src/pages/posts/runnerspace-built-in-under-30-hours.md new file mode 100644 index 0000000..10c213d --- /dev/null +++ b/src/pages/posts/runnerspace-built-in-under-30-hours.md @@ -0,0 +1,231 @@ +--- +layout: "@layouts/Post.astro" +title: "Runnerspace, Built in Under 30 Hours" +date: 2022-03-29 13:56:22 -0600 +tags: ["flask", "hackathon", "utsa", "rowdyhacks", "projects"] +preview_image: https://raw.githubusercontent.com/Xevion/runnerspace/master/static/embed-banner.png +description: "I attended Rowdy Hacks 2022 at UTSA, and while I can't say I left smiling, I did create a pretty cool project that I'd like to talk about." +--- + +I attended Rowdy Hacks 2022 at UTSA, and while I can't say I left smiling, I did create a pretty cool project that I'd like to talk about. + +## Rowdy Hacks 2022 + +At Rowdy Hacks 2022, there were a couple tracks and challenges available to those attending; I'll skip telling you what +was available and tell you what we took - 'General' (the experienced developers) track and the 'Retro' challenge. + +For our project, we initially started about making a tile-memorization game that would have a fast-paced MMO aspect +where players would compete to push each other out of the round by 'memorizing' sequences of information (like colored tiles). +Players would have health and successfully completing a sequence would lower other's health while raising your own, +almost like stealing health, but inefficiently (lossy vampire?). + +We planned to do all of this with Web Sockets ran on FastAPI, although we pivoted to FastAPI for the backend when we +couldn't get web sockets to communicate properly. After 4 hours of fooling around and slow progress, we realized that +neither of us were game developers and that this project would not suffice. + + +## Runnerspace + +[![/runnerspace/ Banner][runnerspace-banner]][runnerspace-github] + +Our new plan was to create a retro social media site based on MySpace - it would use Flask for the frontend, +Sass (to make frontend development a little less painful), and SQLAlchemy as a database ORM. Furthermore, with Flask, +Jinja2 is the standardized templating engine. + +In the spirit of the Retro challenge, we made sure to style it like MySpace and other social media sites of the era. + +We started at a slow pace, but eventually, we got on decently and were quickly beginning to make progress on the +frontend - which happened to be the major bottleneck. Given that our development time was limited and both of us were not +database modeling experts, we had to design out database around whatever the frontend ended up looking like. + +Eventually though, after dozens of hours, we were able to come to a stopping point where the site was presentable to +judges. We submitted, got judged, and went through the rest of the Hackathon without too much sleep. +By the time I got home, I ended up going without sleep for 35 hours straight (from 8AM Saturday until 7PM Sunday) through +the whole hackathon. Never have I ever been so tired in my life, but my peak exhaustion was not right before I went to +sleep - in fact, it was in the hours preceding our project submission. + +After waking up the next day, I began working on finishing the project in a way more suitable for a resume submission +and portfolio project; the project in its current state was not deployable, it had a couple major issues and simply put - +it needed a lot more work before it was truly ready for presentation. + +
The rest of this post will document Runnerspace's development.
+ +### Form Validation + +Introducing *real* form validation into the project exposed a LOT of major issues with login, signup and other forms +that I had never thought of before; it's hard to say if some changes that it wanted were just paradigms I decided to +follow or not, but there were definite issues. + +For example, passwords and usernames had absolutely no requirements on them! +A empty password was valid, spaces, unicode (emojis) and more were completely acceptable. when you're designing a webapp, +you never think about the millions of things users could do to mess with it, whether stupid or malicious. + +In the process of form validation, I also ended up combining separate POST and GET routes into one - allowing GET and +POST methods at the same route. I had never thought of taking them at the same route for some reason, as if Flask +wouldn't be able to handle it or something. I didn't do this for ALL my routes - I later ended up creating a `/post/{id}/like` route +to allow users to *like* a certain post; this request was sent through AJAX with jQuery; only POST requests were allowed +on this route. + +Combining POST and GET routes into the same `@route` function had more than a refactoring purpose; it allowed rejected forms +to be returned to the user properly! Displaying errors could now target specific fields! +At the same time, rendering these fields properly became rather complex; a macro function needed to be built in order to +make sure fields were consistent across the site. + +{% raw %} +```jinja +{% macro render_field(field, show_label=True) %} + {% if show_label %} + {{ field.label }} + {% endif %} + {{ field(placeholder=field.description, **kwargs)|safe }} + {% if field.errors %} + + {% endif %} +
+{% endmacro %} +``` +{% endraw %} + +This code is a modified version from the WTF-Flask docs. + +### Heroku Deployment + +Heroku is a fantastic tool for cheap students or developers like me who don't wanna pay, but do want the project running. +Employers are not going to care that much about a branded domain, and they probably won't care about it taking 7-12 seconds to boot up, +but the fact that you could deploy it all and keep it running long term? That speaks volumes. Or at least, I hope it does. :P + +And speaking of volumes, integrating a project with Heroku generates volumes of issues that one would never find in +development locally. + +#### Flask, Postgres and Heroku + +One of the first things you'll find with Heroku is that the drives your application runs on are +ephemeral and anything written in run-time will not exist afterwards. + +Some applications wouldn't mind this, but ours would, given that during development we ran our database ORM, SQLAlchemy, +off a local `.sqlite` file - a local file. This works great for our local developmnet purposes, but given that our +database should persist between dynos (effectively, different workers that work the Flask application) and restarts, it +would result in chaos and confusion if we actually ran with that. + +So, my solution was to simply boot up a Heroku Postgres database, which luckily is completely *free* and can be setup +within seconds; additionally, SQLAlchemy supports many, many database types, including Postgres. + +Additionally, Heroku provides the private URL for connecting to this database through environment variables, so +accessing it and integrating it's usage is quite easy; check the application's environment type variable to see if it's +running in `development` or `production`, and set the database URL accordingly. + +Normally, that's it, but when I ran it for the first time, it errored: Heroku actually provides a different URL than +SQLAlchemy wants; SQLAlchemy wants `postgresql` for the protocol, but Heroku provides a URL with `postgres` instead. While +it once did support the latter, it no longer does. The solution? Simply using `.replace()` on the string to swap out the two. +Do make sure though to include the FULL protocol as the chance that some other part of the URL could contain `postgres` or +`postgresql` are slim, but not zero, and could result in a very, very confusing error if not. I also took advantage of the +`limit` parameter for `.replace()` as *extra* insurance. + +```python +# Heroku deployment +if app.config['ENV'] == 'production': + app.config['SECRET_KEY'] = os.getenv('SECRET_KEY') + app.config['SQLALCHEMY_DATABASE_URI'] = os.getenv('DATABASE_URL', '').replace('postgres://', 'postgresql://', 1) +``` + +Side-note: Do make sure to provide some kind of null-check or fallback for environment variable retrieval; it is possible the environment +variable could be un-configured and the `.replace` will fail on non-string values like `None`. + +Lastly, with the protocol discrepancy solved, one would believe everything to be sorted out? Not quite; there is one last piece to go. + +To connect to PostgreSQL databases, one actually needs to install a Pypi module; `psycopg2`, the most weirdly named module +I've ever seen that needed to be explicitly named and installed for a functioning application. + +It seems the logic in including this was that if they included adapters for every type of database SQLAlchemy could work with, the dependency graph +would be huge and installing SQLAlchemy would add tons of modules and code that would *never* end up being ran, and worse, +could become failure points for applications which might only want to work with SQLite databases, like mine in the beginning. + +#### spaCy Models + +In the pursuit of making sure my application was safe to keep up indefinitely on Heroku and accessible to anyone, I wanted +to make it at least a little hard for people to post profanity on it; I wanted to spend as little time on this part of +the app as possible while getting the most out of it. It turned out to actually be quite a searching frenzy to find a +fully functional module on Pypi that would do this for me, but I eventually found something suitable. + +Adding this to my application initially was quite easy; `pipenv install profanity-filter` +and then `pipenv exec python -m spacy download en`, and the module was ready to go. The module in total was surprisingly easy to use; +instantiate an object, then run text into a method and respond to the web request accordingly. + +Later, I would end up building a WTForms validator to encapsulate profanity checks into a single function, which is another fantastic +reason for using WTForms (I am very happy with how WTForms works in Runnerspace). See below: + +```python +class NoProfanity(object): + def __init__(self, message: Optional[str] = None): + if not message: + message = 'Profanity is not acceptable on Runnerspace' + self.message = message + + def __call__(self, form, field): + if profanity.contains_profanity(field.data): + raise ValidationError(self.message) +``` + +Based on how this works with `__call__`, you'd note that a function (or *lambda*) is a perfectly fine substitute, +but using an object lets one provide a custom message with your validators. I really like the pattern the documentation +showed me and would love to use it sometime again soon; perhaps other languages and projects could use something similar. + +So, what went wrong? Disk persistence. In the process of getting Heroku to boot, it would completely fail to load the +`profanity_filter` module! The `spaCy` models it needed did not exist. So, I found out that `spacy` provided pragmatic +methods to download models on the fly... + +```python +if app.config['ENV'] == 'production': + # Ensure spaCy model is downloaded + spacy_model = 'en_core_web_sm' + try: + spacy.load(spacy_model) + except: # If not present, we download it, then load. + spacy.cli.download(spacy_model) + spacy.load(spacy_model) +``` + +But again - it didn't work. I submitted a discussion post asking for help, and they told me you need to restart the process +in order to use newly downloaded models. + +So, I had to pivot to a different way to download the model. You may be wondering: why can't I download the model +through requirements.txt or something? Well, the thing is, I use `pipenv` *and* the model it needs is **not** hosted on +PyPi! + +So, I started looking into it - how do I get `pipenv`, which works with Heroku right out of the box, to download the +spaCy model for me? It turns out, it's quite straight-forward. `pip` can install packages directly from third-party URLs +directly using the same `install` command you'd use with `PyPi` packages. + +Furthermore, `pipenv` does indeed support URLs [since late 2018][pipenv-direct-url-github-issue] and adding it +is simple as `pipenv install {url}`. So, that should work, right? It'll download the module, and it's even the smallest model version, +so there's not going to be any issues with [maxing out the tiny RAM][heroku-and-spacy-model-so] the hobby dyno is provided with, right? + +Haha, no, it still doesn't work, and I never found out why. Additionally, the `profanity-filter` project is +[abandoned and archived][profanity-filter-github] by its author. + +So, to replace it, I simply started looking for a new package that had the same functionality without requiring some kind of +NLP text-processing module, and I eventually found [`better-profanity`][better-profanity-pypi]{: .no-underline}. It ended up being a +drop-in replacement, although it seems to have fewer features *and* holes in its profanity detection. But, for now, +it's good enough. + +## Conclusion + +This project is hard to categorize as strictly either a waste of time, practice of skills I still have (kinda), or a +genuine project for my resume. It's hard to say that I learned a bunch of new things, but I also didn't know or remember +half the issues I ran into with it. At the very least though, it has restarted my urge to improve my resume and continue +programming for the first time in months. I ended up putting down 120 commits in less than a week, and I'm still going. + +If you'd like, [check out my project][runnerspace-heroku] and [leave a star for it on GitHub][runnerspace-github]. +Bye. + +[runnerspace-banner]: https://raw.githubusercontent.com/Xevion/runnerspace/master/static/runnerspace-banner-slim.png +[runnerspace-heroku]: https://runnerspace-utsa.herokuapp.com/ +[runnerspace-github]: https://github.com/Xevion/runnerspace/ +[better-profanity-pypi]: https://pypi.org/project/better-profanity/ +[profanity-filter-github]: https://github.com/rominf/profanity-filter/ +[pipenv-direct-url-github-issue]: https://github.com/pypa/pipenv/issues/3058 +[heroku-and-spacy-model-so]: https://stackoverflow.com/a/70432019/6912830 \ No newline at end of file diff --git a/src/pages/posts/test-post.md b/src/pages/posts/test-post.md deleted file mode 100644 index 15a03d3..0000000 --- a/src/pages/posts/test-post.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: "Test Post Here" -pubDate: 2023-11-24 -description: "This is a test post" -layout: "@layouts/Post.astro" ---- - -# My First Blog Post - -This is just a test to test markdown formatting features - -```python -a = None -for i in range(3, a or 93 // 7): - print(list(range(1, i * 2, i / 3))) -```