• GitHub repository
• REST API Documentation

Outdated/Buggy Tutorials

This section is for developers new to the MERN stack.

First of all, do NOT follow MongoDB’s tutorial on MERN stack – it’s outdated in many ways. Some consequences of following this tutorial can be seen here and here.

This FreeCodeCamp tutorial has a good skeleton to build your app from, but it’s also riddled with bugs:

• CORS and react-cookies bugs (details here)
• ToastContainer needs to be moved like this, lest it uses the wrong CSS
• it didn’t tell the readers to specify TOKEN_KEY
• it incorrectly uses bcryptjs where it needs bcrypt

Another note: create-react-app is deprecated; use vite instead!

Where to deploy the backend?

There are numerous free hosts for the static frontend; the real problem lies in where to deploy the backend.

Render is probably the best free-forever solution, although it does make your app sleep after a few minutes of idling – waking up can easily take 40 seconds. I have not tested whether it support stateful apps (AssayPlate is stateless).

Adaptable.io not only makes your app sleep upon idling, it does NOT support stateful interactions (e.g., real-time connections). For example, it cannot functionally host the Resistance Online server.

Warning against Fly.io

I had a very negative experience with Fly.io, and I would like to warn others who are exploring this option for hosting their backend.

Fly.io supposedly offers a one-time, free $5 credit for new users, but this was not my experience. It was actually a bit confusing – suspiciously so. When I registered with my GitHub account, I was prompted to add my credit card before I was allowed to deploy an app, and once I did, I immediately got auto-subscribed to their paid Hobby plan. The whole process made it look like I was just adding a credit card for verification, NOT for subscribing to their paid plan. In fact, someone else had the same experience around the time I “accidentally” subscribed to their paid plan and made a post requesting refund that has still not gotten a public admin response.

That wasn’t the only suspicious practice. A few days after hosting my backend on Fly.io, I started seeing charges for Additional RAM. After digging around, I found out it was because, even though I specified otherwise through their configuration web app, I ended up using the default configuration that uses more RAM than the free allowance. Again, I’m not the only one complaining about this. In fact, you see posts about unexpected charges every so often on their community forum; if anything, this implies their lack of willingness to make the billing process more intuitive/transparent.

The Cookie Saga

I had a LOT of trouble with cookies when working on this project (browser not saving cookie, script not seeing the cookies visible to browser, etc.). Here is a list of things I learned, trying to resolve cookie-related issues for AssayPlate.

how to debug cookie-related issues

Use the browser’s Inspect tool and monitor the network exchanges. On Chrome, this would be the Network tab. Check out the cookie received in each response – the browser will include hints for why certain cookies are discarded. To check what cookies are visible to the browser, go to the Application tab, and find “Storage->Cookies”. Note: cookies that are visible to the browser may not be visible to your JavaScript code! See next section for an example.

different domains?

Initially, I hosted the backend at https://assayplate-backend.onrender.com, which I incorrectly assumed to be on the same domain as https://assayplate.onrender.com. This is because onrender.com is a public suffix (see here for the entire list of public suffixes), so those two are intentionally treated as different domains.

Until I learned about the concept of public suffixes, I was configuring CORS and cookies as if the production backend and frontend were deployed on the same domain, which caused a great deal of bugs.

Now, knowing that the backend is deployed on a different domain from the frontend, we have different issues. As noted here, “Cross-domain cookies cannot be accessed. In case you are building a single page application and your server is on a different domain. You cannot access the cookies on your SPA. The withCredentials flag will just make sure that the network calls made include the cookies and accept any incoming cookies.”

For example, react-cookies would be useless if you want to handle cookies coming from a different domain. A heisenbug that appeared during my development was how, react-cookies was correctly handling cookies when in development, but not during production – turns out it’s because earlier in the local development, both the backend and frontend were hosted on localhost, i.e., the same domain, but in production, they were hosted on different domains, as noted above.

FINALLY, as noted in the MDN, “Browsers block frontend JavaScript code from accessing the Set-Cookie header, as required by the Fetch spec, which defines Set-Cookie as a forbidden response-header name that must be filtered out from any response exposed to frontend code.” As a result, the frontend script cannot directly parse the Set-Cookie header, either.

In the end, I had to rely on the browser to handle the cookie exchanges, since my script cannot access cross-domain cookies through react-cookies nor parse the cookie headers directly. E.g., I had to make the frontend ask the backend to invalidate the cookie rather than having the frontend script delete the cookie locally.

cookie configuration rules

there are some very important cookie configuration rules that, when violated, can lead to cookie-related bugs. Check loginCookieOptions here for the correct cookie configurations for AssayPlate.

• cookie configuration must be done correctly otherwise browser will reject the cookie (e.g., claiming that sameSite is true while sending a cookie from a different domain).
• setting httpOnly to true will make the cookie inaccessible to the JavaScript Document.cookie API, but this only matters when dealing with sameSite cookies, since the script cannot access cross-domain cookies regardless (see previous section)
• Google is restricting third-party cookies! This will soon break some cookie uses that don’t follow their new standard. Attempting to follow the new standard, I tried to set the partitioned flag in the cookie configuration, and ended up uncovering a bug with express.js where the flag couldn’t be set correctly (see here).

react-cookies quirks

even when we can use react-cookies, there are still some things to pay attention to, to avoid bugs:

1. use the CookieProvider wrapper! AND set the correct defaultSetOptions, especially the cookie path.
2. call useCookies correctly… this is more of a JavaScript tip, though. Don’t be dumb like me.

CORS learning

CORS is about letting the browser allow a script from site A to make requests to site B (site B has to define its CORS policy to allow requests from site A). The point is, the browser usually sends the user’s cookies for site B to site B whenever the browser makes a request to site B. If the request was a malicious request NOT made on the user’s behalf, but rather on site A’s behalf, then there is a problem – the cookies can contain authentication information; site A wouldn’t have been able to make requests to site B on its own; site A needed user’s cookies to access site B.

As a result, a number of APIs (e.g., fetch() and XMLHttpRequest) follow same-origin-policy – if the request for content is not by a script from the same origin (site B script requesting from site B), then the request is denied in the preflight check.

Note, again, it’s the browser and the server together that enforces this. When you manually make HTTP requests, you can choose to ignore CORS, fake your origin, etc, although this fact doesn’t help malicious parties that want to coax the browser into making illegitimate CORS requests.

Tips

This section has some tips for good practices in full-stack development.

HTTP tips

GET requests (or DELETE, etc.) shouldn’t have a body: https://www.baeldung.com/http-get-with-body. Just use POST most of the time and structure the actual URLs in terms of CRUD (see this project’s API Documentation)

MongoDB/Mongoose tips

Mongoose documentation can be confusing/lacking at times. Examples:

“Note: Like updateOne, Mongoose registers deleteOne middleware on Query.prototype.deleteOne by default. That means that Model.deleteOne() will trigger deleteOne hooks, and this will refer to a query. However, doc.deleteOne() does not fire deleteOne query middleware for legacy reasons. To register deleteOne middleware as document middleware, use schema.pre(‘deleteOne’, { document: true, query: false }).” Reference here.

I coded a “smart” way to remove dead references upon population from document. It’s more efficient than the answer here, inspired by this downvoted answer in the same post, but does require marking as modified. See GetPlatesAndUsername in PlatesController.js here

react.js tips

• to share states between components, lift them up. For complex state management, use dedicated libraries like Redux.
• state variables should be treated as immutable. Not following this practice easily leads to bugs.

node.js tips

The canonical way to check/set whether the code is running in production is via the NODE_ENV environment variable, as noted here

JavaScript tips

• Empty arrays and objects are truthy – be careful when using comparison shorthands!
• order matters when destructuring an array (as opposed to an object)! Here’s one way it led to a pretty annoying bug, complicated by other issues with cookies mentioned in the earlier section

Axios tips

Axios interceptor doesn’t natively support React hooks. If you really want to use it (e.g., to check all responses for a token expiration flag so you can log the user out), you can try following this workaround. For the record, I hate such workarounds – they tend to be inefficient and result in tech debt.

react-draggable tips

react-draggable currently behaves inconsistently on mobile for conditional renders. See my GitHub issue here.

• Project report paper here
• my code here
• Jonathan’s code here

Tutorials for Dev Tools

I had to learn to use all the tools from scratch, including Docker, Kubernetes, and Istio. Contrary to projects in my previous posts, I am developing on Windows 11 for this project. Here are some tutorials that I found helpful and up-to-date, in order:

• Running Istio with Kubernetes on Docker Desktop (Docker Labs)
• Getting Started with Kubernetes on Docker Desktop (Docker Labs)
• Setting up Istio with Docker-Desktop on Windows 10 (Medium post)
• Getting Started (official Istio guide)

Regarding the official Istio guide, if we wanted, we can do step 3 of “Deploy the sample application” in Bash (e.g., Git Bash). We’ll be able to confirm the whole setup is working interactively later, either way.

The Medium guide says to forward port 80 to 8080 for some reason, but accessing BookInfo like this worked fine for me:

http://localhost/productpage

(now we can access it from other devices in the same local network, after looking up host address with ipconfig)

References and Documentations

Here are some links to references/documentation that were very helpful for development:

• Using Lua filter to modify HTTP response body in Istio (StackOverflow question)
• Lua Filter (Envoy doc), which is one kind of HTTP_FILTER native to the Envoy proxy; in turn, Envoy proxy is the backbone of Istio.
• Envoy Lua script examples (Envoy doc)
• Lua (proto) (Envoy doc)
• EnvoyFilter Docs (Istio doc)
• EnvoyFilter Samples (Istio GitHub doc)

Observations

Docker RAM usage

Docker was eating up my RAM. I don’t know why vmmemWSL still says it’s using around 4 GB of memory after I kill Docker Desktop. I ended up doing wsl --shutdown after I’m done developing for the day. There have been similar reports: WSL2 + Docker causes severe memory leaks in vmmem process, consuming all my machine’s physical memory.

importing Lua libraries

IBM Cloud has a tutorial for writing Lua filters in Istio, which involves loading external Lua libraries. As it turns out, importing external Lua library is NOT trivial; it requires updating the Docker image. References here and here.

Lua filters are stateless

Lua filters are stateless and can’t be used for e.g., caching. (I ended up just using it for per-request compression)

• “Lua state is per worker”
• “Shared state in lua would negate a potential requirement to utilize c++ simply to achieve shared state.”
• alternatively, build it in Rust/Go/… and compile to Wasm. References here and here.

questions about Envoy Filter

Here are some questions that I had trouble finding answers to, regarding configuring the Envoy Filter through .yaml files:

1. What is the difference between envoy.filters.http.lua and envoy.lua for patch.value.name?
2. inlineCode vs inlineString?

Jekyll Plugin Writing

adding static files to site

Turns out that adding static files like the .svg files to the built site requires some fiddling around. Specifically, if the files didn’t exist before and are added by a Generator plugin during build, the generator must specifically mark those files as static so they can be copied to the built site eventually. What’s more annoying is that if those files already exist before the Generator executes, even if the Generator updates those files, they must not be marked as static files again, otherwise Jekyll complains about duplicate static files. This is why I wrote the copy_as_necessary function in jekyll-mahjong-generator

There is another potential pitfall in updating (overwriting) source files with a generator: the generator will run into an infinite loop if it updates files during re-builds of the site (updated files will then trigger re-builds again). That’s why I added in an @has_run flag to the generator to ensure that it only runs once per jekyll serve.

Scalability V.S. Customization

styling inside SVG?

If we want to allow styling individual tags inside the SVGs through CSS (e.g., to allow customizing the tile designs’ colors), we need to embed the SVG text in the HTML directly. However, doing that is not scalable for jekyll-mahjong, since its usage typically involves displaying the same tile multiple times (we don’t want to embed the same whole SVG multiple times).

In general, if we do want to style inside SVG:

• make sure the embedded SVGs are minimized (e.g., take out unnecessary data like Inkscape metadata)
• make the components selectable either through IDs (for when we are not reusing the component) or through classes:
  • utility to turn each Inkscape label to id here.
  • add classes to components in Inkscape via the XML Editor (hotkey: ctrl + shift + x)

sideways tiles through CSS?

The approach in v2+ to let CSS handle turning the tiles sideways also makes the display more computation-intensive than JUST displaying standalone sideways SVGs. However, this saves us the trouble of having to create and upload two versions of SVGs for every tile design, and also exposes the styling for customization.

Plugin Showcase

See this Riichi Doc.

Inkscape

Distribute Objects Along a Path

When you want to distribute multiple objects along a path, make sure to center-align all of them and group them (see below). Select the group and then the path. Next, use the extension:

Extensions -> Generate From Path -> Distribute Along Path (or “Scatter”, in older versions of Inkscape).

Select “If pattern is a group, pick group members”, and the “Sequentially” mode.

Note that you need to adjust parameters like the size of the objects, the shape of the path, etc. to get your desired look. In the demo above, you can see that the resulting spread wasn’t symmetrical and the group of objects is only half-way through the second repeat.

Create Tables in Inkscape?

Although you can use the “Construct Grid” path effect, it’s much easier to make and style the table in any spreadsheet tool like Excel and Google Sheets, export the table in .pdf, and then import the .pdf into Inkscape.

Rounding corners of an object?

Select the object -> Object to Path -> Add Corners LPE -> click and drag to select all desired corners -> drag a corner and see!

IrfanView

Tiled Images

Image -> Create Tiled Images. This is good for printing 4 flyers on one page:

Merge Images

Image -> Merge Images. This is good for showcasing before-and-after, front-and-back, etc.:

GIMP

Healing Tool

This is useful for masking imperfections to ensure a consistent texture. Simply select a contiguous texture that contains no imperfection, paste it over any imperfection, and use healing tool to mask any inconsistencies at the copy-paste border.

Programming

Jekyll, Liquid, Bootstrap, and CSS.

Jekyll

testing locally

bundle install
bundle exec jekyll serve --livereload

(might need bundle add webrick for Ruby 3.0 or later)

Then site is available on http://localhost:4000

serving to other local machines in WSL2

1. go into the advanced settings of Windows Defender Firewall and add an inbound rule that allows connecting to port 4000 via TCP. For more details, see this article.
2. find the WSL2’s IP (NOT its vEthernet (WSL) from ipconfig on PowerShell) with hostname -I in the WSL2 shell. Let’s say this is a.b.c.d.
3. In PowerShell, netsh interface portproxy set v4tov4 listenport=4000 listenaddress=0.0.0.0 connectport=4000 connectaddress=a.b.c.d
4. bundle exec jekyll serve --host=0.0.0.0 (note that --livereload slows down loading A LOT for other devices on the network so it’s disabled here)
5. find the host’s IPv4 address with ipconfig in PowerShell. Say it’s 192.168.x.y
6. access the hosted Jekyll site from other local devices via 192.168.x.y:4000

portproxy

• as noted in this StackExchange response, the default port forwarding for WSL2’s host device doesn’t work for other devices on the local network, and this is why we needed the portproxy steps above.
• Note that WSL2 IP may change upon restarting Windows, so we need to redo the portproxy steps every time WSL2 gets a new IP address. See related discussion here. My WSL2’s IP has thus far remained the same across multiple restarts of both Windows 11 and WSL2 itself 🤷.
• use netsh interface portproxy show all to see all portproxy rules.

Liquid

Iterating through a JSON object can be done as follows:

{% for social_key_value in site.data.socials %}
    {% assign social_name = social_key_value.first %}
    {% assign social_obj = social_key_value.last %}
    <a href="{{ social_obj.url }}"><img src="{{ social_obj.image | relative_url }}" title="{{ social_name }}" alt="{{ social_name }}" /></a>
{% endfor %}

References: shopify doc and Stack Overflow answer.

Also, you can escape Liquid processing with

{%- raw -%}
{% endraw %}

(can you guess how I produced the code block above?)

Bootstrap

• The grid system (xs, sm, md, lg, xl) seems to be at the core of the Bootstrap flow.
• the version of Bootstrap can be found in _scss/bootstrap/bootstrap.scss in our Jekyll site.

SASS / SCSS

Use sass-migrator to address deprecation warnings like this. For Jekyll purposes it’s likely just this, though:

sass-migrator division --verbose --migrate-deps _sass/*.scss

Getting/Hosting Content

CloudFlare, Free Image Hosting,

CloudFlare

Use pull requests to allow previewing deployments. This deploys the site to a preview URL. Unfortunately this still counts against the 500 free monthly deploys.

Free Image Hosting

Google Photos

For a lightweight solution (minimal image hosting), we can embed pictures from Google Photos. This is not scalable but enough for longhornriichi.com (it only needs to show a select few photos). We can adjust the size of the embedded image by editing the link like this.

Cloudinary

Given a greater image hosting demand (like peterish.com), we can use Cloudinary (I have to yet to figure out if there is a storage limit for the free tier). It has built in support for serving cropped, resized, etc. versions of an image (without affecting the uploaded, full image).

NOT Imgur

Do NOT use Imgur as a CDN because this violates their TOS.

They are ambiguous about hosting images through them on your blog, but best not to try your luck.

Free Illustration

いらすとや has cute cliparts for free use; the interface is in Japanese, though.

GitHub

1. install git
2. generate an SSH key for GitHub.
   • Bash

       ssh-keygen -t ed25519 -C "YOUR_EMAIL_HERE"
       eval "$(ssh-agent -s)"
       ssh-add ~/.ssh/id_ed25519
     

   • Powershell

       ssh-keygen -t ed25519 -C "YOUR_EMAIL_HERE"
       start-ssh-agent
     

3. add the public key to GitHub: cat ~/.ssh/id_ed25519.pub here
4. git config stuff:

    git config --global user.name "YOUR_NAME_HERE"
    git config --global user.email "YOUR_EMAIL_HERE"
    git config --global core.filemode true
   

5. (optional for Unix) add keychain: sudo apt install keychain, then add the following to .bashrc:

    # Use `keychain` to load my SSH key
    /usr/bin/keychain -q --nogui $HOME/.ssh/id_ed25519
    source $HOME/.keychain/$HOSTNAME-sh
   

OS-specific

WSL

Add the following to .bashrc:

# allow opening webpages with Chrome
export BROWSER="/mnt/c/Program Files/Google/Chrome/Application/chrome.exe"

This will allow opening URLs with sensible-browser, and was enough to make Chrome pop up e.g., when Google’s Python API wants to authenticate the user via a webpage prompt. To make xdg-open work additional fiddling is needed.

UvUManager

UvUManager is a Discord bot used for Longhorn Riichi’s special intercollegiate tournament “UTA vs UTD”. The main technical challenges are:

1. Discord API (discord.py is relatively well documented but still have some ambiguities)
2. Mahjong Soul API (there is no official API; I ended up using and contributing to mjsoul.py)
3. Google Sheets (the Official Google Sheets API was a mess; gspread was MUCH more user-friendly)

The UvUManager repo has information on how to obtain and set up the secrets for the 3 modules above.

discord.py

1. Reloading an extension doesn’t re-import the Python files (i.e., to load the changed imported file, one must restart the bot altogether)
2. Restarting the application in the same process doesn’t reload the dotenv variables (this is likely due to the fact that dotenv doesn’t override existing environment variables – the environment variable values remain the same as the previous process).
3. ephemeral behaves different for different Interactions. For Component interactions, the followup message can be ephemeral regardless of how ephemeral was set in defer(), but for slash command interactions, the followup message’s ephemeral is governed by the ephemeral in defer(). More testing may be necessary.
4. need to call View.stop() before deleting messages that have a View, especially if the View has a timeout (View.stop is a clean-up that cancels timeout tasks, etc.). This requirement isn’t listed on the discord.py doc as of writing.

Mahjong Soul API

Reverse-engineering API

1. The .proto file (like the one here) contains all the readable and searchable Protobuf API. Developers typically need to search here for their desired interaction message with the Mahjong Soul server (and examine the format of the Request and Response objects).
2. you can learn some protobuf field’s possible values with WebSocket Inspector (e.g., the fact that Twitter OAuth2 protobuf request type field should be 10). You can decode Protobuf with this online decoder, but it’s probably better to decode with protoc given sensitive info and that you have the .proto file.

API discoveries

1. open_live field of createContestGame doesn’t actually do anything. In fact, the same toggle on the management website doesn’t work either, as a result.
2. known but undocumented error: ERROR CODE 1209, which happens when trying to pause an already paused tournament game
3. mjsoul.py raises Exceptions if the response contains an Error object. I modified the relevant module locally (like this version) so GeneralMajsoulError has a field errorCode, to help with catching and acting on specific errors (I’ll merge this change to the mjsoul.py repo soon) .
4. logging out through the tournament manager website (or closing the browser tab after logging in) also logs out the bot. Now the bot has to make a new WSS connection before retrying login – trying to login through the old WSS connection results in a 2504 : "ERR_CONTEST_MGR_HAS_LOGINED. Okay this is further confirmed by the switch-contest behavior on browser – it also switched the contest for all WSS connections. It’s like there is only one state for each account no matter how many connections there are. In other words, each account can manage at most one contest at a time.

GCE VM

screen

1. To run the bot without having it die on SSH disconnection, create a screen session. (could also use nohup as a more lightweight solution).
2. while in a screen, use ^A^D to detach from screen and let it run in the background.
3. screen -ls to list all screens, screen -r to resume a session, and killall screen to terminate all screens.
4. GCE VM-specific bug: uploading files through SSH-in-browser while in a screen session crashes the SSH connection…

Develop with local SSH

I.e., not using the SSH-in-browser on Google Cloud.

Set up SSH with gcloud

gcloud auth login
gcloud compute config-ssh

it seems the default account for the GCE VM console SSH can be different from your local account: doc.

SSH with VSCode for GUI Editor

Guide here.

WARNING: VSCode can crash your server! Relevant reports: here and here.

The crash causes the SSH connection to die and prevent further SSH connections into the VM of any form (e.g., Google’s SSH-in-browser). This can be resolved by restarting the VM (STOP and then START).

(can we figure out a better way to edit on VM with a GUI?)