I have recently been playing around with Fedora Silverblue and other similar spins that promote a container based workflow. Silverblue comes with a rootless Podman setup out of the box, but does not include anything Docker related. I have been hearing good things about Podman and Quadlets and opted to make the switch. Unlike Docker’s reliance on Docker daemon to manage container processes, Podman Quadlets leverage the already existing and mature systemd for this. This has several implications, mostly positive in my experience, but does have at least one drawback that I have encountered: systemd already manages a lot of services and your Quadlets will be mixed in with these. This can cause some minor nuisances when trying to view and manage logs, but its far from a deal breaker. Podman and systemd are setup out of the box on Fedora Atomic spins, and this means it is much easier to reproduce and avoids further configuration of the host system. For the most part it is plug-and-play with only a few bits of host configuration I will discuss below.

I wanted this new homelab setup to accomplish the following:

• Caddy: For reverse proxy to connect everything together.
• ddns: This is a simple Go script that I wrote to update my IP address if my ISP changes it on me.
• Jellyfin: Media server for music, movies, podcasts etc.
• FoundryVTT: This is a TTRPG virtual table top web application that is designed with self-hosting in mind.
• coredns: I use this to resolve domains running locally to the appropriate private IPv4 address.

This was the easy part. Caddy works pretty much out of the box with very little setup. I just created an image using the Caddy image on Docker hub, copied the Caddyfile into the image, and then created a simple Quadlet that exposes the required ports and mounts the data directory into a volume so the logs persist. Mounting the data directory is probably not really needed since the logs are managed by journald. You could also mount the Caddyfile into the container in the .container file, but I prefer to just bake these kinds of config files into the image itself. What I would recommend is, when you are developing the container, mount config files into the container with the podman -v volume flag, and then when you have a stable config, you can optionally bake it into the image itself.

This script fixes my dynamic IP issue, and there are a few things to talk about here. This script is a ’oneshot’ style process that checks if my public IP matches the A record, if so it simply exits, but if it they do not match, it will update the A record (there is no need to update the CNAMEs since the A record it now points to is correct). We need to think about how we want to run and schedule the script. One way to do this is with cron or some derivative like anacron. This works well enough, but cron is not always setup on a system by default, and it can be a minor pain to set it up. If you want to replicate the setup, its just one more thing to deal with. You inevitably forget and then scratch your head in confusion when your A record is not aligned with reality. Alternatively, you can install cron inside the container itself and have the container manage scheduling internally. Another option is systemd timers.

Jellyfin is an excellent open-source media server that is similar to Plex. You can use it to stream music and movies across a network, has built-in authentication, create users, define permissions, and perform many other media management tasks. Jellyfin is built with containers in mind, and the docs provide detailed installation instructions for using it with containers. One thing to keep in mind is, while you would be able to stream audio just fine without a GPU, streaming video requires transcoding which will DESTROY your CPU (its gonna be red hot!). Streaming just one 4k video causes my CPU to run all 12 cores at 90-100%, and my older t480 CPU couldn’t do it at all. The good news is that a decent GPU barely needs to get out of bed to transcode a video, and the gtx 3060 I use for this setup can handle several in parallel without any issue at all. This requires a little bit of extra setup that is not there by default on Fedora Atomic spins. Even though the laptop I am using cannot easily passthrough its GPU to a full VM, by following these instructions I was able to get it working with Podman in a few minutes.

Jellyfin also, realistically, requires you to mount several directories that give it access to your media, config, data, and cache to be persistent. One hiccup I encountered here was that I initially attempted to create volumes for the config, cache and data directories because you do not actually need to manage them directly on your system, and I did not want to clog up my filesystem with files I do not actually need to interact with, but in practice I ran into very annoying file permission issues. Jellyfin runs as user that may not be able to read and write properly to a directories separate volume. At this point, I recommend just putting these directories alongside your media (which you obviously need to access), and mounting those directly in your container file.

There is not a lot to say here. Initially I used a Docker compose to manage this and I simply translated this into a Quadlet container. I just needed to point the Containerfile to the .zip file used to install Foundry and the persistent data directory where my systems and worlds. That was about it! pretty easy to setup because its designed with this kind of self-hosting in mind.

The final element of this was an experiment with coredns. I wanted to create a local DNS server that resolves my domains to my local IP without needing to reach out the a public DNS server that resolves to my public IP. This way, if I use my domain name locally, it will immediately resolve to the private IP address of the server running the service in question. All other domains “fallthrough” this rule and are forwarded to a public DNS server; such as the one provided your ISP. To accomplish this, I wrote a very simple Corefile that matches the relevant domains, and redirects and forwards all other requests.

.:1053 {
    hosts {
        192.168.1.200 media.domain.io
        192.168.1.201 foundry.domain.io
        fallthrough
      }
    forward . 1.1.1.1:1
    errors
    log
}

The main thing to note above is the is directed to listen on port 1053. The default DNS port is 53, but this is a privileged port and the setup uses a rootless podman setup, which means that it cannot expose port 53. Since I am running this on Fedora it comes with firewalld pre-configured. Firewalld is able to proxy a privileged port to an unprivileged port with a simple command like

firewall-cmd --permanent --add-forward-port=port=53:proto=tcp:toport=1053

This will proxy all external requests to port 53 to localhost port 1053, which rootless podman is then able to access. It is important to note, that some systems use different firewalls, such as ufw, and you may need to find a different solution.

Initially I had hoped to be able to set the default DNS server in my router, but when I attempted to use a local IP for default DNS server, it broke the Ethernet link to that computer. This may be router specific, and I hope to resolve this in the future. In the meantime, I have to set each computer to use the server computer as their DNS server, which seems difficult if not impossible on things like smart TVs.

The biggest issue I faced in the project was getting information from an Org mode document into the HTML for the website. In the previous iteration, I baked the metadata for a post directly into the HTML itself using a BEGIN_EXPORT html org source block. org-html-export-to-html will pick this up and insert the snippet into the produced HTML document. This worked well because it meant I only needed to send one HTML file to the server; however, there were drawbacks. For example, if I didn’t want to display parts of the snippet, I needed to remove them somehow on the server side. It was also cumbersome and ugly to have all the Org versions of my blog posts starting with a big block of HTML. For Go Emacs, I didn’t have to send files to the server because everything was taking place on my host machine.

After some research, I realized that the org-collect-keywords function could not only collect Org keywords, but could also extract arbitrary user-defined keywords. This felt like a good “Org native” way to store and collect post metadata, but the question remained: how to get the metadata from Elisp land into Go land?

I wanted to add a searchable “tags” feature that allowed for posts to be sorted by tag or topic. I created an org keyword to store the tags for the post, for example #+tags: Emacs Go. In go-emacs-publish I then write those tags into the TOML file by collecting the tags with (org-collect-keywords '("TAGS")), which returns an alists of the keywords and their values. You can then extract the values with something like (car (alist-get "TAGS" keywords nil nil #'equal)). A quirk of alist-get is that it uses the eq function to test the keys to find the correct one to return, but eq only works on symbols and org-collect-keywords returns an alist where the keys are strings. To resolve this, you need to pass a TESTFN argument, such as #'equals or #'string= for this to work correctly. Once the tags are collected it can be inserted into the metadata file. The full function looks like this:

(defun go-emacs-publish-post-metadata ()
  "Collecs metadata for a post and writes it to a metadata.toml file."
  (let* ((keywords (org-collect-keywords '("TITLE" "DATE" "TAGS" "SUMMARY")))
         (title   (car (alist-get "TITLE" keywords nil nil #'equal)))
         (date    (car (alist-get "DATE" keywords nil nil #'equal)))
         (tags    (car (alist-get "TAGS" keywords nil nil #'equal)))
         (summary (car (alist-get "SUMMARY" keywords nil nil #'equal))))
    (with-temp-file "metadata.toml"
      (insert (format "title=\"%s\"\n" title))
      (insert (format "tagString=\"%s\"\n" tags))
      (insert (format "summary=\"%s\"\n" summary))
      (insert (format "datestring=\"%s\"\n" date)))))

Finally, there isn’t much work to do on the Go side. The package I am using to deserialize the TOML file into Go is able to extract the keywords directly into a slice if the struct field you define to hold the data has a slice datatype. I used this same basic pipeline to get data from Emacs into go.

I have a few ideas for things to add next. For example, I can’t add custom HTML to the home page, or the footer. It would be nice to have a way to optionally add a snippet to the footer if there is something I want to appear at the bottom of every page. I would also like to do some incremental changes and improvements for the CSS and to make thumbnails optional for the homepage “list” mode.

In simple terms, an IRC bouncer is an application that is able to connect to one or more IRC servers, such as Libera Chat, and sending and receiving messages via the IRC protocol in a similar way to a regular IRC client. There are some key distinctions though. An IRC bouncer is designed to be connected to an IRC server continuously and is able to log messages that are sent when we aren’t connected with a client. Another distinction is that when we connect with an IRC client, our username, or nickname, is registered with the server. This means that our nickname will be unavailable if we try to connect again with another client. The workaround to this is that clients provide the functionality to define aliases or backup user names that look very similar to our normal user name, but are slightly different. For example, if my normal user name is Naokotani, I could log in as Naokotani~ from a second client and people will most likely be able to deduce who I am. Still, people don’t know for sure that it is you, and it is a bit jarring to many of us that are accustomed other modern chat software where logging in with the same username from different clients is common place.

IRC bouncers pose a solution to this problem. Instead of logging into a server separately from different clients, the IRC bouncer logs into the server once with our nickname and stays logged in. The bouncer then acts as a proxy that we can connect to multiple times from different clients and, as far as the server is concerned, we have only logged in once.

IRC bouncers also solve the problem of missed messages. They are able to log messages that you miss while you are not connected with a client because they are connected all the time. The bouncer can log messages and use a “playback” to show you the messages that were sent while you were not connected. This is similar to other chat software that will use a horizontal rule to separate the messages that you have “seen” from those that were sent while you were away.

As is customary with open protocols like IRC, there are many ways to implement this functionality. In my case I opted to host a ZNC bouncer on a VPS, in my case I used Digital Ocean, but any provider should suffice, run it in Docker, reverse proxy it with Nginx and then connect to it from Doom Emacs using the Emacs Circe client, the default IRC client for Doom Emacs.

These selections are somewhat arbitrary, but determined by infrastructure that I had set up previously for other projects, and there are many viable alternatives to these. Soju is a popular alternative to ZNC that you might want to check out. I am using Nginx as the reverse proxy for my setup and, while it is very capable of performing this task well, Nginx can be a little bit arcane and hard to setup. Caddy is another popular choice for a reverse proxy. It is generally easier to setup and has built-in automatic SSL. If you don’t like Docker for whatever reason, you can always use Podman, or of course you can just run the bouncer directly on your host machine. As for clients, you could also use ERC, a CLI app like Weechat, or a GUI client like Polari. I could digress and talk about all of these, but there are enough moving parts here already that it’s best we get to it.

The first step is to get ZNC itself running on your machine. You can do this in two simple commands. First run:

docker run -it -v znc-cfg:/znc-data znc --makeconf

This will pull the official ZNC image if it’s not on your machine, create a persistent volume for the ZNC configuration data called znc-cfg, run it with an interactive terminal, the -it part, and then run ZNC with the --makeconf flag which, as the name suggestions, will setup an initial configuration file.

When you do this you will be prompted to make some initial configuration entries. You should get prompts that look like this.

[ ** ] -- Global settings --
[ ** ]
[ ?? ] Listen on port (1025 to 65534): 12345
[ ?? ] Listen using SSL (yes/no) [no]:
[ ?? ] Listen using both IPv4 and IPv6 (yes/no) [yes]:
[ .. ] Verifying the listener...
[ ** ] Unable to locate pem file: [/znc-data/znc.pem], creating it
[ .. ] Writing Pem file [/znc-data/znc.pem]...
[ ** ] Enabled global modules [corecaps, saslplainauth, webadmin]
[ ** ]
[ ** ] -- Admin user settings --
[ ** ]
[ ?? ] Username (alphanumeric): User
[ ?? ] Enter password:
[ ?? ] Confirm password:
[ ?? ] Nick [User]:
[ ?? ] Alternate nick [User_]:
[ ?? ] Ident [User]:
[ ?? ] Real name (optional):
[ ?? ] Bind host (optional):
[ ** ] Enabled user modules [chansaver, controlpanel]
[ ** ]
[ ?? ] Set up a network? (yes/no) [yes]:
[ ** ]
[ ** ] -- Network settings --
[ ** ]
[ ?? ] Name [libera]:
[ ?? ] Server host [irc.libera.chat]:
[ ?? ] Server uses SSL? (yes/no) [yes]: yes
[ ?? ] Server port (1 to 65535) [6697]:
[ ?? ] Server password (probably empty):
[ ?? ] Initial channels:
[ ** ] Enabled network modules [simple_away]

First, it’s important that Listen on port is set to the same port that you exposed in your Docker command. The Docker command only binds the container’s port to your hosts port, but the Listen on port is what actually tells ZNC to listen on that port inside the container. Additionally, we want to set Listen using SSL to no because ZNC itself is not going to be handling SSL in our setup, Nginx will do that. ZNC should only be exposed internally on the server, so encryption is not necessary. Finally we need to create an admin user that we will use later to connect to the ZNC web front end. You can set the username and password to whatever you wish, but keep in mind that this will be publicly accessible on the web, so probably avoid something like user: admin pass: so-secret. It is convenient to setup the network settings now because it will be mostly applying default settings, but you can also set it up in the web front end later if you wish.

Once this is setup, you can run this command:

docker run -d \
  --restart unless-stopped \
  --name znc-server \
  -p 12345:12345 \
  -v znc-cfg:/znc-data \
  znc

This command will run ZNC in a Docker container. --restart unless-stopped will make it so the container will automatically restart if the process ends, for example if your server restarts. --name znc-server will name the container. If you don’t do this it will still work, but Docker will automatically assign a silly name like wonky_wombat or something. Naming the container makes things easier if you want to make changes later. -p 12345:12345 will expose the port 12345 inside the container to the host outside the container. This makes ZNC accessible from outside the container. The ports don’t need to be the same, and you can choose whichever port you would like in the available range of non-privileged ports: 1024-49151. The main thing is if you have another process running on a port you cannot reuse it, so you have to make sure the port is available. If there is a conflict, you can run lsof -i :12345 to check what is running on port 12345 on your host, or more simply just choose a different port. Finally, -v znc-cfg:/znc-data means the container will use the volume we put our configuration file in earlier. If we don’t do this, we will lose our configuration data when the container restarts.

Now that we have ZNC up and running, we need some way to access our ZNC process from outside our server. You can expose ZNC directly to the public internet, but it is common practice to route public traffic through a reverse proxy like Nginx or Caddy and have that handle directing the TCP and HTTP requests to your ZNC server. This also means that Nginx or another reverse proxy will handle SSL instead of ZNC, which is why we said no to SSL encryption in the initial ZNC configuration.

Our first step will be to ensure that we have a FQDN (like irc.mysever.com) pointing to our server. It is possible, but cumbersome to enable SSL without a FQDN, and in most cases not worth the effort. The ZNC wiki gives two options for this. You can either use a sub domain, for example irc.myerver.com, or you can use a route on an existing domain, such as myserver.com/irc. We are going to use the sub domain method in our setup. Once the domain is pointed to our server’s IP and the ZNC server is running, we can make a location block like this in our nginx.conf file:

server {
    server_name irc.myserver.com;
        location / {
            proxy_pass http://[::1]:12345/;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
}

This means that any http requests that Nginx gets to irc.myserver.com will be directed to port 12345 on our host machine where ZNC is listening because we added the Docker -p 12345:12345 flag and we input for Listen on port when you ran znc --makeconf earlier. It will also forward the headers from the original request on to port 12345. Once we have reloaded Nginx and this takes effect, requests should already be routed correctly, but we still need to set up SSL before we can connect the ZNC via HTTP.

While Nginx does not have built-in SSL like Caddy, it is fairly easy to set up. Certbot will setup certificates for us and update our nginx.conf file to point to the correct certificates.

At this point, you should be able to access the ZNC web front end, and you can input the username and password you input earlier to access the configuration for your ZNC server, but there is something else we need to do with Nginx. The location block we setup earlier listens specifically for HTTP and HTTPS requests on ports 80 and 443 respectively, which is exactly what the ZNC web front end uses, but the IRC protocol itself uses TCP protocol. Nginx needs a way to appropriately proxy TCP requests to ZNC. Furthermore, since port 443 is the standard port for HTTPS requests, we need to configure Nginx to listen for TCP requests on a different port. To do this, we need to create a stream block (you can find the documentation for this here) similar to this:

stream {
    upstream znc {
        server [::1]:12345;
    }

    server {
        listen 54321 ssl;
        proxy_pass znc;

        ssl_certificate /etc/letsencrypt/live/irc.domain.dev/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/irc.domain.dev/privkey.pem;
    }
}

We need to create a stream block that defines a server and upstream block. The server defines the port Nginx will listen on with the listen directive, which we set to port 54321 in this example. The listen directive also specifies that we are listening for SSL connections. We can use the same SSL certs that we created for the location block for the web front end earlier by simply copying the directives from our HTTP location block. The proxy_pass directive passes that request on to the named upstream znc block defined above, which defines the internal port that Nginx should pass the TCP stream on to, 12345. The port defined here must match the port that we bound the Docker container to, which in our case was 12345. You can double check this by running docker ps and then under PORTS you should see something like 0.0.0.0:12345->12345/tcp, [::]:12345->12345/tcp, which tells us that both IPv4 and IPv6 port 12345 inside the container is bound to the same port on the host. I used different ports, 54321 and 12345 here to make the distinction between the two, but they can be the same number. Another way to think of this, is that it’s very similar to Docker -p 12345:12345; its binding TCP requests outside our host to the port inside our host.

You might need to create a firewall rule to ensure that your firewall is not blocking port 54321, or which ever port is defined by your listen directive. For example if you are using ufw you can use the command ufw allow 54321/tcp with root privileges which will allow Nginx to receive TCP requests on port 54321.

At this point we should be done with the work on the server itself and it’s time to do some work on configuring ZNC to talk to an IRC server like Libera.

Now that everything is set up on the server, we should be able to navigate to https://irc.myserver.com on the public internet to access our ZNC configuration to do the final bits of work needed to connect to the Libera server.

First and foremost, navigate to the Global Settings tab and scroll down to skin select Forest. This is by far and away the most important step so don’t skip it!

Next, we need to make sure that ZNC is correctly configured to connect to Libera. Select the Your Settings tab and scroll down to the Networks section. If we configured it during the znc --makeconf step, we should see Libera there already. We can click Edit to get further settings. This is where we will input all of the important information we need to log into Libera. Ensure that SSL is ticked and that the port is set correctly to 6697, the default port for libera.chat.irc SASL connections. Finally, scroll down to the Modules section and tick the sasl module. Once everything is set, we can select save and return near the bottom of the page.

Once we have saved these settings we should see Network Modules (libera) section with a SASL hyperlink. We need to click that and input our Libera Nickname and password. This allows ZNC to connect to Libera via SASL. At this point ZNC should be able to connect to Libera and all that is left is to connect to your ZNC server via Doom Emacs, or another client of your choosing.

On Doom Emacs, to add custom configuration we can use M x doom/open-private-config to open up our personal configuration file. In order to connect to our server, we need to add something like the following:

(after! circe
  (set-irc-server! "znc/libera"
    (let* ((auth (car (auth-source-search
                       :host "irc.myserver.com"
                       :port 54321
                       :user "StampChad"
                       :require '(:user :secret))))
           (secret (plist-get auth :secret)))
      `(:host "irc.myserver.com"
        :port 54321
        :tls t
        :nick "StampChad"
        :pass ,(if (functionp secret) (funcall secret) secret)
        :channels ("#stamp-collecting")))))

This will connect to our ZNC server at irc.myserver.com port 54321. You might notice that instead of our plain text password, we are getting our password from our authinfo.gpg file. The process of using an authinfo.gpg file is explained in this excellent blog post. Simply put, you need to ensure you have a gpg key generated by running gpg --full-generate-key and following the prompts and then simply create a file called ~/.authinfo.gpg and input your information in the following format:

machine irc.myserver.com login StampChad password StampChad/libera:s0-s3cr3t port 54321

In the above example, the let* binds the return value of auth-source-serach to the secret variable. Instead of passing our plain text password to :pass, we wrap our Circe config in the let* and then we pass it (if (functionp secret) (funcall secret) secret). This checks whether auth-source-search returned a function or a string. If it returned a function, (funcall secret) calls secret as a function and returns our password, otherwise it just returns the value of secret, which should be a our password as a string.

And that’s it! We should now be able to run M-x =irc, the Doom Emacs command to run Circe, and we are ready to chat. ZNC will stay connected and update us about chat that we have missed in the channels we are connected to. We can also connect with as many clients we want. There are a lot of ZNC modules to explore and this is really just scratching the service, but getting a basic set up going is a big first step.

One of the issues with the old script was that the monitor names were hard coded. Any time I changed computers or monitors it would break, and I had to update the script to work with my new setup. I couldn’t just stick it in version control and forget about it.

# Use hyprctl to get a list of monitors and store them in an array.
readarray -t monitors < <(hyprctl monitors | awk '$1 == "Monitor" { print $2 }')
num_monitors=${#monitors[@]}

This uses awk to parse the output of the hyprctl utility that lists information about the monitors connected to your system in a format that is trivially read by awk. The < <( ) is used for process substitution. This creates a virtual file of the output from awk and passes that to readarray, which expects a file instead of a string. This creates an array from the output of awk. finally, ${#monitors[@]} gets the length of the newly created array and stores it as a variable for later use.

I also updated the logic to get a random wallpaper. As I said in the previous post, it is possible to use something like shuf -n num_monitors to get a list of random wallpaper, but I would still need to handle cases where there weren’t enough wallpapers per the number of monitors, and I thought it would be more fun to create the logic myself (Hint: it was).

# Get a random paper, add it to the array of papers to be used, and
# remove it from the array of possible papers to choose from only if
# there is at least one remaining in the array.
function get_random_paper() {
    # Get a random index from from the length of the papers array.
    local random=$((RANDOM % ${#papers[@]}))
    local paper=${papers[random]}
    #This initializes an empry array to appened to below
    local temp_array=()

    # Create a temporary array of all remaining wallpapers except for the one
    # that was just choosen This will remove it from the array so it will not
    # be duplicated later.
    for p in "${papers[@]}"; do
        if [ "$paper" != "$p" ]; then
            temp_array+=("$p")
        fi
    done

    # If temp_array is length 0, then all papers have been removed.
    # If so, it should not be removed from the array of avaialable
    # wallpapers. This wallpaper will then be duplicated on all remaining
    # monitors. If more wallpapers remain, set the papers array to newly
    # created array.
    if [ "${#temp_array[@]}" -gt 0 ]; then
        papers=("${temp_array[@]}")
    fi

    # Add the selected wallpaper to the array of papers to be used.
    papers_array+=("$paper")
}

# Run the random wallpaper function once per avaialable monitor
for ((i = 0; i < num_monitors; i++)); do
    get_random_paper
done

I made some additional small fixes. I was changing directory in the script to run commands, but I updated it be directory agnostic by setting the location of the required files as a variable at beginning of the script. The only thing that should need to change about the script is updating this if the hypr config files are not located in the default location. Additionally, I used find with readarray to create the array of the absolute file paths to the wallpaper files.

I also ran ShellCheck, which recommended splitting the export of the hyprlctl instance into two lines, one to get the instance, one to export it, and pointed out some spots where I had forgotten to quote variables.

This application utilizes Java Spring framework for the API, React for the front end and PostgreSQL for persistence. For development, the React is served by Vite, but for production, JavaScript is served by Nginx as static files. The entire application is containerized in 4 separate services: Spring, React/Vite, React/Nginx, and PostgreSQL. The front end uses Leaflet JS to generate the map and cluster groupings.

Initially when I was writing the application, I could only have one dataset in the database at any given time. If a new dataset was uploaded, the old needed to be deleted. This meant that if a new dataset was uploaded and, for some reason, did not parse correctly, perhaps as a result of changes in the upstream data source, then the old data could potentially be lost; rendering the application unusable until the problem is fixed. I created a data set table in the database to resolve this. Each time new data is uploaded, metadata about the dataset is stored in this table. Additionally, an “active” boolean column is set to true if it is intended to be sent to the front end, and queries return only rows that have the active data set foreign key in the dataset column. If the application is functioning as expected, the old dataset can then be deleted, and if not, it is easy to roll back to the previous dataset until the problem is resolved.

A major drawback in the application is that the amount of processing required to fetch the data is fairly significant. The application makes a SQL query for the data for each province, and then calculates the aggregate emissions for each emitter based on the year or years selected by the user. Given the limited server resources, response times for the data made the application feel extremely unresponsive. The solution for this was to enable caching on the service that returns the DTO. The data changes infrequently, as little as once a year, making this a perfect use case for caching the JSON data. To accurately invalidate the cached data, I annotated the function that sets the active dataset to invalidate the cache when a new dataset is activated. This way, up-to-date data is always returned, while ensuring that data is cached as long as possible. This lowered the response time from 4k to 5k ms to 50-100 ms during testing.

Aside from minor bug fixes and improvements, I would like to make a version 2 that covers the whole country, but this will significantly increase the amount of data that the back end needs to process. Caching the data will assist greatly in resolving this problem, but a further solution would be to attempt to determine which region the user is in when they load the app and load that region first making the app usable before loading the data for the rest of the country. In the meantime, I will be moving on to other projects for a while. Hopefully, I will find some time to work on NUTS a bit more!

NUTS, Naokotani’s Unusable Text Software (I am also playing with the idea of PEANUTS, Portable Executable Application: Naokotani’s Unusable Text Software), is my new hobby coding project to create a rudimentary TUI text editor similar to Vi or Nano in C. The plan is to create a text editor that stores text in a rope-like data structure based on logical text divisions; buffers, lines and words (“words” are really text snippets, they can be punctuation, white space, etc.). A linked list of buffers (IE, vim, Emacs buffers etc.) points to linked lists of lines, which point to linked lists of words, which contain the strings that make up the text buffer. The intention is not to create the latest greatest vscode competitor, but to do a deep dive into strategies to store text in computers at a relatively low level.

I was recently doing some work implementing a simple linked list in C along with associated functions for working with the data structure. The basic implementation was not very interesting, so I decided to see what other interesting directions I could take the code in. Initially the linked list stored integers, but I started to play with storing dynamic character pointers in the linked list. Next I implemented the functions as Clojure inspired recursive functions (functions like first, rest). Once I had this working with small basic hard coded strings, I created a few simple functions to parse text and place it into my linked list. I fed it some books like the King James Bible and Moby Dick (no rhyme or reason here, its just what came up on Project Gutenberg), and to my utter shock and horror, it instantly crashed with a segmentation fault. The issue was that the recursive functions were causing stack overflows when I was feeding them these longer texts (I believe Moby Dick parsed out to somewhere around 280 000 words and the program got to around 90 000 and got very sad). After some investigation I found that GCC actually supports tail call optimizations as long as you pass the -O2 flag and the return of the recursive function is the last piece of logic in the function. In order to make the tail call optimization work, I had to jump through some hoops and the code got a bit messy, but it worked, and performed pretty well. I didn’t bench mark it, but it was able to parse Moby Dick, put it in a linked list, a recursively copy the list to pull out a random quote in a split second.

This project, while interesting, didn’t feel like it was going anywhere. I would like to spruce up the UI a little bit for creating quotes, but programs that do nothing put pull random quotes out of .txt files are not exactly in hot demand at the moment. Furthermore, the tail call optimization was fun to explore, but I had to bend over backwards to conform the functions to it’s requirements. I do think that there are niche situations where it might be helpful, so its nice to know how it works and some of the pitfalls of recursion in C, but there are costs associated with it as well.

I realized at some point that what I had was already an extremely rudimentary form of text editor. It could only operate on words and couldn’t manipulate white space, but I was able to make arbitrary changes to long passages of text fairly efficiently. By extending the idea up from the word to the line and buffer and down from the word to the character I realized I could create a rudimentary text editor with the knowledge I had gained. More surprisingly, this happened to be at least approximately how text editors often actually work. Sadly, the recursive functions wouldn’t make it to the next step. There is already infinite complexity to explore within the paradigm of a text editor without needing to arbitrarily complicate it.

So far I have created the basic project structure using a template provided by Unity testing framework and created the basic outline of the types I will need. I have started working primarily in Word.c where I am building out the functions needed to manipulate strings. The ability to add and delete characters and to add and remove regions of text. The following snippet gives a good overview of my approach and is pulled largely from my addString() function in Word.c.

// Save the start index.
int index = startIndex;

// Get length of the sub string to copy over from the
// difference of the indicies
int length = endIndex - startIndex;

// Decrement the startIndex until it reaches zero. Increment
// and copy the derefenced pointer to the new string.
while (startIndex-- && (*newStrPtr++ = *oldStrPtr++))
    ;

// Copy the newStr from the new string passed as an argument to the fuction.
// Because the pointer was already incremented and hasn't been reset, its
// already in the correct postion to start copying.
while ((*newStrPtr++ = *inputString++))
    ;

// Increment the index by the length of the copied String.
// This can be used to determine how much if any of the old
// string remains to be copied.
index += length;

// I need to decrement here because the pointer is after the '\0'
// character because of the final increment after the while fails in
// the preceeding loop.
incNew--;

// If more of the string remains, append it to the to new string
// I could also use a while loop, but strcpy internally uses the
// same approach, so it works well to tack on the last bit.
if (index < (int)word->size) {
    oldStrPtr += length;
    strcpy(incNew, incOld);
}

I took this logic for copying strings from “The C Programming Language” by K & R (especially pages 86-88), which describes the logic in detail and is an interesting read. I could make more liberal use of memset or strcpy, but especially in the case of strcpy, the implementation used in the GCC source code uses a nearly identical approach, so I don’t think there is huge advantage to using them over simple while loops. This approach makes it easier to save the position of the pointer for further operations. The function to delete a region of a string uses the same approach. It copies up the start index, increments the pointer by end index minus start index and copies what, if anything, remains after the deleted region.

After doing a bit reading about ropes, I realized that they tend to be immutable and favor statically sized arrays, while my implementation uses malloc to assign memory to the character pointers. This prompted me to do a little bit of investigating with some very rudimentary benchmarking programs.

#define ITERATIONS 1000000

int main(void) {
    char testTemplate[] = "Hello, World!";
    float fixedStartTime = (float)clock() / CLOCKS_PER_SEC;

    for (int i = 0; i < ITERATIONS; i++) {
        char testChar[14];
        strcpy(testChar, testTemplate);
    }

    float fixedEndTime = (float)clock() / CLOCKS_PER_SEC;
    float fixedTimeElapsed = fixedEndTime - fixedStartTime;
    float mallocStartTime = (float)clock() / CLOCKS_PER_SEC;

    for (int i = 0; i < ITERATIONS; i++) {
        char *testChar = malloc(14 * sizeof(char));
        strcpy(testChar, testTemplate);
    }

    float mallocEndTime = (float)clock() / CLOCKS_PER_SEC;
    float mallocTimeElapsed = mallocEndTime - mallocStartTime;

    printf("Fixed time elapsed: %f\n", fixedTimeElapsed);
    printf("Malloc time elapsed: %f\n", mallocTimeElapsed);
    return 0;
}

This created an output on my laptop of approximately 0.002 seconds for copying the fixed size array and 0.02 seconds for the malloc version without any compiler optimizations (I also tried it with calloc, which was slightly slower, but the difference was quite negligible). This is a huge difference in performance between the two, but its not really a realistic scenario, and there are a lot of other considerations.

For example, either the fixed arrays would need to be quite large, yet often would only need to hold a small number of characters, or they would be small and then it would be a common scenario where they would need to be split into multiple nodes in the rope structure to hold single blocks of text. Operations that print a single word might require multiple string copies distributed between multiple nodes. That being said, I do think it is likely that you could find a sweet spot where the array was not too large, but in which the vast majority of words could fit. Additionally, making strings dynamically sized greatly simplifies higher level navigation functions such as next word, delete word etc. If strings were distributed across multiple nodes, even very infrequently, a next word function would need to be able to navigate multiple nodes until it determined it was at the node that held the end of the string for the word.

I attempt to mitigate this problem somewhat by over sizing the memory allocated to the strings with malloc. The addChar() and delChar() functions, theoretically the ones that will be used for normal one-character-at-a-time (IE. regular typing vs. a put-from-kill-ring style function for example), will only need to resize the buffer once every n number of inputs where n is the BUFFER_SIZE defined. If adding a new character would make the word size equal to a multiple of BUFFER_SIZE, reallocarray is used to resize the buffer appropriately by increasing its size by BUFFER_SIZE. Later on it will be interesting to play with that value and see how it would perform in different real world scenarios, but for now I’m left only speculating. Out of curiosity, I made a similar program and, without compiler optimizations, it was able to put a single character in a static and dynamic char array using array indexing 100 million times, and while the static array was faster, the difference was trivial. Let’s just say I don’t think you will notice a big difference unless you spend WAY too much time on monkeytype.com.

All this speculating is interesting, but ultimately largely academic because performance should be well within the range of the acceptable even for very large text files, and certainly more than acceptable for an unusable text software interface. Right now my biggest concern is creating functioning logic so I can start to work on the myriad problems of user input, saving, loading, managing buffers and so on.

When I created my blog app, I decided to use PM2 to manage the node application that runs my blog. This created a few small problems. First my blog is a Node application, which PM2 handles quite well, but my other applications are a mixture of Rust binaries, Java jars, and Node applications. Additionally, my blog relied on git to pull some data from a git repository, which was triggered by a shell script. This meant that I had to manage PM2 processes for node, systemd processes for Rust and Java and Nginx, cron jobs for shell scripts, I had to figure out my development environment and copy files to the server in a convenient way. Suffice it to say this was all becoming something of a mess. The problem came to the forefront when I had an issue with PM2 where I had neglected to restart it after an update, which caused the running PM2 processes and the version of PM2 on the server to be out of sync, which, I assume, was the cause of the PM2 processes continually using virtually my entire CPU. Once I had realized the issue and queried PM2 about how its day was going, it was an easy issue to fix, but it was sort of the straw that broke the camel’s back on getting me to reorganize how my server was managed.

Predictably, I decided to just dockerize everything. There wasn’t anything super interesting here aside from how I set up my cron job within the containers. Previously, I had installed git on my server, and a shell script that lived in my filesystem would do a git pull and copy the relevant files to where they needed to go. Instead, I decided that it would be better to have this managed in a separate image and push it to Docker Hub separately. This way I could make small changes to it without affecting the main application, which could function independently. First, I created a simple shell script:

cd /usr/src/repo-code
git pull https://repos-r-us.com/some-repo.git
mkdir -p /usr/app/static/json
cp /usr/src/repo-code/websites.json /usr/app/static/json/

For the Dockerfile, I use an Alpine image and then install git, which is not included in the image, clone the repo, and create the cron job. The approach is more portable and appears to work well. I believe cron is installed on this Alpine image, so there is no need to install it independently.

FROM alpine:3.14
RUN apk add --no-cache git
WORKDIR /usr/src
RUN git clone https://repos-r-us.com/some-repo.git
COPY ./copy-json.sh /usr/scripts/
RUN chmod +x /usr/scripts/copy-json.sh
RUN mkdir /usr/websites
RUN /usr/scripts/copy-repo.sh
RUN echo "0 4 * * * /usr/scripts/copy-repo.sh" > /etc/crontabs/root
CMD ["crond", "-f"]

There were several other small optimizations. I minified the CSS with clean-css, a node library for minifying CSS. Most of the clean-css documentation is devoted to writing a JavaScript application that uses clean-css as a library to minify the CSS, but it also provides a second node CLI application, and the same basic functionality can be achieved with a simple bash script. Clean-css has a long list of options and optimizations, but for a simple application, a lot of it feels overkill. This approach should be portable because the dependency is in the Bun application itself and, conveniently, Bun seems to detect that the dependency is in your project and you don’t need to install it globally or give it an absolute path. Additionally, I realized that I did not defer HTMX in the script tag, which was blocking my HTML from loading and other small optimizations.

Moving forward, I’ve considered setting up Watchtower or something similar to manage the image updates, but for now I am happy with how everything works. In terms of content, I am hoping to start writing a lot more posts moving forward. Next up, I am going to do a short post on NUTS, my new C project.

I spent several months paralyzed by indecision before I actually started to write any code. I wanted to take the time to deliberate over which of the various Lisp dialects I would like to try first. I even briefly considered trying out Fennel to do some work on my Neovim configuration, but I didn’t use Neovim enough to find the motivation to try this (though I would be surprised if I didn’t do this eventually). Finally I spent some time looking more seriously into Clojure and realized that it had several features that interested me.

The main thing that stood out to me about Clojure was the runtime, or runetimes. Not only does it run on the ubiquitous Java Virtual Machine, but it can also compile down to the even more ubiquitous JavaScript. In addition to the runtime systems, Clojure is also interoperable with both Java and JavaScript libraries: in short, all of the libraries. Clojure is also a functional style language, which intrigued me. I had been interested for some time to try out a functional language in addition to a Lisp, so I was effectively killing two birds with one stone by trying Clojure. Don’t worry: I am a vegetarian, and no actual birds were harmed in the making of this application.

Once I had decided on Clojure, I started reading Clojure for the Brave and True, which is both hilarious and a great book even it can be a bit dense at times, Web Development with Clojure, which covers building a guestbook application with the Luminus framework, and consuming a healthy diet of Rich Hickey talks on YouTube.

Normally I am an advocate for starting simple, but in the case of Clojure I find the process of building web applications a bit complicated to set up. The code required to set up a web application from scratch using Ring and getting ClojureScript to play nicely and cross compile with Clojure is a bit daunting to a Clojure neophyte, let alone someone relatively new to programming in Lisp in any form. After spending some time with Luminus (and if I am being honest, finding all the boilerplate very confusing), I finally settled on making a single page Reagent application with the Kit framework. I would recommend trying Kit for someone who is interested to try out Clojure web development for the first time. I was able to get working with a simple hello world Reagent app within a few minutes and the project structure and boilerplate code is well laid out and relatively easy to understand. There are comments throughout that basically say “Put code here”.

Deciding on the topic for my first Clojure/Script project was a little like putting on a blindfold and playing darts. I didn’t really care what I made, partly because I knew that my first attempts with the language would probably be relatively naive, yet I wanted to make something. Several months ago I had played with the idea of making a subnetting calculator when I was working through some of the Cisco CCNA course content in Cisco Packet tracer. The way a subnet mask is applied to an IP address to create subnets intrigued me, and I thought that creating an application that displayed both the decimal notation information alongside the actual 32 bit representation of the IP addresses in binary form would be an excellent way to learn more about it and do a bit of an IPv4 deep dive. I did learn a bit more about subnets by doing the project, specifically how CIDR /31 and /32 subnets work alongside with their purpose in a network, so I think the project was well worth the time even for that.

As expected, my implementation of the application was indeed quite naive, which led to a lot of issues throughout the project ( GitHub repo ), but it was quite instructive and will help me to build better Reagent applications in the future.

Despite the mistakes and challenges that I encountered, my overall first impression of both Clojure and Reagent has been very positive. The combination of a functional approach combined with Lisp allows you to make quite elegant and concise functions from fairly simple building blocks. I think this might be what makes Clojure and perhaps Lisp quite daunting to read at times when you are first learning it as a lot can be going on in a fairly small snippet of code. As you start to learn various building blocks of the language and how to effectively define and operate on sequences, you start to find yourself quickly writing more concise and effective code.

Reagent itself is likely to quickly become my favorite way to write React code, at least for personal projects. It’s minimalist approach and simplicity can be deceptive; not only does it seem capable for creating complex UIs, but of doing so in a way that is quite ergonomic. Furthermore, the REPL driven development creates a very effective feedback loop for developing. In fact, I wrote the entire application without even bothering to use the Clojure language server. That’s not to say you should not use the LSP to write Clojure, but the REPL driven development is in itself powerful enough that I didn’t miss having it enough to even bother turning it on or put up with it yelling at me. I probably would use it with a more complex application if only for the benefit of better code navigation.

Despite it’s strengths, I did notice some drawbacks to working with Reagent and ClojureScript. What stood out the most was the relative complexity of a ClojureScript project. your code is compiling both to run on the JVM, but also to JavaScript to run in the browser, There are separate REPLs for Clojure and ClojureScript on account of their differing runtimes and, as I mentioned before, the overall project structure and boilerplate required to make all of this work in concert is significant when compared to setting up an application with, for example, NextJS or SvelteKit. Notwithstanding this complexity, Kit does an excellent job of structuring the project skeleton and setting all of this up so that you can start writing Reagent code with just a few minutes of set up time. Furthermore the benefits outweigh, in my opinion at this time at least, this drawback. As for my opinion of how this scales with more complex applications, only time will tell, but I am sure that I will continue my Clojure journey and find out in time.

I was initially turned off by the simplicity of hyprpaper, but after closer inspection I realized that with a bit of effort and a shell script, it provided me with the tools I needed to configure it to work pretty much however I wanted. I set to work on making this script to control how my wallpapers are loaded. I thought I would share how I did it in the hopes that it could be helpful to someone else doing a similar setup or perhaps someone could recommend some changes to improve it.

# Setup wallaper directory and monitor names.
dir="/home/naokotani/Pictures/wallpapers/"
monitor1="eDP-1"
monitor2="DP-1"

cd $dir

# Get a list of the .png files in the directory.
papers=("${dir}"*.png)

cd /home/naokotani/.config/hypr/

# Get the length of the list of files.
length=${#papers[@]}

This section sets everything up. I set the values for the wallpaper directory and my monitor names. This makes it easy to change later because I just need to change it once at the start of the file. I then cd into the directory and get the list of .png files. hyprpaper only works on .png, so I want to filter out any .jpg or other files that might be left in there so it doesn’t break. I also remove my old hyprpaper.conf because I am going to be generating a new one from scratch. Finally, I get the length of the list of files that I will be using to choose a random wallpaper later.

# Get the first random index.
first=$((1 + $RANDOM % $length - 1))

function dif_random {
    second=$((1 + $RANDOM % $length - 1))

    # Check to see if the second random index is equal
    # to the first, if so, find a new one.
    if [ $1 -eq $second ]
    then
        dif_random "$1"
    else
        return $second
    fi
}

# Make sure that there is more than one file.
# This avoids an infite loop.
if [ $length -gt 1 ]
then
    dif_random "$first"
    second=$?
else
    second=$first
fi

This little section is responsible for selecting a random paper by taking the length and deducting one to select a random index. Then the function dif_random() will get a second random number and check to see if it is the same as the first, if it is, it will run the function again with the first random number as the argument. Once it finds a valid number it returns it. There is no doubt a better way to do this, but this works well enough. Finally I check to make sure that the length is greater than 1 before I actually call the function. If it’s not, it doesn’t call dif_random() because that would end up causing a stack overflow or infinite loop, either way it would break the script.

# Use the random indexes to get the two filenames
paper_one=${papers[$first]}
paper_two=${papers[$second]}

# Check to see if hyprpaper is running
if pgrep -x "hyprpaper" > /dev/null
then
    # If hyprpaper is running, select two new preloaded wallpapers
    hyprctl hyprpaper wallpaper "${monitor1},${paper_one}"
    hyprctl hyprpaper wallpaper "${monitor2},${paper_two}"
else
    # Remove the old conf file.
    rm -f hyprpaper.conf

    # If it's not running, generate a new hyprpaper.conf
    for p in "${papers[@]}";
    do
        echo "preload = ${p}" >> hyprpaper.conf
    done

    echo "wallpaper = ${monitor1},${paper_one}" >>  hyprpaper.conf
    echo "wallpaper = ${monitor2},${paper_two}" >> hyprpaper.conf

    # Start hyprpaper
    hyprpaper &
    disown
fi

Finally, I use my newly created numbers to index into the papers list and grab two random file names. I then check to see if hyprpaper is running, if it is I load the new wallpapers, if it is not I generate a new hyprpaper.conf and start hyprpaper.

I place this script in two places in my hypland.conf file. First I put it as an exec-once command. This way whenever hyprland starts, it will run the script, find that hyprpaper is inevitably not running, generate the hyprpaper.conf, select two random wallpapers and start hyprpaper. If at any point I want to change my wallpaper list, I just need to change the directory in the script, or put new wallpapers in the wallpapers directory, pkill -9 hyprpaper and run the script. This will regenerate a new configuration from the directory and reload hyprpaper. I also bind the script to a keybind so that, whenever I feel the need, I can pick two new random wallpapers.

Finally, you can use cron to have the script run on a schedule once an hour, once a day or whatever you like to periodically change your wallpaper. This shouldn’t have cause any significant performance issues if it comes up in the middle of doing something else because the wallpapers are already pre-loaded into memory.

I initially ran this script on about a dozen very large image files. As is pointed out in the documentation for hyprpaper, this both took a long time to load (I believe it was around 8 seconds), and took a decent chunk of memory. I was able to mitigate this problem a little bit by getting rid of a few of my least favorite wallpapers and then using ImageMagick to reduce the size of the images down to my actual screen resolution by running magick image.png --resize 2560x1440 image.png you can also use this to switch from a different file type to a png, for example: magick image.jpg --resize 2560x1440 image.png reducing the size of the images greatly reduced the time it took to load going down to perhaps a second or less (the initial images were something in the order of 8000x6000, which was quite excessive anyway).

Another small issue with the script is that it will always choose two different images, but one or both of them might be the same as the one that is already loaded. I thought about using the suggested command from the wiki, hyprctl hyprpaper listactive to find the active paper, but for some reason it just wasn’t working. I finally determined that this wasn’t such a big issue. The wallpapers almost instantly, so its not a big deal to simple press the keybind twice if I get a result I don’t like.

(Credit to trev for these observations)

My method for getting a random image involved using a recursive function that keeps getting called until two different random numbers are selected. It’s also possible to use a while loop to accomplish the same thing, but the effective difference between the two is probably pretty trivial. Both would have the limitation of creating an infinite loop if fewer than two filenames are supplied, which requires the check to make sure $length is greater than 1. This solution requires quite a few lines of code and is a bit complex. It also depends on $RANDOM, which is a bash internal. A similar solution could be based on using the shuf coreutil, which would be less complicated, require fewer lines of code and be more readable. If you wanted the script to be fully posix compliant, you could generate a random number yourself in the script instead of relying on $RANDOM.

Lucky for me after my initial investigation, I realized that the Bun team was good enough to add in a simple SQLite driver module to Bun itself (Not to mention I recently stumbled across their nifty shell module). I am a big fan of the inclusion of these features into Bun because, when I am working on a simple prototype application, the last thing I want to do is sift through NPM packages and try to parse through whatever random method was chosen for the documentation; I just want to open up Emacs and start writing some JavaScript. That is precisely what Bun’s SQLite driver allows me to do. You still have the option to use something from NPM, but you don’t have to go searching for a means to use something as simple as a basic SQLite database or a .env file in the early stages of building a project.

Generating a table is dead simple. You just need to import the Bun SQLite module, create a database object and then use the query() and run() methods to create a table.

import { Database } from "bun:sqlite";

const db: Database = new Database(":memory:");

db.query(`
CREATE TABLE books (
id integer primary key,
title text,
author text,
genre text,
price real,
image text,
available boolean
);
`).run()

I put the database in memory because its just a demo, but you could of course supply a file path to the Database constructor as well.

All I had left to do was to parse the CSV file and insert it into the database. I chose to use Papa Parse to parse the CSV file (I chose it, admittedly, somewhat randomly), and it was serviceable, but for some reason it was adding a junk object that only had an BookID key that was empty. Instead of figuring out why, I just filtered it out of the results. If I were doing more than just a demo, I probably would have dug into this rather odd result a bit further, or better yet, it might be just as easy to create a custom function to parse the CSV file manually.

Before I could actually work with the data, I needed to create some types. I try to make a habit of using TypeScript where possible on small projects, so that I am more familiar and practiced with it when I need it for larger projects. First I created a BookCsv type that would hold the result form Papa parse and a Book type that represented the actual database schema. I created the extra Book type because Papa parse wanted the object’s keys to match the field names. I figured it was easiest to just create two types, so that I could make sure that the object keys followed js naming conventions.

import { Database } from "bun:sqlite";
import Papa, { type ParseResult } from 'papaparse';
import type { Book, BookCsv } from './types.ts'

const db: Database = new Database(":memory:");

const csv = await Bun.file("data/booksdata.csv").text();
const books: ParseResult<BookCsv> = Papa.parse(csv, { header: true });

books.data.forEach(book => {
    if (book.Title) {
        const insert: Book = {
            id: book.BookID,
            title: book.Title,
            author: book.Author,
            genre: book.Genre,
            price: book.Price,
            image: book.ImageFileName,
            available: book.Available,
        }
        db.query(`
INSERT INTO books (title, author, genre, price, image, available)
VALUES ($title, $author, $genre, $price, $image, $available);
`).values({
    $title: insert.title,
    $author: insert.author,
    $genre: insert.genre,
    $price: insert.price,
    $image: insert.image,
    $available: insert.available,
});
    }
});

I use the Book type to insert into the database and then when I need to get the data out I can use that same type with a select statement. I used the object method for inserting the data, but you can also use the question mark style and then supply an array of values that match the question marks.

The amount of code and effort it would have required to use node for this would have been virtually the same, but I like not always have to make decisions, and Bun is fast becoming my go to js runtime for these kinds of projects for that reason alone. You can access the wealth of NPM packages if you need them, but very often I find myself installing the same packages over and over just to perform relatively simple tasks, and in those cases having something that is a bit more opinionated and has some of these quality of life modules is exactly what I am looking for.

The web application I am designing is intended to host different types of assets for table-top role-playing games such as books, maps, music and so on. I want to be able to represent different types of ownership in this schema. A user can be both an owner of assets and a creator of assets, but I do not want the information related to creators in the user table because most users will probably not also be creators. A simple boolean will not suffice because there would be no way to store creator-specific information. I can use the user.id in a creator table to represent an optional one-to-one relationship. A user is either a creator, or they are not, and if they are they can and should only be a creator once. Therefore the user.id foreign key can act as the primary key in the creator table. Each asset must have one and only one creator who is the sole source of ownership for the asset. This creator.id can then be stored in the asset table as a foreign key in a one-to-many relationship to represent creator ownership. User ownership, an individual who has purchased an asset, is represented in a separate linking table that can represent a single user owning many assets. This schema will allow for a single user to have multiple forms of ownership, a many-to-many relationship for purchased assets, and a one-to-many relationship for created assets, while keeping the data normalized. The below markup will create these tables in a PlantUML diagram:

' hide the spot
' hide circle

' avoid problems with angled crows feet
skinparam linetype ortho

entity user {
    * id <<PK>>
    --
    * username
    * email
}

entity creator {
    * id <<PK, FK>>
    --
    creator_name
}

entity user_asset {
    * user_id <<FK, PK>>
    * asset_id <<FK, PK>>
}

entity asset {
    * id <<PK>>
    * creator_id <<FK>>
    --
    * title
    * summary
    * image
}

This creates the four tables, but does nothing to link them up. The few lines at the top are used to correctly display an ERD diagram as it differs slightly from the default UML class implementation. The * creates a NOT NULL field and the -- is designed to indicate which of the fields are identifiers.

PlantUML uses a simple syntax in order to represent linkages. It is important to note that it is simply a drawing tool and will not prevent you from creating invalid or incorrect linkages between tables.

user     ||--o|  creator       : is
creator  ||--o{  asset         : creates
asset    ||--o{  user_asset    : is purchased in
user     ||-down--o{  user_asset    : purchases

Adding these lines below the table definitions will create linkages between the tables using a visually representative syntax that is very easy to remember. For example |o--|{ represents an optional one to mandatory many relationship and ||--|| would represent a mandatory one-to-one relationship. Detailed examples for all the possible combinations can be found in the ERD class documentation.

I didn’t exactly make a UML diagram because Rust uses structs and traits instead of classes and interfaces, but I created a diagram that was inspired by UML. To do this I had to add some custom symbols to correctly display Rust data structures like traits and enums. I started by adding classes for each table and then created structs that would work well with Diesel. Diesel is an ORM that uses Rust structs along with macros to dynamically create SQL queries that are safer from SQL injection or other attacks. It also speeds up development by creating a schema of your database in Rust so that you will get compile time and language server errors if you try to do something incompatible with your schema.

struct User {
        + id: i32
        + username: String
        + email: String
        + create(user_id: i32) Self
        + read(user_id: i32) Self
        + update(user: UserNew) usize
        + destroy(user_id: i32) usize
}

struct UserNew {
        + username: String
        + email: String
        + logo: String
        + create(user_id: i32) Self
}

struct Creator {
        + id: i32
        + first_name:  String
        + last_name: String
        + read(creator_id: i32) Self
        + update(user: UserNew) usize
        + destroy(user_id: i32) usize
}

struct CreatorNew {
        + first_name:  String
        + last_name: String
        + create(user_id: i32) Self
}

class AssetTrait << (T,#FF7700) >>  {
        read(asset_id: i32) Self
        update(self, conn) usize
        destroy(asset_id: i32) i32
        paginate(user_id) Page
}

struct Page {
        + thumb: String
        + display_name:String
        + main_image: String
}

struct Asset {
        + id: i32
        + creator_id: i32
        + title: String
        + summary: String
}

UserNew o-left- User
CreatorNew o-right- Creator
User *-- Creator
Creator o-- Asset
Asset <|.. AssetTrait
AssetTrait <|.. Page

Similar to the ERD diagram, you create classes, but instead of using * you would use + to show public methods and variables. In order to display the Asset trait I used the << (T,#FF7700) >> syntax, which allows you to control the icon and the icon color for any arbitrary data structure.

Bun is a really interesting alternative to Node. It comes with a really nice Bun.serve() method for receiving and sending HTTP requests and responses so its very easy to just make a nice REST api right out of the box. It also comes with the ability to read .env files into process.env, which is a nice convenience. It also has Bun.file(), which is a replacement for Nodes fs.readFile(), that can read relative paths without the need to join a relative path with __dirname. A lot of these things come down to convenience replacements for common tasks, but the Bun website makes some claims about being super fast. Without doing a lot more research, I will take that with a grain of salt for now.

Similar to my other project I use the following function to get my data from Emacs and my local machine to the remote server.

(defun nao/export-org-to-blog ()
  (interactive)
  (setq org-html-doctype "html5")
  (let ((filename (org-html-export-to-html nil nil nil t)))
    (copy-file (concat (file-name-sans-extension filename) ".png")
               "/rsync:server:/path/to/images" t)
    (copy-file filename "/rsync:server/path/to/html" t)))

Org mode also comes with a series of publish functions that, I am assuming, accomplish a similar task. This above function did almost exactly what I needed, so I haven’t looked into the org publish functionality, but perhaps I will look into switching to that in the future if the website becomes more complicated.

On the server I use a very simple REST api to direct traffic to resources by making use of Bun.serve():

try {
    if (/\.(css|png|svg)$/.test(req.url)) return await getStatic(req.url);
    if (url.pathname === "/") return await getHome();
    if (url.pathname === "/blog") return getBlogs();
    if (url.pathname === "/cards") return getCards();
    if (/\/blog\/.*\.html/.test(req.url)) {
        const match = req.url.match(/\/blog\/(.*\.html)/);
        return getPost(
            match![1],
            req.headers.get("hx-request") === "true"
        );
    }
    return new Response("404!");
} catch (e) {
    console.error('Error:', e);
    return new Response("500 - Internal Server Error");
}

I took the basic structure from the Bun documentation and I use Regex tests to match routes. The first route matches routes with css or image filenames to server those images as they are requested and the other routes direct to modules where I use Mustache templates to create the response for the client. req.headers.get("hx-request") === "true" being based to getPost() tests to see if a request has the hx-request header. I do this because I use HTMX to avoid page reloads, but I need a way to determine if a resource was requested from outside the page. If the header is not present, I know the user navigated from an external link so it knows whether or not it needs to serve the index.html.

if (hx) {
    return new Response(post);
} else {
    const view = {
        API_URL: API_URL,
        post: post,
    }

    const html = Mustache.render(index, view);

    return new Response(html, {
        headers: {
            "Content-Type": "text/html",
        },
    });
}

inside getPost() if hx-request is true, I send just the HTML for the post, if its false, then I use Mustache to add the post to the index.html and send it as one HTML file.

The module that routes to the static file resources users regex similar to the main REST api:

if (/\w+\.css$/.test(filename)) {
    try {
        const file = Bun.file(`static/css/${filename}`);
        return new Response(file);
    } catch (e) {
        console.error(e);
        return new Response("404: file not found");
    }
}

String.match() extracts the filename from the url and then I use regex to map the file extensions to the appropriate directories and then return the requested file. The most likely reason for an error is that the file does not exist, but regardless of the error, I log the error on the server and respond with 404.

Initially to create the images for the cards I was simply taking screenshots and then uploading the results as is. This meant that the files were much larger than they needed to be and slow to load. Furthermore, they were inconsistent in terms of size and aspect ratio making the cards inconsistently sized. In order to solve this problem, I installed sharp in order to transform the images, but I wanted a way to automate it.

I created a simple module to achieve this. First I took the example code from the Bun documentation to watch my images directory for changes. When I use rsync to upload a new file the watch will trigger and provide me with the filename that was changed. I also add some logic to make sure that the file has appropriate extensions, in this case I check for png and jpg, and isn’t a thumbnail itself.

Next I check to see if I have access to the file using fs.access(). If an image is moved from the directory, then sharp might try to transform an image that no longer exists, causing an error.

async function checkAccess(filename: string) {
    try {
        await access(`${imagesPath}/${filename}`, constants.R_OK | constants.W_OK);
        resizeImage(filename);
    } catch {
        return;
    }
}

Finally after everything has been appropriately validated, I check to see if there is already a thumbnail generated for that image, and if there is, I delete it so that, if the image has changed, the thumbnail will be updated appropriately. In theory this might cause an issue where, if the thumbnail is requested at the exact moment that I am remaking it, it would not be available. Perhaps a solution to this would be to create a separate staging directory and sync the images after they are updated. I may do this in the future if I create module for updating the post HTML to semantic HTML5.

I then run sharp().resize() with first two arguments being the width and height of the new image in pixels. In the options argument, I set fit to “cover”, which has a similar effect to a CSS overflow hidden, but it will actually crop the image. This might cause it to be cropped in an undesirable way, but I think the best solution here is to crop the image properly before I upload it.

async function resizeImage(filename: string) {
    const images = await fs.readdir(imagesPath);
    if (images.includes(`thumb-${filename}`)) {
        try {
            await fs.rm(`${imagesPath}/thumb-${filename}`);
            console.log(`Deleted old thumb for ${filename}`);
        } catch (e) {
            console.error("Error deleting old thumb: " + e);
        }
    }

    try {
        console.log(`Processing image ${filename}`);
        sharp(`${imagesPath}/${filename}`)
            .resize(384, 185, {
                fit: "cover",
            })
            .toFile(`${imagesPath}/thumb-${filename}`);
    } catch (e) {
        console.error("File processing failed" + e);
        return;
    }
    console.log(`${filename} resized successfully`);
}

This project required a bit of extra attention to testing and validation to ensure that there were as few data inconsistencies entered into the database as possible. I was working with unsanitized data directly from the web that was additionally entered in a very inconsistent matter. It would appear that CSS classes were entered manually and inconsistently such that it was more difficult to correctly target the correct elements in a consistent way. I approached validation in two steps.

After scraping the traits from the web, I first ensured that the trait strings were properly trimmed:

  let mut clean_traits: Vec<String> = Vec::new();
  for t in traits {
      let anchor = Html::parse_fragment(&t)
          .select(&anchor_selector)
          .map(|x| x.inner_html())
          .next();
      if let Some(s) = anchor {
          clean_traits.push(s.trim().to_string());
      } else {
          clean_traits.push(t.trim().to_string());
      }
  }

In some cases the traits contained anchor tags, while in others they didn’t. I removed the anchor tags by retrieving the inner HTML in the cases where the string contained an anchor tag. As this function returned an Option<String>, I used an if let to check if the anchor was Some or None and then trimmed the remaining string.

Next I checked to see if I had successfully retrieved data for each trait in the struct. In some cases the size trait was contained in a <span class="size"> with a class of size to correctly color it, and in others it was in a <span> with no class. To handle these cases I first checked if the scraper had returned Some(String) and in that case, extracted the string from the Option<String> in the else of the if let statement I looped over the traits and used and used a match to see if any of the traits matched a valid size.

  let size: String = if let Some(s) = size_base {
      s
  } else {
      let mut size_string = String::from("NO SIZE");
      let mut new_traits: Vec<String> = Vec::new();
      for t in traits {
          match t.as_str() {
              "Tiny" => size_string = t,
              "Small" => size_string = t,
              "Medium" => size_string = t,
              "Large" => size_string = t,
              "Huge" => size_string = t,
              "Gargantuan" => size_string = t,
              _ => new_traits.push(t),
          };
      }
      traits = new_traits;
      size_string
  };

The minor mode is centered around three interactive functions. The first, emmet-react-expand-component expands a shorthand version of a of a component with props that are separated by periods. If you write the shorthand starting with the component name and then separate the props with periods in the follow form Foo.bar.baz, it will expand to <Foo bar='' baz='' /> and move the cursor between the first single quotes. If you want to use {} or "", you can set emmet-react-prop-pair to any pair of characters you want to wrap with.

(defun emmet-react-expand-component ()
  (interactive)
  (message (string (region-end)))
  (let ((component)
        (propList))
    (setq propList (emmet-react-get-component))
    (setq component (pop propList))
    (emmet-react-insert-component propList component)))

emmet-react-wrap-component is useful for situations where you want to wrap Svelte or JSX in a component. You will be prompted to enter the component in the echo area. If you want to add props to the component, you can separate them with periods and they will be automatically separated. For example Foo.bar.baz will be expanded to <Foo bar='' baz''></Foo> and the cursor will be moved between the quotes after baz.

(defun emmet-react-wrap-component ()
  (interactive)
  (let ((str (read-string "Enter component: "))
        (propList)
        (body (buffer-substring (region-end) (region-beginning)))
        (component)
        (propPos))
    (setq propList (split-string str "\\."))
    (setq component (pop propList))
    (delete-region (region-end) (region-beginning))
    (insert (format "<%s" component))
    (js-indent-line)
    (while propList
      (insert (format " %s=%s"
                      (pop propList)
                      emmet-react-prop-pair))
      (if (not propPos)
          (setq propPos (- (point) 1))))
    (insert ">\n")
    (insert body)
    (insert (format "</%s>\n" component))
    (if emmet-react-auto-indent
        (web-mode-buffer-indent))
    (cond ((eq evil-state 'visual) (evil-insert-state)))
    (if propPos
        (goto-char propPos))))

If you want to generate a wrapping style JSX component with props you can use emmet-react-expand-wrap-component. This function is not used to wrap highlighted text, it simply generates the component but with an opening and closing tag. For example Foo.bar.baz will expand to <Foo bar='' baz=''></Foo>.

(defun emmet-react-expand-wrap-component ()
  (interactive)
  (let ((component)
        (propList))
    (setq propList (emmet-react-get-component))
    (setq component (pop propList))
    (emmet-react-insert-wrap-component propList component)))

There are a few things I would like to add to this project over time. I would like the functions to be smart enough to recognize when the user has set the prop to a specific value. I think the easiest way to do this will be to continue to split the strings on the periods, but to later also split them by = and wrap what comes after the equals between the selected characters. I think it would also be nice to have a variable to determine where the cursor should be moved after expanding the component.

The following in this blog all remains the case for the current production version of the blog, but I have recently started a new version using Bun in Place of Node and I am using TypeScript in place of vanilla JS. Instead of using Express, I am creating a new API from scratch with the built in Bun.serve(). I also decided to replace squirrelly templates with mustache.js templates to try a different approach to HTML templates. For now I think I will stick with using Cheerio to parse the HTML for data, but perhaps I will switch that out as well as the project moves forward. I am also adding some better logic for dealing with images so I can handle different file formats. So far I am happy with this new implementation. Using Bun’s built in method for sending and receiving HTTP requests is much more in keeping with the overall theme of simplicity I am trying to go for with this project. I will probably switch to this newer version when its done.

I played around with a few ideas of how I would make this blog, but finally settled on a workflow where I could upload my org files to a remote server using a simple Node app, HTMX, and functionality that comes built into Emacs. I organize my notes with Denote.el, so I settled on a setup where I didn’t need to leave my editor to update my blog. I will be much more liable to keep my projects blog up to date if I have an easily accessible workflow. Here is a basic outline of the steps I took to get this up and running excluding the setup of the sever itself (I used Digital Ocean and Nginx, but any VPS will do fine). The full code is here on my GitHub account.

First I save the file to a blog directory on my local file system and use rsync with OK-IF-ALREADY-EXISTS set to t in order to update the blog post on my remote file system. I just export the body without any head section or doctype. If this is a new blog, I put a thumbnail for it in the blog/images directory which will be updated when I run the function.

(defun nao/export-org-to-blog ()
  (interactive)
  (setq org-html-doctype "html5")
  (let ((filename (org-html-export-to-html nil nil nil t)))
    (copy-file (concat (file-name-sans-extension filename) ".png")
               "/rsync:server:/path/to/images" t)
    (copy-file filename "/rsync:server/path/to/html" t)))

In order to get the file to save with the correct name, I manually set the file name using EXPORT_FILE_NAME like so #+EXPORT_FILE_NAME: ~/Documents/denote/blog/programming-blog.html. That’s it on the Emacs side, not much to it.

Server side I use a simple Node app that uses Express to create a REST API that grabs the filenames, which are already in slug format, to create the list of links for the blog posts. The if/blocks are used to either make the response the full page if they navigate form outside the website, or just the content if they use internal navigation.

app.get('/blog/*', async function (req, res) {
    const slug = req.params[0];
    if (req.headers["hx-request"] === "true") {
        const path = '/path/to/blog/'
        const data = await fs.readFile(path+slug, 'utf8')
        res.send(data);
    } else {
        const route = [slug]

        const data = {
            route: route,
        }

        const html = Sqrl.render(newRoute(), data);
        res.send(html);
    }
})

I use the squirrel templates to create the different content such as the lists of links and cards. For example, you can set up links like this:

    <nav>
    <h2 class="post-header">Projects</h2>
    <ul>
    {{@each(it.files) => val, index}}
    <li>
    <a href="#" hx-push-url="true" hx-get="a-domain.com/blog/{{val}}" hx-target="#contentDiv" hx-swap="innerHTML">
    {{it.labels[index]}}
</a>
    </li>
    {{/each}}
        </ul>
      </nav>

This will create a list of links from the filenames and then when the links are clicked, they will swap with the content that is currently in the #contentDiv.

In order to get a summary for each file I add a <p> tag with the summary class at the top of my org file and then I use cheerio to parse the html and pull out the summary text. I then just use a CSS display: none; so the summary doesn’t appear in the actual blog post. To get the image I simply split the filename and replace .html with .png. If you wanted to use multiple file types it would be simple enough to create a more elaborate solution, but my screenshot app defaults to png, which is acceptable for the purposes of a simple blog.

#+BEGIN_summary
Using Emacs as a CMS with a simple Node application using HTML templates and HTMX
#+END_summary

const files = await fs.readdir(path);

for (file of files) {
    const img = file.split(".");
    const data = await fs.readFile(path+file, 'utf8');
    links.push(file);
    titles.push(makeTitle(file));
    images.push(img[0] + ".png")
    const $ = cheerio.load(data);
    summaries.push($('div.summary p').text());
}

The major issue with this workflow is that I don’t have a lot of control the HTML that org export creates. It is possible to customize it further, but it requires quite a bit of customization. As is stands you would most likely need to write custom CSS to target the classes that org export creates, which are named well and extensive enough to do what you need. If you want to make use of bootstrap or another CSS framework, there are probably much easier ways to produce a blog than this, but if you don’t mind writing CSS, its a pretty simple elegant solution.

Aside from tweaking the CSS to get it work better on different screen sizes and improve the overall look and feel, I would like to add some logic to add a default thumbnail if I don’t specific one by putting it in the blog directory. As the blog grows I might also try to implement a directory structure that would be reflected on the UI by nesting the posts under categories, JavaScript, Rust, Ideas, etc.

Class is the admin only command. It allows users to create, add, remove and change the status of classes by using interactive commands. Because the tasks tables have a foreign key relationship dependency on the class table a class can only be removed if all of the tasks for that class have been removed. To solve this problem I decided to make a column in the class table that determines whether or not the class is “active” or “inactive” and if the class inactive, then no tasks will appear for that class when a user shows tasks. Once all tasks that use the classes primary key as their foreign key are removed, you are free to remove the class. By default, classes are set to be active when they are created.

The task command is how users create the tasks themselves. Using the task gives the user a prompt with select style menus with available classes and provides fields to add the date for the task. After entering this data a modal allows for confirmation of the task and gives an opportunity enter additional longer form information and set the time.

The next and show commands are how users display the tasks. I decided to make these commands ephemeral to reduce spam in the server, while the task command is not ephemeral to make it less likely for people to try to enter the same task multiple times.

I decided to use SQLite for this project mainly for the sake of simplicity. For personal use it would just as easy to use, for example, a PostgreSQL, but if I wanted to make the bot available for others to use SQLite has the advantage of being much easier for users to install use as it can set itself up without any need for extra input from the user. In this case simplicity seemed more important than scalability or efficiency of the database.

I designed this bot so that it could be made available for others to use, but as it stands it is not robust enough to be made available for others to use.

I had initially considered using TypeScript for this project because I knew that I wanted a type system. Calculating the budget requires enough calculations that I knew that using vanilla JavaScript would create major headaches and be prone to bugs. I ended up settling on Rust because I thought that it would be able to handle manipulating the data structures more efficiently and make the app feel snappier. I don’t know what kind of performance gain, if any, this actually gives me and benchmarking it would require me to rewrite large portions of the API in JavaScript.

While my initial decision was driven by pure speculation about Rust vs. JavaScript performance, using Rust had other advantages that I did not anticipate.

In order to test the app, I am using three different overall groups of tests. First, I am running smaller scale unit tests on functions, second an integration test that tested the back-end as a whole, but bypassing the network, and finally I wrote a separate test app with Node, Puppeteer and Mocha. The latter two tests ended up being the most helpful in finding bugs so far.

My strategy for the integration test was to begin and end with the two structs that implemented the serde json #[derive(Deserialize)] and #[derive(Serialize)] macros that deserialized the URL query string coming in from the client and then serialized the json response back to the client. The goal was the isolate just the back-end logic, ensure that it was correct so that I could rule that out when doing the end to end tests that tested the entire route from client, server, database, server and then back to the client. Any remaining issues then should lay with either the network or the front-end. This test aims to iterate through as many data combinations as possible and isolated some bugs that cropped up in edge cases.

Once I had isolated bugs with the server side integration test, I used puppeteer to enter similar data to what I had hard coded into the tests on the server side in the integration test. I then captured the json with puppeteer:

  await page.waitForResponse(async response => {
    return (await response.text()).startsWith('{"id"');
  });

{"id" being the first part of the json response. Once I had the json I used the mocha library to test the data.

I ran into a rather frustrating issue with puppeteer that was a valuable lesson. I kept getting incorrect data back from the server when running the Node end to end tests, but after some time I realized that, after I loaded the page, the select input that I was trying to manipulate did not exist yet. It wasn’t causing an error, but the form was not being submitted correctly. I used await page.waitForNetworkIdle(); in order to ensure that the page was fully loaded and then submitted the form data and the tests passed as expected.