This week, I was re-deploying hyperlog/hyperlog after about 4 months and I had forgotten close to everything about the deployment. Last time around, I had used a similar stack except for Capistrano which I am glad I learned this time around.

There are a few things you should know about the choice of my stack - why Puma, nginx, Capistrano (and systemd?). While Puma webserver ships by default when we run rails new, Phusion Passenger is a very popular choice as well. I saw a forum post, where Chris Oliver, the creator of GoRails explained that Phusion Passenger is easier to get started with. This could be true, but I had already deployed Hyperlog with Puma in the last deploy so I decided to stick with it.

Why Nginx? Caddy was a good option, but I had used nginx before and I was learning more about it. But Caddy would've been a solid and quick-to-implement choice as well.

Why Capistrano? I wanted to make the deployments as reproducible as possible. I wished dearly this time that I had something that will deploy everything as it was before, but I hadn't thought of any such thing. This time around, I made sure to keep my service files and nginx configs to push them to version control. At all times during the deployment I tried to make sure that my local files were in sync with the remote files.

Why systemd? This was a no-brainer. It comes with most linux distributions and uses a syntax that I'm familiar with.

Now the fun part:

1. Get your proxy headers right with Nginx

Getting SSL to work was the most problematic part of the problem. Part of the reason for this was the infamous magic that Rails does behind the scenes. For example, with the config.force_ssl setting, and secure authenticity tokens that are automatically added by Rails to all cookies as a security measure. I got errors for all of these. After trying multiple times, and fixing bugs with solutions that created different bugs, I landed on a single line of change that worked:

Here's a part of my initial Nginx configuration:

listen 443 ssl;
...
try_files $uri $uri/index.html @app;
location @app {
    proxy_pass http://puma;
    proxy_set_header Host $host;
}

As you see, while listening on port 443, nginx still needs to proxy to http://puma. Puma/Rails both are not aware that the request is indeed secure. To work around this, an X-Forwarded-Proto header needs to be added that specifies the protocol used by client to communicate to the server.

location @app {
    proxy_pass http://puma;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;

This ended up working for me, and fixing a lot of my problems.

2. Rails SSL can work over both network ports and Unix sockets

First, let me tell you the reason I thought it may not work over network ports. Because, every documentation/tutorial on Puma out there will be using Unix sockets. And, it's natural for one to think that if nginx is proxying a request to http://, then Rails will read the URL as not being SSL enabled and that somehow we could fake it for Unix sockets.

That is, in hindsight, silly. I tested that SSL definitely works for both network ports and Unix sockets. It makes no difference at all in terms of what you can do with both.

3. rsync is incredibly handy

I had never used rsync until very recently when I read Kailash Nadh's blog post on the software ecosystem - nadh.in/blog/javascript-ecosystem-software-...

I read about it and my curious ass was all over it in the next 10 minutes. I was rsyncing left and right, all the time. Previously, whenever I've needed to put something on a VM, I've used git push from local and git pull on the remote. It is much more tedious than the very elegant rsync.

I used rsync a lot this time. I even used it to deploy one of the sidecar services (not a k8s deployment, but just using the concept), just rsync-ed the whole thing to the VM.

4. Capistrano makes life a lot easier

I was originally planning to use Ansible, but then turned to Capistrano because it had a lot of official and community support for Rails. This is the first time I've used a proper 3rd-party tool to automate deployment of repositories to a VM from the local environment itself. This is while I have used (and configured) CI/CD pipelines, release scripts that deploy container images to a container registry and update the image version via the k8s API. Reminds me again of knadh's blog.

That's about it. My confession of the things I did not know yet but I was moved by. All in all, I would say that my deployment experience wasn't great. A lot of the times, I didn't find the relevant documentation for the errors I faced and I had to rely on hunches. I feel this was primarily due to the Nginx + Puma configuration, and the lack of documentation for it.