Getting an app to redeploy as fixes and upgrades are uploaded to a linux server

For most webapps I oversee it is desirable to have a mechanism in place not only for having them start when the server reboots, but also to restart the app when I need to deploy an update or (most recently) a fix to the application.¹ This process requires sudo access² and five files for a full redeployment system³:

1. Base .service file which simply runs the app
2. Watcher .service file which knows how to restart the app
3. Watcher .path file which will trigger the restart if the on-disk target changes
4. on-server .sh file for archiving and replacing our jar files
5. Local .sh file that builds our war, uploads, and launches the shell script above

Server-side Directory Structure

The following is the app-specific directory structure that is to be referenced throughout this process.

root@server:/srv/webapps/myapp# tree
.
├── archives
│   ├── myapp.jar.2022.008.11.17:51:19
│   ├── myapp.jar.2022.010.22.16:48:50
│   ├── myapp.jar.2022.011.10.15:20:37
│   ├── myapp.jar.2022.011.11.09:55:41
│   ├── myapp.jar.2022.011.11.11:17:59
│   └── myapp.jar.2022.011.11.11:30:04
├── deploy.sh
├── docket
├── myapp.jar
├── index.html
└── log

3 directories, 10 files

note the docket is emptied as the process moves the former myapp.jar into the archives and moves the file in docket up a directory to myapp.jar.

Server-side setup

Base .Service file

This file, myapp.service, simply does the job of launching⁴ my app. My app is a Java uberjar .jar file⁵. Remember once this file is in place to coll systemctl enable myapp to cause this to run at boot. If the program were not receiving updates, that would be sufficient and this post could end here.

[Unit]
Description=My great app

[Service]
Environment=MYAPP_PORT=3010
WorkingDirectory=/srv/webapps/myapp
ExecStart=/usr/bin/java -Xms128m -Xmx256m -jar myapp.jar -p ${MYAPP_PORT}
User=jvmapps
Type=simple
Restart=on-failure
RestartSec=10

[Install]
 WantedBy=multi-user.target

Watcher .service file

This file, myapp-watcher.service, does the job of restarting the service we created above. Eventually it will be triggered by a target file change (next section). I’m not sure if it was necessary, but I used systemctl enable myapp-watcher.service to ensure it’s available.

[Unit]
Description=Restarts myapp on upload
After=network.target

[Service]
ExecStart= /usr/bin/env systemctl restart myapp.service

[Install]
 WantedBy=multi-user.target

Final key, watcher .path file

This file, myapp-watcher.path, is the final server-side key to the process. It watches for my jar file to be replaced (or touched, etc) and kicks off the above service file when it occurs. I used systemctl enable myapp-watcher.path to make sure it’s always on.

[Unit]
 Wants=myapp-watcher.service

 [Path]
 PathChanged=/srv/webapps/myapp/myapp.jar

 [Install]
 WantedBy=multi-user.target

Supplemental (not SystemD): remote deploy.sh

The following file, deploy.sh, exists in the deployment directory (the place in which the .jar file, logs, etc are located on the server). It will be kicked off remotely after a new uberjar is scp’d into the docket directory, and serves to archive the present jar (which is probably still running) into the archives directory, then move the new file from the docket into its place, which change kicks off all the systemd stuff above.

#!/usr/bin/env bash
deployment_path='/srv/webapps/myapp';
date=$(date +%Y.0%m.%d.%T);
filename="myapp.jar";
archive_filename="$filename.$date";
deployment_file="$deployment_path/$filename";
docket_file="$deployment_path/docket/$filename";
if test -f "$docket_file"; then 
   # Archive existing thing
   if test -f "$deployment_file"; then
       mv "$deployment_file" "$deployment_path/archives/$archive_filename" &&
           echo "File archived: $archive_filename"
   else
       echo "No file to archive."
   fi
   # Deploy thing
   mv "$docket_file" "$deployment_file"
else
    echo "No file to deploy at $docket_file"
    exit
fi

echo "deployment archived and repositioned for service-watcher.path to [re]deploy";

Really final step: local project deployment

Once the server is ready for me, the following script is run from within my project to get the file positioned. This is the only file which is in version control for the project.

#!/usr/bin/env bash
lein clean
lein uberjar
scp ./target/myapp.jar myserver:/srv/webapps/myapp/docket/
echo "------------------------------"
echo "build and deploy to humforms docket complete"
ssh -t forms "/srv/webapps/myapp/deploy.sh" &&
    echo "Service archived seated for launch by watcher"
exit 0

Conclusion

I am always up for improvements; I am a full-stack dev who also does devops in a pinch. Any improvements or innacuracies in this, please let me know!

Footnotes

¹ in the past we used WildFly and Clojure Immutant to simplify this process; it only needed to put a .war file into a directory and wildfly took care of the rest of redeployment. However, Immutant was deprecated by RedHat years ago and is broken now (in small and big ways), so good ‘ol uberjars are the answer I choose. They only require the server has Java.

² Sudo access is for the creation of the three systemd files, and might me needed for ensuring all permissions are as expected. But it won’t be needed for deployment once this setup is complete.

³ This works for systems using SystemD, which includes Ubuntu and SUSE servers.

⁴ note that it hos a user of JVMApps; this user was created on my system to own processes like this, per an earlier effort that only set up a startup process at https://tech.toryanderson.com/2020/09/04/clojure-app-setup-for-auto-deploy-with-raw-systemd/

⁵ My app is actually a Clojure one, created by lein uberjar, but there is no Clojure code in this example.