How to deploy python microservices app updates?

Question

I'm working on python microservices based app and I'm wondering how to ship updates.

Let's start by a summary of what I have:

• Code base on git (several repos)
• The app is running on several remote raspberry pis
• I access these via vpn network
• Raspberry pis have a limited access to internet via GSM MtoM network
• The main custom lib is installed with pip
• Dependencies are installed at the first launch in house with good internet connection
• The code of each service is ofuscated in place At the moment when I update my services I wrote a shell script which goes and download latest version for each service. It feels like it's such a repetitive task for each device.

I only see 2 simple options:

• connect to each device via vpn + ssh and run the update script (even an enhanced version)
• At the end of the CI run a script that send latest.zip and create a service (script + cron ? python + systemd ) on each device that check for new .zip to update the services. Each of those options doesn't look pretty clean.

What is usualy done in similar cases ?

Do you have any idea to share with me I don't find anything that suits my needs.

Thanks for your inputs !

Answer 1

Device Management is quite hard to achieve so we've seen the rise of many SaaS, from big providers and small startup, which automate the deploy of edge services onto IoT devices fleet. Some of them rely on containerization (and Docker is pushing towards a top level support on ARM archs) some other else act in a "serverless fashion" which means that let you load some script in some language and then copy it through your fleet

Basically what you can do is

1. Evaluate these tools (eg. Azure IoT Edge)
2. Work With some configuration management tool (eg. Saltstack)
3. Roll you own solution

Evaluate Edge Computing Tools

It's clear that this is the safest choice since you have to do nothing but some benchmark and then integrate your pipeline. But as with all cloud services, they come with their costs and their constraint

Work With some configuration management tool

Yes, I'm not crazy. We know config management tools (Ansible, Terraform etc) since we use them to provision hundreds of cloud VMs, but there is not so much difference between a cloud VM and a linux device accessible through SSH and VPN. You just have to make sure that you are using a tool that is scalable enough and has the needed resiliency to work over unreliable channels. Saltstack dose this job quite good, it uses ZeroMQ as event bus and have small footprint agents. You can define your desired state through versioned files and change them accordingly to requirements, or take control of some devices for some specific maintenance tasks. Pay attention on managing all Ops aspects (security, scalability, availability) that are the major burden that this solution carries to you project

Roll your own solution

If you have a very simple use case, you wouldn't be eager to pay cloud bills or to manage large scale configuration application for High Avaliability and so on.... You are able to communicate with your devices in a bidirectional way, you could write some platform service able to send events to the edge whenever a config update is available. Then the edge send back some tracking event to understand whether you should retry on some unavaliable device, rollback the deployment or perform some deployment strategy such as canary. But this worth only with the simplest scenario, because building a full fledged management solution takes a huge effort and distract your team for the real valuable activities