Building a .deb file for BirdNET-Pi: questions & discussion

[saintaardvark]

Hi @mcguirepr89 -- I'm finally getting a chance to dig into how one might build a .deb file for BirdNET-Pi. Right now I'm considering this a spike/research; I'm working on a fork to see just how much work it would take to get a .deb going, how feasible it is, and so on After that, if things look reasonable, I'll figure out how to start submitting PRs.

Given that, there are a few questions I have so far:

1. In scripts/install_services.sh, the systemd unit files are being templated out and then symlinked into place. Are you amenable to breaking those out to template files within the repo (eg: systemd_templates/birdnet_recording.service), then using install_services.sh to copy them into place (modified as needed)? This approach seems to be a better fit for .deb files.
2. More generally, are you amenable to rearranging the directory structure of the repo? I'm not sure that I have anything in particular in mind right now; I'm more trying to figure out what you're open to.
3. Are there any thoughts yet about how or where a .deb file would be hosted? I can do research here, but just curious if you had something in mind.
4. One of the things I want to ensure is that everything works well with any package that gets produced. I'm thinking here of both from-scratch installations and upgrades. Is there anything like an integration test (maybe a script, maybe just a checklist) that's used to verify functionality right now?

I'm sure I'll have more questions, but this ought to be a good start. 😀

[mcguirepr89]

It sounds like you're on the right path so far!

1. In scripts/install_services.sh, the systemd unit files are being templated out and then symlinked into place. Are you amenable to breaking those out to template files within the repo (eg: systemd_templates/birdnet_recording.service), then using install_services.sh to copy them into place (modified as needed)? This approach seems to be a better fit for .deb files.

Definitely -- the way the templates are created now (I think) mirrors how a .deb gets installed, using templates and then inserting the user environment where needed, but I could be wrong about that. Either way, the templates are only made dynamically now to insert those user environmental variables and the settings in birdnet.conf for things like $RECS_DIR, $EXTRACTED, and $PROCESSED, etc. I imagine the .deb packaging can handle a basic default birdnet.conf during installation that the user configures to their needs after installation.

Note: It's not well documented, but if you just

git clone https://github.com/mcguirepr89/BirdNET-Pi.git

and then copy the included birdnet.conf-defaults to create a birdnet.conf file

cp ~/BirdNET-Pi/birdnet.conf-defaults ~/BirdNET-Pi/birdnet.conf

you can then edit birdnet.conf with whatever values you want and then you can install the system with those custom values from the start by just running

~/BirdNET-Pi/scripts/install_birdnet.sh

2. More generally, are you amenable to rearranging the directory structure of the repo? I'm not sure that I have anything in particular in mind right now; I'm more trying to figure out what you're open to.

Yes, so long as the structure is still semantic. The symlinking that is used to build the webroot is really dumb and not user/developer-friendly, so truly I'm pretty open to any changes that improve readability and aid in development through semantic naming and intuitive structure whenever/wherever possible.

3. Are there any thoughts yet about how or where a .deb file would be hosted? I can do research here, but just curious if you had something in mind.

I do not yet have any plan in mind for where a .deb package might be hosted, but depending on how big it is, I can probably host it on the same server as BirdNETPi.com, though a CDN would provide much faster download speeds.

4. One of the things I want to ensure is that everything works well with any package that gets produced. I'm thinking here of both from-scratch installations and upgrades. Is there anything like an integration test (maybe a script, maybe just a checklist) that's used to verify functionality right now?

Not right now, but it should be easily implemented I would think. Adding one example file to the package, then placing it in the directory that is configured for $RECS_DIR after installation should suffice to test birdnet_analysis.service, birdnet_server.service and the extraction.service. The service scripts can just be run in a shell and then some small additional line to check for the extraction file it should produce. Likewise for testing the recording service--just run its service script manually and then test for the presence of a file.

👍 👍 👍

[saintaardvark]

Awesome, that all makes a lot of sense to me. I'll keep plugging away at this, and look for small chunks along the way that can be submitted as standalone PRs...much better than some enormous dump of work. 😁

(One option for package hosting is Github itself: https://docs.github.com/en/packages. That might be the simplest way to get started once we get there.)

[saintaardvark]

Another question: are you amenable to creating a separate user for running the BirdNETPi processes? Currently, the installation uses the account of the user running the install script; the usual way with packages is to create a new, purpose-specific user (though running as, say, nobody:nogroup might also be an option).

With the default pi user going away, it might be good to think about a project-specific ID. That would probably complicate migrations/upgrades from a scripted install, but I shouldn't think it would be terribly hard to make happen.

[mcguirepr89]

I think that is totally fine -- a birder user or birdnet user or something with perhaps a /usr/sbin/nologin shell? Would that make sense to ensure there is no shell access through that user?

You can see where my waters get a little murky, but I'm definitely on board with securing these things and know that I'm somewhat hacking systemd to get things done at the system level. Most of the services ought to be /etc/systemd/user/HERE, but I've just leveraged the passwordless sudo policy for the default RaspiOS user to the point where a lot of Linux system administrative best praxis have been abandoned for Raspberry Pi-isms.

I'm not terribly concerned with migrations from old installations into a packaged installation -- when software is still new and growing, breaking changes will become part of the territory.

In theory, though, a packaged version of BirdNET-Pi could replace a current installation with relative ease by backing up the birdnet.conf file, the birds.db database and the $EXTRACTED directory, uninstalling, then installing from the .deb and replacing the database and extractions wherever the .deb package expects to find them. Something like:

cp ~/BirdNET-Pi/scripts/birds.db{,.backup}
cp ~/BirdNET-Pi/birdnet.conf{,.backup}
cp ~/BirdSongs/Extracted/By_Date{,.backup}
uninstall.sh
[command to add the .deb repo to apt-sources list]
sudo apt update && sudo apt -y install birdnetpi
sudo mv ~/BirdNET-Pi/birdnet.conf.backup /etc/birdnet/birdnet.conf
mv ~/BirdNET-Pi/scripts/birds.db.backup /etc/birdnet/birds.db
restart_services.sh

🤔

[saintaardvark]

Awesome -- again, that all makes sense to me. Not sure off the top of my head if having nologin for a shell would break the services, but it'll be easy enough to find out. And good to know that backing up/porting an existing installation would be pretty simple.