This is an attempt to automate the creation and mirroring of git-bundles, i.e. single-file representations of complete git repositories. It imitates what is included in the build scripts of the i2p.i2p project, here to allow to bundle other projects as well. Background is myself hitting the wall when trying to clone i2p.www (but also noticing the several years old bundle for i2p.i2p at that time) ;)

bundle-bot@i2p:~/tmp$ du -hs *
545M	i2p.i2p
327M	i2p.www

This is what we are talking about. If a normal git-clone gets interrupted, you'll start from scratch after that. It took me like 20 tries to clone the i2p.www repo over http inside i2p - even with tricks like --depth 1 followed by --unshallow and setting tunnel lengths lower as well as more tunnels (doh).

So in the spirit of maybe being able to make this easier to some follow-up-fellow, I started looking into what's behind all this and sharing the results here. Uh, making public means license. Not too knowledgable about that, but will go with gpl2 to have something said and make my intentions clear.

User / Just want to use ...

... use the product of this? I'm currently looking for feedback on the created bundle and seeding the following via torrent:

... i2p.www

i2p.www Magnet - posting this link into i2psnark should allow the download to start. This is not uploaded to Postman or so, yet, and you will not get the torrent-comment, therefore here a copy:

This is a git-bundle of the official I2P web-site and documentation repository for people having trouble getting the initial cloning done.

Download the file and then do:
git clone -b master --single-branch i2p.www.bundle-complete-git-xxxx-xx-xx i2p.www
cd i2p.www
git remote set-url origin git@git.idk.i2p:YOUR-GITLAB-ACCOUNT-HERE/i2p.www
git remote add upstream git@git.idk.i2p:i2p-hackers/i2p.www
git pull upstream master

You are supposed to connect this to the upstream repository at git.idk.i2p as above with remote add. When pulling from there you also ensure I have not tempered with the sources (or you would notice) :)
You can use http-transport using http://git.idk.i2p/i2p-hackers/i2p.www.git instead, if you do not require write access.

Based on http://i2p-projekt.i2p/en/docs/applications/git-bundle
Also see http://i2p-projekt.i2p/en/docs/applications/git for more on git/gitlab over I2P and a proposed git workflow.

I've tested downloading and cloning from it as described. I will keep this running until I notice there are "official" bundles available (I think idk is looking into this).

... i2p.i2p

Use this torrent provided by idk: i2p.i2p.i2p-2.1.0.bundle I'm trying to help seeding that and developing a script to take care of that here (see mirror below).

Admin / Requirements for Running

Software and User

I basically did

apt install git
apt install mktorrent
adduser bundle-bot

This is on a Debian-stable based install where java-i2p is running as a service (i.e. /var/lib/i2p/i2p-config/i2psnark and user i2psvc). If your setup differs, you should look for needed adjustments accordingly and carefully.

Git-Setup

You need to setup git to work with the transport and URL you wish to use for keeping the repository up-to-date and obviously need to get the repository before you bundle it also. Possible variations for transports:

ssh/git on any of
- git@git.idk.i2p:i2p-hackers/i2p.i2p.git or
- WHATSTHECLEARNET or
- GITHUB-READONLY-MIRROR
http on any of
- http://git.idk.i2p/i2p-hackers/i2p.i2p.git or
- WHATSTHECLEARNET or
- GITHUB-READONLY-MIRROR

Maybe see the I2P project page on git for details if in doubt.

I2PSnark

The scripts assume an i2psnark directory to be available. Accordingly you will need a router with i2psnark running on the same machine as a service (or you need to adjust accordingly). Due to the service-install permissions can be messy if not running as root. And running as root (where not technically needed) is usually unwise. I'm trying to cope with it, but check for yourself how far I've gotten ...

Scripts

bundle-bot-creator.sh

This creates bundles for i2p.i2p and i2p.www. After realising that idk is building bundles for i2p.i2p and said there will be some for i2p.www, too, I currently do not further this. You'd probably want to use a tag for i2p.i2p in there (as idk is doing), not sure what's best for i2p.www as there are no tags in that repository.

bundle-bot-mirror.sh

For the already available i2p.i2p bundles I figured I will start mirroring those instead of creating additional ones then. When I started this the only one I was able to find was 3 years old. It still seemed to fullfill its purpose just fine though, btw. Anyway, this script is supposed to:

download the page listing torrents from idk,
parse it for the latest i2p.i2p version,
download the .torrent for that version, and
move the file to the i2psnark directory.

The idea is to keep one older version (as the new one needs to be downloaded first then), so it should delete the current "old" and move the current "current" to be the new old. The code currently only cares about .torrent files, not actually downloaded files. I will probably extend this to delete the "old" downloaded file as well.

CurrentState / ToDo

This is utmost alpha ;) So check the code and see if you can find something useful, but don't blame me if you implode some universe by running it. I actually just changed code and can't even test it now, as I don't want to overwrite files. You have been warned :P

i2psnark doesn't autostart the torrents so far - either need to restart router, do it manually, or find another way
Does the .torrent needs to be published at postman to be "accessible"? Maybe grep the magnet link somehow and just publish that on a website?
mktorrent is called as root, this doesn't seem sensible. Just need to figure out how to run the lengthy line with su ...
Some error handling - checking exit codes at the very least - before trusting this to cron/service
service file/timer file then btw. once/twice a year for new torrents maybe?
And for cron/service lockfile handling always seems like a good idea
logging/notification-mails/-messages
automated tests? Do I wanna be that fancy?
Automatic distribution to mirrors? Download won't be very fast if only one serves them.
Cleaning up - I decided to change my "root makes rm --force bla*" into something less frighening, but if supposed to run automatically, old versions would need to be removed at some point (it's currently around 780 MB for a version with both repos)
The other cleaning up - regular job once a week or so to keep everything up-to-date?

git fetch origin +refs/heads/*:refs/remotes/origin/*
git fetch upstream +refs/heads/*:refs/remotes/origin/*
git fetch --all
git pull --tags
git pull --all
git remote prune origin
git gc --prune=now
git bundle create i2p.i2p.i2p-2.1.0.bundle i2p-2.1.0
sha256sum i2p.i2p.i2p-2.1.0.bundle