Over the years, various syncing applications have bugged me with how often they spike to 100% CPU usage, their memory usage, pricing, and other issues. So I set out to write my own for fun! Spoiler: in the end, I learned you really should just use Dropbox 🤣 However, I did manage to make it work, and reliably enough that I’m using it for some personal files, so I’m open-sourcing it in case anyone wants to check it out.

Flare

The project is called Flare, which is in reference to the ‘blaze’ part of the backend service’s name.

It is essentially a Dropbox clone that performs 2-way sync of files between a local folder and ‘the cloud’ (in this case, Backblaze B2). These files will then sync to many computers automatically. You could use it to keep a folder synced for members of a team, for instance. And because it uses Backblaze, your storage costs are miniscule.

It’s written in Swift, available at github.com/chrishulbert/flare, and (more-or-less) works!

You really shouldn’t use this for important use, it’s a hobby project!

Features

• Uses zero CPU/RAM at idle because it only syncs hourly (and upon wake from sleep), and has no process running the rest of the time.
• Even when running, uses little CPU/RAM due to simplicity and native code.
• Syncs to Backblaze B2, which has extremely generous free tier and inexpensive pricing: If you used Flare and could live with its limitations, you could theoretically save a bucketload vs an office of people paying for a popular sync service.
• If you delete a file, it will remain accessible in the Backblaze B2 admin panel for the duration of their garbage collection period (a week or a month, potentially).

Limitations

• Both macOS and Windows (probably Linux too, I didn’t check) do not update the ‘last modified’ date of the tree of folders above a modified file. Eg, say you have a structure similar to foo/bar/yada/blah.txt and modify blah.txt. Foo, bar, and yada’s last modified date will unfortunately not change. In a slightly better case, if you rename blah.txt to blah2.txt, the immediate parent folder (yada) will change it’s last modified date, however grandparents and great-grandparents (foo and bar, respectively) will not. This is because those operating systems want to remain efficient. However this means that Flare cannot simply look at a last modified date and determine that it can skip an entire folder tree: it must still scan every subfolder and every file to see if anything has changed, every time it syncs. This is the biggest limitation of Flare. This could be resolved with an always-running agent, but then you’d also need a full-scan when launching the agent to catch any changes that occurred while it wasn’t running. However, for a hobby project I simply do not have the time to pursue this (I have children to chase!). Also, B2 does not give you folder last modified dates either, so you’d need a custom backend.
• Flare handles subfolders very poorly. If you create a subfolder, it will sync fine. But if you delete that subfolder, it won’t have the smarts to know to ‘push’ that deletion to the server and ‘pull’ it down and apply it to other computers. IIRC this was due to the lack of metadata that could be reasonably stored with B2, which is totally understandable from their perspective. I really pushed B2 too far with all this.
• Renames count as a deletion of the old file, and creation of a new file.
• Huge files are simply thrown in the too-hard basket and ignored.
• The macOS service that it used to schedule it is flaky: It launches Flare on wake from sleep only about half the time.
• In general: What this project taught me is that 2-way sync is an exponentially more complicated problem than 1-way sync (although not an impossible problem: I got reasonably close to a useful solution!) and you should use one of the popular services for anything important, and be amazed when they ‘just work’.

Installation

These instructions are only for macOS. However, the app should compile fine on any Swift-supported platform if you see fit.

• Create a Backblaze B2 account and bucket.
• Install homebrew if needed: brew.sh
• Install flare: brew install chrishulbert/flare/flare
• Configure it: flare configure
• Schedule it to run hourly: flare schedule

Musings

Swift really won me over with it’s ability to model all the edge-cases of synchronisation in a way that wouldn’t compile unless I handled every one. A great example of this is the switch statement in ListingReconciliation.swift. This is how a good language fosters safe code, and I’m a huge fan. So: thanks, Chris Lattner.

Thanks for reading, I hope this interests someone, and have a great week!

Legals: I take no responsibility; give no guarantee/warranty for this project.

Photo by Christopher Cambpell via Unsplash

Chris Hulbert

(Comp Sci, Hons - UTS)

Software Developer (Freelancer / Contractor) in Australia.

I have worked at places such as Google, Cochlear, Assembly Payments, News Corp, Fox Sports, NineMSN, FetchTV, Coles, Woolworths, Trust Bank, and Westpac, among others. If you're looking for help developing an iOS app, drop me a line!

Get in touch:
[email protected]
github.com/chrishulbert
linkedin

---

Please subscribe to my mailing list:

 Subscribe via RSS