Symirr -- Symmetrical File Tree Mirroring Tool

Intro | Sync operations | Downloads and manual

Introduction

Symirr is a recursive file/directory synchronisation tool for Linux (and potentially other POSIX systems if ported). It is for mirroring changes made to duplicate copies of the same directory tree kept in two or more locations. This is done in a symmetrical fashion, i.e. there is no sense of a "master copy" -- changes can be made on either side (or both sides) and then sync'd to the other. This is ideal when it is necessary to keep duplicate synchronised copies of mail folders, project directories, and so on, on several different machines. It is also useful for creating a 'teleport' directory to quickly transfer files between various machines.

Advantages:

• Handles immense trees without problem, streaming the data, never attempting to keep the whole tree into memory. It is designed to be a small efficient tool, with a small memory footprint.
• Shows exactly what will be done in each sync before performing the actions, invoking the user's favourite editor on the action command-list. Gives user full control to edit the action command-list, permitting partial or modified syncs if required.
• Can safely run fully automated after the first sync.
• Allows sync between many machines. All problems may be resolved by fixing files on the local machine and syncing again, even problems on far-remote machines (two or more links away).
• Copes well with transfers/syncs of files/directories done outside of Symirr (e.g. with rsync or cp -a or tar), so long as modification-time/etc is also synchronised by these tools.
• User may safely abort the sync part-way through with ^C. Also robust to various other types of confusion or failure.
• As a final safeguard, keeps TGZ backups of old versions of changed or deleted files when the -b option is given.
• Test-suite to ensure that program performs correctly; this is constantly being expanded as bugs are found and fixed.

Limitations:

• Does not attempt to detect renamed files or directories; these appear to Symirr as a deletion and a new file or directory.
• Does not attempt to detect multiply hard-linked files; these appear as independent files to Symirr. Symbolic links are handled, though.
• Does not handle special files (fifos, sockets, devices, etc), nor files with names containing control characters. Symirr simply ignores these files.
• At present always copies whole files. This means that Symirr is not the best choice for syncing large files of which only a small part changes in any update; however, you can use rsync manually in parallel with Symirr to handle these files if necessary.
• At present requires a round-trip on the communications link for each action-command; this ensures good consistency between the two ends in case of failure, but can be much slower than ideal in some situations.

The current code of Symirr has been refactored many many times. I'm happier having a small codebase so that I can make sweeping global changes whenever I find a better approach. To me this is much better than having a huge codebase that I'm afraid to touch for fear of breaking something. This means always looking for the small/simple/efficient/optimal way of getting the job done, rather than going for the comprehensive/extensive-feature-set style, which perhaps explains to some extent why Symirr's design is the way it is.

Sync operations

There are really two distinct sync operations done by Symirr: the first sync and subsequent syncs. (See the IMPLEMENTATION section in the manual for full details of the algorithms.)

The first time Symirr does a sync, it does not know anything about the history of the files it finds there. Ideally the first sync you would do with Symirr would be between the original directory tree and an empty directory (i.e. letting Symirr copy the files across for you), or you would be syncing to an identical copy of the file tree that you have copied across yourself, either using cp -a or tar or some other tool. However, it is possible that you made a duplicate copy of the file tree at some point in the past and you have made changes on both machines and now you want to use Symirr. Since there is no history information in this situation, Symirr cannot detect deleted files, and neither can it detect clashes (files changed on both sides). For this reason, on the first sync it is important to check over the action command-list file carefully to see that Symirr is doing what you expect it to be doing, to avoid any possible problems. In some cases Symirr inserts comments into this file which provide alternatives.

However, after the first sync, Symirr can accurately detect updates and deletions, and recognise clearly when clashes occur. At this point it is safe to allow Symirr to run unattended (e.g. to run automatically from cron). If clashes are detected, i.e. if file changes have been made on both machines, then *.CLASH-######## files are generated which allow the situation to be examined and resolved later on with no loss of information. This is a much more complicated and accurate sync operation (see the manual for the algorithm).

In addition to the default symmetric basic first-time sync (-B), there are also two asymmetric basic first-time syncs, -S 'local-is-secondary' and -M 'local-is-primary'. These are designed to duplicate exactly the version of the tree on the primary, overwriting or deleting anything on the secondary that disagrees. Even a newer file on the secondary will be overwritten with an older version from the primary, although a warning is given in this case. These are intended for the first-time sync when it is certain that no useful changes have been made to the secondary. One example use is to sync an old copy of an MH-style mail tree with the current version, where there may be many deletions required to the old copy which would result in (incorrect) attempts to copy files the other way if the default -B were used. So, -S and -M provide an alternative way to start Symirr off, better in some situations. After this point the option should be dropped, allowing the extended sync mode to take care of all deletions and updates.

Downloads

Documentation

Source-code download

Note that Symirr is still in beta, really: there are still a few bugs in Symirr still to be tracked down and a few known problems still to be fixed, which are described in the TODO file above. However, Symirr already works well and is in daily use by the author. My preference would be to wait until Symirr reaches "absolute perfection" before doing a release, but it is probably best to follow the open-source principle of "release early, release often" and get more people using it. So, if anyone finds a bug, please report it, if possible with a short bash-script which demonstrates the failure. Well-defined repeatable failures help me a great deal in tracking down the bugs. Also, in this way I can add more tests to the test-suite, to further refine and improve Symirr.

These pages and files, including applets and artwork, are Copyright (c) 1997-2020 Jim Peters unless otherwise stated. Please contact me if you'd like to use anything not explicitly released, or if you have something interesting to discuss.