Synchronising data between computers/system using Unison for Linux/OSX
Introduction: What is Unison and why would I use it
Imagine you are working on a project. All files and directories related to this are in a directory in one of the departmental data spaces. You are going away and will be working on your project while on the beach without internet access. You copy your project directory onto your laptop.
While on holiday you do some project work as planned When you return to your office in CMS you continue with your project, but with the files you have in the data space. Then one day you remember you edited some files while you were away but you now cannot remember which files you edited on holiday on your laptop and which since returning on the departmental system. You will need to go through both systems picking out the newest copy of each file and watching out for any files you changed in both locations. Painful.
If you had used Unison to do the copying you wouldn't have this issue. Unison would sync both the laptop directory and the departmental one so that both contained the most up to date copy of all the files. If you had edited a file in both locations, Unison would notice this and flag this for your attention, so that you can decide what to do.
Unison can be very useful if you work on different computers on a regular basis; maybe work and college or work and home.
This document describes how to use Unison to sync files between your department home directory and your Laptop for Linux and MacOSX. While people running windows on their laptops are welcome to use unison, we recommend that they use WinSCP instead.
Install Unison
This is a two step process. The same version of Unison needs to be installed both on your laptop and in your Maths home directory.
-
To install Unison on your laptop check to see if your operating system already has a Unison package. Systems like Ubuntu will, or you can download a binary version or build your own version from source. The examples on this page use the graphical versions.
-
If your laptop is running the same version of Ubuntu as the Maths desktops, you need not install Unison in your Maths home directory as it is already installed centrally. Otherwise follow the link above, download the same version as you have on your laptop, compile it and place the unison binary in your Maths home directory's bin directory calling it unison (not unison-number) If you cannot find a match contact help@maths letting them know which version you need.
Profiles
Start the GUI interface to Unison on your laptop. On some systems you will have a Unison Icon, on some linux systems you may need to type unison-gtk or similar. Your system should have its own documentation to guide you if this step isn't obvious.
Step one: Create your profile. Some systems ask you to enter all the information on one screen, others over two or more screens. You will be asked for the following information.
- Profile name: use a meaningful name
- Full path of Maths Directory you are using (make sure it exists)
- Full path of your laptop directory (make sure it exists)
- Host name of Maths computer - see below
- Username for Maths computer
- Depending on which Unison package you are using, you may be asked to select the connection method. SSH is the answer
At the time of writing, please do not use ssh.maths.cam.ac.uk, ssh.damtp.cam.ac.uk or ssh.dpmms.cam.ac.uk. These are "round robin" systems where one name can connect you to any of a group of computers. This confuses Unison and causes it to throw up an error. Please use your desktop computer or discorectangle.maths.cam.ac.uk or obround.maths.cam.ac.uk which are the machines which make up ssh.maths.
Next open your profile, your system may do this part automatically for you, and enter your Maths password when asked. You must use your Maths unix password here because your Maths home directory and the Maths data spaces both reside on unix computers. If you only ever use Windows machines you may need to find your "New Account" letter as password changes on windows machines do not change your unix password.
You will now find yourself at a general transfer screen where Unison shows you which way it is transferring any files. At the top of that screen you will see various actions you can take if you do not want to take Unison's default action for any file, for example you may prefer to copy an older copy of a file back over a newer version. The picture under the introduction shows a typical screen.
Finish transferring the files, work on them and next time you want to sync, Unison will show you a list of your profiles, select the profile you want and transfer your files.
Technical part: In your Maths home directory you may notice a new directory called .unison. Don't delete this. This is where your Unison profiles are kept; these can be customised.
Linux laptop to departmental directory
This is the easiest case, you are transferring from a linux computer to a linux computer. The above instructions should work fine. The images may look slightly different but are in essence the same. Make sure you install both unison and unison-gtk packages on your laptop.
Open Unison on your laptop and set up a profile as described above.
Then let Unison transfer your files.
Mixed Case File names - WATCH OUT
Some systems are case insensitive, OSX, Windows and even your USB stick may be. This means you cannot create one file called "eggs.doc" and another called "Eggs.doc", as to these systems they are the same file. However the departmental system is case sensitive and allows you to do this. This causes problems when it comes to using Unison.
The best way to avoid problems with Unison and Mixed Case Filenames is not to have any files with the same name but different case (no mouse.c and Mouse.c for example) Run the command below to catch all clashes. Either rename the files to get rid of the clashes before starting Unison e.g., to filename.old or filename.dup, or if this is inconvenient (e.g. with system files) tell Unison to ignore them (see later section on Profiles).
find . -print | tr 'A-Z' 'a-z' | sort | uniq -cd
Also make sure you don't have file name case clashes between the files in your remote and local directories you are syncing, as otherwise bad things may happen to your files. The OSX notes below demonstrate what happens if there is a clash and how to get Unison out of its confused state.
Case insensitive example (uses OSX for examples)
As described above OSX is a case insensitive system, along with Windows and many formatted USB sticks.
In this image I have two files, one called "Pig" and one called "pig" in my Maths directory. Unison notices the conflict for the two files and reports it with a big red X and a message at the bottom of its screen.
I notice the problem, I quit from unison and I rename "pig" to "bacon". I keep the file "Pig". I run Unison again. But Unison is now confused and it cannot see the "Pig" file.
The best thing to do is avoid the conflict in the first place by following the advice given above.
If Unison does get into the confused state above you can get out of it by renaming the departmental directory you are transferring and creating a new unison profile. There is no need to rename the directory on your laptop. Creating a new unison profile on its own is not enough as unison leaves a state file behind when it runs and the new profile also finds this state.
Just renaming files when in this situation does not work and can lead to lost files as shown above. You must rename the target directory and create the new user profile.
Moral of story: Don't have names that clash for example "Pig" and "pig" when running Unison; rename them before using Unison and you'll save yourself a lot of time in the long run.
Note:
You may also wish to remove old archive files. Only do this if you don't use Unison for any other purpose as otherwise you'll remove your working archive files as well.
On OSX the unison archive files are in
~/Library/Application\ Support/Unision/ar*
Linux, MacOSX and Profiles (and problems with the hash sign)
For each pair of directories being kept in sync, unison creates a profile from the choices you make when syncing files. You can also edit your Unison profile files by hand.
A typical profile may look like
root = /Volumes/DAMTP_home/abc123 root = ssh://abc123@hostname.damtp.cam.ac.uk/ ignore = Path mail ignore = Path .mozilla ignore = Path .openoffice.org ignore = Path .thunderbird ignore = Path .cxoffice ignore = Path .java ignore = Path \#
Here we are keeping two homedirectories in sync. We don't want to copy over changes to mail and web related folders. We are also asking unison to ignore all files with a hash (#) sign in their name as unison is known to crash when it encounters this character.
Where to find your Unison's profiles.
Linux: in your homedirectory's .unison directory
OSX: in your home directory
Library/Application Support/Unison/
If you are running a version of Unison greater than 2.9.20 on OSX then Unison will use a .unison folder in your homedirectory for your profiles if one exists, otherwise it will use your Library folder.
Conclusion
I have only described the basics of Unison here, with an emphasis on how it interacts with the departmental systems. For further guidance on using Unison itself check the well documented official documentation. Some of the options differ on different versions.
