Waverley Blog

There are a variety of solutions for performing backup on the Mac that aim to make things easy for you through the use of great user interfaces. For those of us that are more interested in setting up fully scripted solutions, the Mac provides all the tools you need to do sophisticated backup tailored to exactly what you need. The great news is that everything you need is all based on free and open source solutions already installed on your Mac.

I’ll cover two key types of data you’re likely to need to backup: databases and files. I’ll assume that your databases are in MySQL, but there are similar tools for working with other popular databases. This same technique can also be used for Linux systems, but I’ll focus on the particulars for the Mac. I’m also assuming that you have root permission (sudo) to perform these commands.

First you’ll need to setup a special database user that we’ll only use for backups. You’ll need to enter your MySQL root password at the prompt, and then replace BACKUP_PASSWORD with a unique and secure password for this new user.

$ mysql -u root -p
mysql> grant select, reload, lock tables on *.* to backup@localhost identified by 'BACKUP_PASSWORD';
mysql> flush privileges;


You’ll then want to create an excludes file to avoid backing up data files that don’t make any sense. My install of MySQL is in /usr/local, but if yours is elsewhere you’ll need to change the last line to the location of the MySQL data files. There’s no need to backup these files as we’ll backup all your databases in a later step.

$ cat >> /usr/local/etc/backup_excludes.txt << EOF
/tmp/
/Network/
/Volumes/
/cores/
/afs/
/automount/
/private/tmp/
/private/var/run/
/private/var/imap/socket/
/private/var/imap/proc/
/private/var/launchd/0/sock*
/private/var/spool/postfix/
/private/var/vm/
/Previous Systems.localized
.Trash/
.Trashes/
.Spotlight-*/
/usr/local/mysql/data/
EOF


Next you'll need to mount the backup drive using the diskutil command. The exact disk name is likely different from my device name disk1s6. You can figure out the device name for your backup drive using the diskutil list command.

$ diskutil mount disk1s6


You then have to turn on ownership settings for the backup volume to ensure you get an exact copy of the files on your source drive. The default for external drives doesn't maintain original file ownership. We'll use the mdutil command.

$ mdutil enableOwnership disk1s6


Now we'll use the powerful rsync command to mirror an exact copy of all your files. There are a lengthy set of confusing parameters you can pass to rsync, but those you need are shown below. Archive mode (-a) makes sure everything is copied exactly as it is on your source drive. Only your source drive will be backed up, it will not cross filesystem boundaries (-x). Sparse files (-S) will be handled efficiently. Extended attributes (-E) and resource forks will be copied as well. Files that have been deleted on the source drive will also be deleted from the backup (--delete). The directories and files specified in the excludes listing we created earlier (--exclude-from) will be skipped.

$ rsync -axSE --delete --exclude-from /usr/local/etc/backup_excludes.txt / /Volumes/Backup/


It's also a good idea to turn off Spotlight on the backup as it's not really needed and only takes up space and time.

$ mdutil -i off /Volumes/Backup
$ mdutil -E /Volumes/Backup


Let's make the backup bootable while we're at it using the bless command.

$ bless -folder /Volumes/Backup/System/Library/CoreServices


Now let's backup all your MySQL databases using mysqldump. You'll need to replace BACKUP_PASSWORD with the backup database username we created earlier. The backup will be archived and compressed using the gzip command. The backup filename will be created using the current date for easy reference if you need to restore things to a previous date. You can read all about doing backups for MySQL here.

/usr/local/mysql/bin/mysqldump -u backup -pBACKUP_PASSWORD --all-databases --lock-all-tables --flush-logs | gzip > /Volumes/Backup/usr/local/mysql/backup/all.`date +%u`.sql.gz


Finally you can unmount the backup drive now that we're done.

$ diskutil unmount /Volumes/Backup


You can take all these steps and turn them into a single script for doing your backups. You can also schedule the script to run at whatever frequency you like. Other steps and ideas you can take next include:

• Rotate multiple backup drives and keep at least one physically at another location.
• Make application specific database backups available to your teams for easy download. Use the mysqldump command and replace DATABASE_NAME with your application's database name:


/usr/local/mysql/bin/mysqldump -u backup -pBACKUP_PASSWORD DATABASE_NAME --lock-all-tables --flush-logs | gzip > /usr/local/apache2/htdocs/backups/DATABASE_NAME.`date +%u`.sql.gz


• Make compressed archives of specific directories available to your teams for easy download. Use the tar command and replace directory paths and file names with appropriate values.


$ tar -C /usr/local/apache2/htdocs -czf /usr/local/apache2/htdocs/backups/appdata.tar.gz application/data


• Use the asr command to clone an exact copy of your disks as an alternative.

You can find more information on each of the commands I've talked about in the Mac OS X Manual Pages section of Apple's developer site. The Mac is an excellent UNIX platform for building powerful server platforms for all your infrastructure needs.

A challenge many employees face is remote access to their company network. People want to be able to work remotely exactly as if they were sitting in their office. The IT department needs to help their employees remain productive, while maintaining the high security standards necessary for internal servers and applications. The traditional approach to this problem has been the use of proprietary VPN networking using specific network appliances from companies like Cisco, SonicWall, NetGear, and others. This method requires special software installed on each computer, so your computer will appear as if it is on the company’s local network.

A large challenge with this approach has been the security issues when granting access to people working with the company, but not as employees. A better solution would be to grant different levels of access for each person, rather than granting access to the entire corporate network. A typical VPN solution to this problem has been through the careful configuration of your network. This is clearly a non-optimal answer. More modern VPN techniques such as SSL VPN have attempted to leverage more common standards for connecting client computers, but they suffer from the same basic issues.

Let me suggest a better approach to solving this problem that many technical people are already familiar with, Secure Shell (SSH). People doing software development commonly use source control software such as CVS, Subversion, Git, and others to manage the development of software across many developers. The most common method for access to this common source code is through the use of secure networking via SSH and public key cryptography. An SSH tunnel is used to encrypt all network traffic between their computer and the corporate network computers. Typically developers maintain both a VPN connection for most network services, and an SSH connection for their development work.

My suggestion is that there is no need for two distinct solutions. SSH can be used to provide totally secure point-to-point network communication for source code development, email, web access, and more. This can be controlled on a per-user basis for any number of services. An SSH approach to remote access can be as totally transparent and easy to use as any existing VPN technology. There are a few issues with achieving this simple and intuitive approach that are all easily solvable with the help of major operating system and browser companies. Note that all the solutions below are freely available as part of time-tested open source solutions.

1. The first step is creation of your private and public key pair. Today you must run a separate application every time you wish to create these keys. Some operating systems have a command built into the system, while others require download of an SSH application such as PuTTY. It’s a simple matter of performing these steps automatically, every time a new user account is created.
2. A user interface is required to identify which network services should be routed over the secure network using SSH. This is easily done using URLs, for example internal.company.com.
3. The operating system can then identify remote network access to these resources and setup SSH tunnels as required. For example, access to a remote IMAP mail server would actually connect to a local SSH port, which is then sent over an encrypted SSH tunnel to the actual mail server. Accessing a web page on the remote network would similarly connect to a local port, but use the SOCKS protocol to dynamically route traffic to the remote web server. Note that it is important that the SOCKS v5 protocol is used with remote DNS resolution since remote network names are typically not exposed on the public Internet. The operating system and web browser can totally handle setup, teardown, and configuration of these tunnels entirely behind the scenes with no user input being required.
4. Improved server tools can greatly simplify and standardize making secure configuration of users and services. Open source SSH server solutions permit configuration of each user and the services they are permitted to access based on their public key. Powerful, yet easy to use GUI and browser based solutions are straightforward to build upon this foundation.

SSH offers the best solution for secure remote network access, but it has been relegated to use by the technical elite. Developers, hackers, and sysadmins have used these technologies for years and recognize the value it provides as a superior form of secure communication. Only through the adoption as the defacto method for VPN access by companies like Apple, RedHat, Microsoft, and others, can this powerful technology break into mainstream usage. When it does, everyone will benefit from much more secure, stable, and easy to use networking from any location.

Developing software requires expertise across a wide range of disciplines. Areas such as project management, software development, and software testing are performed by skilled people that have had years of practical experience delivering on time and within budget. The areas within each of these disciplines that contribute to successful software delivery include people, process, communication, tools, and so on. I’ll be discussing the role of tools in this article, particularly as it relates to outsourced software development projects.

Companies looking to outsource software projects fall into one of a few categories. Each of these cases is an opportunity to find an outsourcing partner that can best meet your needs. Let’s look at each of these cases in detail.

You’re an established software company that already has a suite of tools to support your current internal development team.

The optimal solution for most of these cases is to find a partner that can adapt to the tools you are currently using. Look for experience with your specific development infrastructure. They should have experience with the most common tools across a variety of functions such as:

• Source control systems such as CVS, SVN, Perforce, SourceSafe, or ClearCase.
• Bug tracking systems such as Bugzilla, Mantis, TestTrack, Quality Center, and ClearQuest.
• QA automation tools such as Tcl, Expect, Perl, WinRunner, QuickTest Pro, and Rational Robot.
• Development environments such as Visual Studio, NetBeans, IDEA, and Eclipse.
• Project management and communication tools such as wikis, conferencing, mailing lists, IM, Skype, and Trac.

Your partner must be capable of working independently with periodic updates to your tools, or most commonly use a VPN to connect directly with your internal systems. Your outsource team should meet the same expectations you would have of a remote development team you hired yourself. Your outsourcing team’s past experience with your company’s tools will greatly contribute to their ability to become productive quickly. Be wary of partners that work independently in cases such as these.

You’re a new software startup that doesn’t have the tools in place yet.

This can be an excellent opportunity to learn from an experienced software development partner. Their experience with great tools and process can help get your own tools and process going in the right direction. Some partners may hide the inner working of their approach to developing your software, but the best partners will open their own systems for everyone within your company to access. Look for tools that leverage the best open source solutions for great software development. They must provide for secure access to everyone on your team while promoting great transparent communication regarding all your projects. Make sure all your data is easy to transition to your own internal tools once you are ready.

You’re not in the business of developing software.

In this case, you may think your partner can do whatever they want as long as your project is delivered on time and within budget. In reality you will find that you still must take ownership of software products to some extent. The future could include transition to another team either internally or through another partner. Since the work is typically done work-for-hire, there’s no way around the fact you are now in the business of software.

Look for a great partner that can help you understand software development at the level that makes sense for you at this stage. You should also be looking for great open source tools for the very same reasons stated in the previous case. The deeper understanding you will gain from transparent access during development will improve your satisfaction with the entire project.