Waverley Blog

Remember “Management By Wandering Around,” the somewhat old-fashioned notion that no amount of presentations, reports, meetings and conferences can substitute for getting out of your office and walking around (see Tom Peters’ great book: Little Big Things: Excellence).  Great idea, but many experts say it’s impractical when your employees are spread all over the world.  These days teleconferencing, phone conferencing, email, texting and IM substitute for face-to-face discussions.  Unfortunately, good as is, electronic connections cannot replace genuine human relationships.   And business teams who have healthy relationships are more productive.

Managing Waverley software development teams located worldwide, I’ve learned the importance of MBFA – Management By Flying Around.  Software development is complicated, difficult work.  Often there are unexpected problems and setbacks, so openness, honesty and trust among team members is critical to our success. There’s an old adage that says that the three most important keys to retail success are location, location, location.   Well, at Waverley we believe the three keys to outsourcing success are communication, communication, and communication.

I believe that if engineers and customers haven’t been face-to-face in the past six months, our relationship is probably set back to zero.  My visits (alone, with our Silicon Valley Management team, and with our clients) to our software engineers in their home offices creates goodwill. Flying our engineers to our clients builds relationships. Everyone knows that hiding problems creates problems.  As we all listen to each other, air our concerns, share our solutions we create an atmosphere of transparency and accountability.  And, we create better solutions for our clients.

You may be faced with the daunting task of establishing an outsourcing contract with a new outsourcing vendor. Typically you are confronted with a few well established approaches to tackling this challenge. I’ll review the options you’ve likely considered, and offer a newer model that Waverley has been using for many years that we believe is a much better approach.

First, the big contract approach is to establish a multi-year legal structure that defines the areas you wish to outsource combined with various terms to enforce the goals you wish to derive from the relationship. This long-term commitment is highly lucrative for the major outsourcing vendors, but requires you to make remarkable insights into where the relationship will go, and how you must keep the vendor on track. This approach is similar to fixed bid contracts we’ve discussed before, which can be appropriate for some projects, but is often fraught with peril due to the uncertainty of future events and the unknown evolution of your vendor partnership. You want the best terms with your new vendor, but you need to ask yourself how much a huge contract investment and long-term commitment help you to build a truly win-win situation for the two of you.

Another approach is to establish detailed project specifications and contract the outsourcing vendor to execute on these precise plans. This also has many parallels to typical fixed bid projects, but fails to lay a foundation for the building of a relationship with the vendor. Are you just looking to satisfy the short-term needs of a specific project, or are you interested in building a real vendor partnership that can deliver value across many projects?

What you may really want is legal protection for critical issues like pricing and intellectual property, combined with a framework for both you and the vendor to grow to trust each other and build a foundation for long-term work. This approach uses a boilerplate Professional Services Agreement (PSA) combined with simple Statements of Work (SOW) for each project the vendor works on. The PSA makes no commitments on your part to engage the vendor for more than a month or two on termination. The vendor is constantly proving themselves to you each month through the quality of their work. You have the legal and price protection you’re looking for in the PSA, and each new project only requires a new SOW without PSA renegotiation. You start small and build a mutually beneficial relationship with the vendor from the very beginning. As you both grow to get comfortable working together, you can adapt the growing team to the areas that best fit your business. You’ll find the value in this arrangement is substantial.

As you can see, there is a much nimbler and adaptive approach available to you when establishing an outsourcing relationship and contract with a vendor. Use it with your next outsourcing partner for great results.

There are a variety of solutions for performing backup on the Mac that aim to make things easy for you through the use of great user interfaces. For those of us that are more interested in setting up fully scripted solutions, the Mac provides all the tools you need to do sophisticated backup tailored to exactly what you need. The great news is that everything you need is all based on free and open source solutions already installed on your Mac.

I’ll cover two key types of data you’re likely to need to backup: databases and files. I’ll assume that your databases are in MySQL, but there are similar tools for working with other popular databases. This same technique can also be used for Linux systems, but I’ll focus on the particulars for the Mac. I’m also assuming that you have root permission (sudo) to perform these commands.

First you’ll need to setup a special database user that we’ll only use for backups. You’ll need to enter your MySQL root password at the prompt, and then replace BACKUP_PASSWORD with a unique and secure password for this new user.

$ mysql -u root -p
mysql> grant select, reload, lock tables on *.* to backup@localhost identified by 'BACKUP_PASSWORD';
mysql> flush privileges;


You’ll then want to create an excludes file to avoid backing up data files that don’t make any sense. My install of MySQL is in /usr/local, but if yours is elsewhere you’ll need to change the last line to the location of the MySQL data files. There’s no need to backup these files as we’ll backup all your databases in a later step.

$ cat >> /usr/local/etc/backup_excludes.txt << EOF
/tmp/
/Network/
/Volumes/
/cores/
/afs/
/automount/
/private/tmp/
/private/var/run/
/private/var/imap/socket/
/private/var/imap/proc/
/private/var/launchd/0/sock*
/private/var/spool/postfix/
/private/var/vm/
/Previous Systems.localized
.Trash/
.Trashes/
.Spotlight-*/
/usr/local/mysql/data/
EOF


Next you'll need to mount the backup drive using the diskutil command. The exact disk name is likely different from my device name disk1s6. You can figure out the device name for your backup drive using the diskutil list command.

$ diskutil mount disk1s6


You then have to turn on ownership settings for the backup volume to ensure you get an exact copy of the files on your source drive. The default for external drives doesn't maintain original file ownership. We'll use the mdutil command.

$ mdutil enableOwnership disk1s6


Now we'll use the powerful rsync command to mirror an exact copy of all your files. There are a lengthy set of confusing parameters you can pass to rsync, but those you need are shown below. Archive mode (-a) makes sure everything is copied exactly as it is on your source drive. Only your source drive will be backed up, it will not cross filesystem boundaries (-x). Sparse files (-S) will be handled efficiently. Extended attributes (-E) and resource forks will be copied as well. Files that have been deleted on the source drive will also be deleted from the backup (--delete). The directories and files specified in the excludes listing we created earlier (--exclude-from) will be skipped.

$ rsync -axSE --delete --exclude-from /usr/local/etc/backup_excludes.txt / /Volumes/Backup/


It's also a good idea to turn off Spotlight on the backup as it's not really needed and only takes up space and time.

$ mdutil -i off /Volumes/Backup
$ mdutil -E /Volumes/Backup


Let's make the backup bootable while we're at it using the bless command.

$ bless -folder /Volumes/Backup/System/Library/CoreServices


Now let's backup all your MySQL databases using mysqldump. You'll need to replace BACKUP_PASSWORD with the backup database username we created earlier. The backup will be archived and compressed using the gzip command. The backup filename will be created using the current date for easy reference if you need to restore things to a previous date. You can read all about doing backups for MySQL here.

/usr/local/mysql/bin/mysqldump -u backup -pBACKUP_PASSWORD --all-databases --lock-all-tables --flush-logs | gzip > /Volumes/Backup/usr/local/mysql/backup/all.`date +%u`.sql.gz


Finally you can unmount the backup drive now that we're done.

$ diskutil unmount /Volumes/Backup


You can take all these steps and turn them into a single script for doing your backups. You can also schedule the script to run at whatever frequency you like. Other steps and ideas you can take next include:

• Rotate multiple backup drives and keep at least one physically at another location.
• Make application specific database backups available to your teams for easy download. Use the mysqldump command and replace DATABASE_NAME with your application's database name:


/usr/local/mysql/bin/mysqldump -u backup -pBACKUP_PASSWORD DATABASE_NAME --lock-all-tables --flush-logs | gzip > /usr/local/apache2/htdocs/backups/DATABASE_NAME.`date +%u`.sql.gz


• Make compressed archives of specific directories available to your teams for easy download. Use the tar command and replace directory paths and file names with appropriate values.


$ tar -C /usr/local/apache2/htdocs -czf /usr/local/apache2/htdocs/backups/appdata.tar.gz application/data


• Use the asr command to clone an exact copy of your disks as an alternative.

You can find more information on each of the commands I've talked about in the Mac OS X Manual Pages section of Apple's developer site. The Mac is an excellent UNIX platform for building powerful server platforms for all your infrastructure needs.

I recently attended a Certified Scrum Master training course and came away from the experience convinced it was worth the time and cost.  I have, of course, been exposed to Agile development over the years, but more from an informal, incidental perspective and from reading books on the subject.  The major concepts of short iterations, fully functional results, and the opportunity to rearrange priorities during project development I knew and understood well.  However, seeing the entire framework described – and used – during the course of the training really tied the entire concept together for me.  I have not only a fuller, more complete picture of what Scrum is and why it works, but also hands-on experience garnered in the rich, guided environment of experienced teachers and other motivated and smart colleagues.

The questions asked by others taking the course really increased the value of the training; their own concerns and past experiences brought out answers and advice that will probably come in very handy in the future but that I would not have thought (or known) to consider myself. Of course our training “exercises” were not nearly the same intensity or complexity as a real-world project, but they provided an excellent combination of getting to “try out” Scrum while having coaches readily available to answer questions and provide guidance.  I am much better prepared to initiate using Scrum with a team after this course than I would be after reading 10 books.

As Waverley’s newest CSM, I recommend that if your company is considering implementing Scrum, formal training is an investment that’s well worth it.

CIO magazine published this article, Keeping the Cash: IT Leaders Can Slash Costs which recommends steps to take to reduce IT costs. One of them caught our attention:

“Halt the development of certain projects temporarily or permanently. If any of these projects are outsourced and your paying the outsourcer for time and materials, stopping the project, even for just a few months, can immediately save you money.”

This may make sense in some cases, but I can’t think of too many. From the perspective of working on high priority mission critical projects, stopping randomly in the middle of development makes no sense because there is as much or more to lose as there is to gain. Firstly, If you’re working with great people, they will not sit around and wait for you – they will move on to other projects where their skills are useful. Outsourcers can’t leave top talent stagnant for months at a time. Secondly, when you’re ready to get going again, you’ll have new knowledge transfer and ramp up time issues to go through and this will delay the end date much more than just the several months you chose to postpone work.

Here’s a much better way to proceed. Rather than stopping and starting, look at your project and focus on delivering value as quickly as possible. It’s pretty well accepted that most of features of a large IT application are not going to be used and not all features have the same value to the organization. Divide up your features, prioritize each feature, then implement the work in small steps and have the project stakeholders evaluate the results at the end of each small step. Be willing to change priorities to match the organization’s goals. This is classic Agile development at work. At some point, you’re going to conclude that you have enough of the project done to meet critical needs and you can stop development and deliver something useful, rather than stopping development with little or nothing to show for your investment to date. It makes a lot of sense for a lot of reasons to keep development teams running smartly towards a goal unless there is a significant mitigating factor, so get the most from your partners but don’t let your project run on autopilot until a crisis occurs and you have to stop before you have something to show for your hard work.

Last week, Waverley attended it’s second MSVP vendor summit at the Microsoft campus in Redmond. Like almost everyone else, Microsoft is working hard to manage and reduce costs more effectively and get the most out of the dollars they spend on their vendors. Comparing what’s going on at Microsoft to some other large companies we work with, we see some trends.

Providing services to large companies is going to require more effort on the part of both the procurement departments, the consumers of the services provided and the vendors they work with. Procurement processes will face increased standards and solutions will need to be scalable across groups when possible. In order to control costs, large buyers are centralizing procurement functions and instituting policies regarding RFP’s and PO’s that help ensure buyers are getting more for their money. Governance will be more important than ever. I expect all vendors will have to provide more information to justify costs, work to creatively address the needs of their customers and actively drive their customers to find new ways of doing more with less. Buying in bulk will become more common and those vendors that can be flexible and scale have an advantage. Operational excellence has been and continues to be critical to the success of the vendor/buyer relationship. I expect the buyers of services to work harder to learn from their vendors and not just buy bodies to fix problems. I also expect more limited strategic vendor relationships. Standing still has never been a path to success and will be even less likely to work in this economic environment. Competition is going to drive relationships to be more effective and productive and the innovation that results should improve our mutual interests. We’ve always worked hard to have the best people, management and process and I think that effort should pay off for everyone.

When you are an organization providing services to clients it is important to check in every once in a while to make sure the relationship between the two companies is still as mutually beneficial as possible.

At the beginning of an engagement both parties typically complete at least some amount of due diligence and agree that the business goals are good for both sides. If through the course of time, high-level strategic communication doesn’t take place, then that due diligence becomes stale and potentially worthless. And you may not find that significant disconnects exist until major damage is already done to the relationship through project failure, or even a complete dissolution of the engagement. These risks can be mitigated by having what I’ll call a Customer Review (similar to what’s known as a QBR) on a regular basis, about 3-4 times a year.

Some customers won’t want to spend the time. The Customer Review process is external to what they’re paying you to do and they can sometimes have a hard time seeing the benefit, but it is crucial for the long term success of the relationship. It’s key that you help them understand that finding strategic misalignment early will save them money over the long run and help verify that you are providing the best service you can to your customer.

The Customer Review should be a face-to-face meeting in a conference room with the management of both companies in attendance. Any sponsor executives as well as direct managers of the project should participate and attend from the client’s side. Following are some of the things that should be covered in the meeting as well as items important to you and/or each of your customers.

Presented by you:

• History of the relationship, accomplishments since last Customer Review
• Your company health, growth of the company, new areas of expertise, etc

Presented by the customer:

• Past project success in the marketplace, lessons learned, health of current project, etc
• Future roadmap, business strategy, product plans

Group input during meeting:

• Issues and difficulties currently hindering productivity
• Action items for next Customer Review

If you meet in this brief, semi-formal way with your customers on a regular basis, you will find you understand them better and can anticipate their needs more proactively, which enables you to provide the highest quality of service.

This article from eWeek discusses the results of survey from VanDyke/Amplitude that found a larger number of network intrusions at companies that outsourced tech jobs. There aren’t a lot of details, unless of course you go and get the full survey, which I don’t have access to. If I were to guess, I would say most of this comes from the requirement for such companies to open network access to their company’s internal network. This of course is fraught with a variety of security issues that I suspect many companies aren’t taking seriously enough.

There are a few ways to mitigate against these kinds of risks. Here are a few ideas:

• Choose an outsourcing partner that has lots of experience working via VPN and SSH with their partners. Their experience can help you ensure the right steps are taken. They will also be able to demonstrate how their own internal networks are kept secure.
• Another option is to find an outsourcing partner that has their own secure network and services distinct from your own corporate network. The best partners have great capabilities for securely working with you without the need to access your internal network. You can then sync work at periodic intervals without full VPN access.
• Specifically contract a network security specialist to assist in setting up access to your internal corporate network.
• Isolate access to the networks and services your outsourcing partner needs. Keep these separate from what your internal people use. This might not be practical in many cases, but it could work in your case.
• Use SSH with public keys as much as possible. It is as secure as a VPN, but offers many advantages to control access at a fine-grained level.

On a recent family road trip through California and Oregon, we lost track of time and found it hard to find a restaurant that was open after 8 pm. While driving around town, we happened to end up at a train depot at the same time a long and slow moving freight train was rumbling by. The depot was a combination Amtrak station and a freight railroad support location with offices, staff and rows of railcars waiting to be moved to their destinations. We were hungry, a little bit lost and tired from a long day of traveling and wildlife watching, but my five year old son was especially enthralled and wanted to watch for a bit. Sounded good to us.

We waited for the train pass in the last light of the evening, well after the sun had actually set, with the string of lead engines impressively shaking the ground and the following cars clanging by. To our surprise, the train stopped as its last car, a locomotive pushing from behind, stood opposite the depot. Three people came out of their offices and started to work on uncoupling the engine from the back of the train. My son was very interested and vocal about watching – you know how young kids can be vocal – and to my initial dismay, he attracted the attention of the three guys working on the train. I thought “uh oh”…

We got surprised again when we got invited to come watch up close. Our host walked us to the front of the locomotive, we greeted the other guys and he invited us to climb up the stairs. Pretty cool. We keep on going, next through the crew door and up more stairs to the inside of the operator’s cab. Getting even cooler now. My son got to sit on the left seat, turn the main headlights on and off , honk the train’s horn then watch the crew change a switch on the track and move the locomotive off to a siding, much to everyone’s delight. The people were super friendly and pleased to explain how their impressive machine worked.

Afterwards, we accompanied the manager to his office, got a few souvenirs and got ready to say goodbye. One of the engineers came in and the manager briefly switched his attention to business. Before leaving, they filled out some forms and signed them. I noticed with interest that the forms were carbon copies, not forms on the web or on a handheld device. The last one was peeled off the back and handed to the engineer. “Goodnight, have a safe drive home”. We said our good byes too and headed back to our car. Our excitement during the last hour masked our increasing hunger and now we scrounged some snacks in the car to keep us going.

I started to wonder how the railroad worked, how they handled crews signing in and out, how they scheduled people and freight and what their information technology infrastructure looked like. Could it be improved? Could someone with the ability to help a large industrial company figure out how to be not just as good as the rest but to rise above the competition in the information technology area? Could we apply our skills in managing complex multi part programs from concept to final production roll out help such a company? How many other companies needed expert guidance to really rise to the top of their market segment? It was fun to think about.

We’ve been working on several large projects that remind me of our experience in the train yard. I know it is possible to do a great job working together as a team to not only build the product, but define it too and make sure what we build meets clear objectives. And we know that focusing on delivering value is critical – this means doing the work in a timely manner and focusing on top priorities first, involving stake holders and constantly evaluating and reducing risk.

A challenge many employees face is remote access to their company network. People want to be able to work remotely exactly as if they were sitting in their office. The IT department needs to help their employees remain productive, while maintaining the high security standards necessary for internal servers and applications. The traditional approach to this problem has been the use of proprietary VPN networking using specific network appliances from companies like Cisco, SonicWall, NetGear, and others. This method requires special software installed on each computer, so your computer will appear as if it is on the company’s local network.

A large challenge with this approach has been the security issues when granting access to people working with the company, but not as employees. A better solution would be to grant different levels of access for each person, rather than granting access to the entire corporate network. A typical VPN solution to this problem has been through the careful configuration of your network. This is clearly a non-optimal answer. More modern VPN techniques such as SSL VPN have attempted to leverage more common standards for connecting client computers, but they suffer from the same basic issues.

Let me suggest a better approach to solving this problem that many technical people are already familiar with, Secure Shell (SSH). People doing software development commonly use source control software such as CVS, Subversion, Git, and others to manage the development of software across many developers. The most common method for access to this common source code is through the use of secure networking via SSH and public key cryptography. An SSH tunnel is used to encrypt all network traffic between their computer and the corporate network computers. Typically developers maintain both a VPN connection for most network services, and an SSH connection for their development work.

My suggestion is that there is no need for two distinct solutions. SSH can be used to provide totally secure point-to-point network communication for source code development, email, web access, and more. This can be controlled on a per-user basis for any number of services. An SSH approach to remote access can be as totally transparent and easy to use as any existing VPN technology. There are a few issues with achieving this simple and intuitive approach that are all easily solvable with the help of major operating system and browser companies. Note that all the solutions below are freely available as part of time-tested open source solutions.

1. The first step is creation of your private and public key pair. Today you must run a separate application every time you wish to create these keys. Some operating systems have a command built into the system, while others require download of an SSH application such as PuTTY. It’s a simple matter of performing these steps automatically, every time a new user account is created.
2. A user interface is required to identify which network services should be routed over the secure network using SSH. This is easily done using URLs, for example internal.company.com.
3. The operating system can then identify remote network access to these resources and setup SSH tunnels as required. For example, access to a remote IMAP mail server would actually connect to a local SSH port, which is then sent over an encrypted SSH tunnel to the actual mail server. Accessing a web page on the remote network would similarly connect to a local port, but use the SOCKS protocol to dynamically route traffic to the remote web server. Note that it is important that the SOCKS v5 protocol is used with remote DNS resolution since remote network names are typically not exposed on the public Internet. The operating system and web browser can totally handle setup, teardown, and configuration of these tunnels entirely behind the scenes with no user input being required.
4. Improved server tools can greatly simplify and standardize making secure configuration of users and services. Open source SSH server solutions permit configuration of each user and the services they are permitted to access based on their public key. Powerful, yet easy to use GUI and browser based solutions are straightforward to build upon this foundation.

SSH offers the best solution for secure remote network access, but it has been relegated to use by the technical elite. Developers, hackers, and sysadmins have used these technologies for years and recognize the value it provides as a superior form of secure communication. Only through the adoption as the defacto method for VPN access by companies like Apple, RedHat, Microsoft, and others, can this powerful technology break into mainstream usage. When it does, everyone will benefit from much more secure, stable, and easy to use networking from any location.