Waverley Blog

This article from eWeek discusses the results of survey from VanDyke/Amplitude that found a larger number of network intrusions at companies that outsourced tech jobs. There aren’t a lot of details, unless of course you go and get the full survey, which I don’t have access to. If I were to guess, I would say most of this comes from the requirement for such companies to open network access to their company’s internal network. This of course is fraught with a variety of security issues that I suspect many companies aren’t taking seriously enough.

There are a few ways to mitigate against these kinds of risks. Here are a few ideas:

• Choose an outsourcing partner that has lots of experience working via VPN and SSH with their partners. Their experience can help you ensure the right steps are taken. They will also be able to demonstrate how their own internal networks are kept secure.
• Another option is to find an outsourcing partner that has their own secure network and services distinct from your own corporate network. The best partners have great capabilities for securely working with you without the need to access your internal network. You can then sync work at periodic intervals without full VPN access.
• Specifically contract a network security specialist to assist in setting up access to your internal corporate network.
• Isolate access to the networks and services your outsourcing partner needs. Keep these separate from what your internal people use. This might not be practical in many cases, but it could work in your case.
• Use SSH with public keys as much as possible. It is as secure as a VPN, but offers many advantages to control access at a fine-grained level.

On a recent family road trip through California and Oregon, we lost track of time and found it hard to find a restaurant that was open after 8 pm. While driving around town, we happened to end up at a train depot at the same time a long and slow moving freight train was rumbling by. The depot was a combination Amtrak station and a freight railroad support location with offices, staff and rows of railcars waiting to be moved to their destinations. We were hungry, a little bit lost and tired from a long day of traveling and wildlife watching, but my five year old son was especially enthralled and wanted to watch for a bit. Sounded good to us.

We waited for the train pass in the last light of the evening, well after the sun had actually set, with the string of lead engines impressively shaking the ground and the following cars clanging by. To our surprise, the train stopped as its last car, a locomotive pushing from behind, stood opposite the depot. Three people came out of their offices and started to work on uncoupling the engine from the back of the train. My son was very interested and vocal about watching – you know how young kids can be vocal – and to my initial dismay, he attracted the attention of the three guys working on the train. I thought “uh oh”…

We got surprised again when we got invited to come watch up close. Our host walked us to the front of the locomotive, we greeted the other guys and he invited us to climb up the stairs. Pretty cool. We keep on going, next through the crew door and up more stairs to the inside of the operator’s cab. Getting even cooler now. My son got to sit on the left seat, turn the main headlights on and off , honk the train’s horn then watch the crew change a switch on the track and move the locomotive off to a siding, much to everyone’s delight. The people were super friendly and pleased to explain how their impressive machine worked.

Afterwards, we accompanied the manager to his office, got a few souvenirs and got ready to say goodbye. One of the engineers came in and the manager briefly switched his attention to business. Before leaving, they filled out some forms and signed them. I noticed with interest that the forms were carbon copies, not forms on the web or on a handheld device. The last one was peeled off the back and handed to the engineer. “Goodnight, have a safe drive home”. We said our good byes too and headed back to our car. Our excitement during the last hour masked our increasing hunger and now we scrounged some snacks in the car to keep us going.

I started to wonder how the railroad worked, how they handled crews signing in and out, how they scheduled people and freight and what their information technology infrastructure looked like. Could it be improved? Could someone with the ability to help a large industrial company figure out how to be not just as good as the rest but to rise above the competition in the information technology area? Could we apply our skills in managing complex multi part programs from concept to final production roll out help such a company? How many other companies needed expert guidance to really rise to the top of their market segment? It was fun to think about.

We’ve been working on several large projects that remind me of our experience in the train yard. I know it is possible to do a great job working together as a team to not only build the product, but define it too and make sure what we build meets clear objectives. And we know that focusing on delivering value is critical – this means doing the work in a timely manner and focusing on top priorities first, involving stake holders and constantly evaluating and reducing risk.

A challenge many employees face is remote access to their company network. People want to be able to work remotely exactly as if they were sitting in their office. The IT department needs to help their employees remain productive, while maintaining the high security standards necessary for internal servers and applications. The traditional approach to this problem has been the use of proprietary VPN networking using specific network appliances from companies like Cisco, SonicWall, NetGear, and others. This method requires special software installed on each computer, so your computer will appear as if it is on the company’s local network.

A large challenge with this approach has been the security issues when granting access to people working with the company, but not as employees. A better solution would be to grant different levels of access for each person, rather than granting access to the entire corporate network. A typical VPN solution to this problem has been through the careful configuration of your network. This is clearly a non-optimal answer. More modern VPN techniques such as SSL VPN have attempted to leverage more common standards for connecting client computers, but they suffer from the same basic issues.

Let me suggest a better approach to solving this problem that many technical people are already familiar with, Secure Shell (SSH). People doing software development commonly use source control software such as CVS, Subversion, Git, and others to manage the development of software across many developers. The most common method for access to this common source code is through the use of secure networking via SSH and public key cryptography. An SSH tunnel is used to encrypt all network traffic between their computer and the corporate network computers. Typically developers maintain both a VPN connection for most network services, and an SSH connection for their development work.

My suggestion is that there is no need for two distinct solutions. SSH can be used to provide totally secure point-to-point network communication for source code development, email, web access, and more. This can be controlled on a per-user basis for any number of services. An SSH approach to remote access can be as totally transparent and easy to use as any existing VPN technology. There are a few issues with achieving this simple and intuitive approach that are all easily solvable with the help of major operating system and browser companies. Note that all the solutions below are freely available as part of time-tested open source solutions.

1. The first step is creation of your private and public key pair. Today you must run a separate application every time you wish to create these keys. Some operating systems have a command built into the system, while others require download of an SSH application such as PuTTY. It’s a simple matter of performing these steps automatically, every time a new user account is created.
2. A user interface is required to identify which network services should be routed over the secure network using SSH. This is easily done using URLs, for example internal.company.com.
3. The operating system can then identify remote network access to these resources and setup SSH tunnels as required. For example, access to a remote IMAP mail server would actually connect to a local SSH port, which is then sent over an encrypted SSH tunnel to the actual mail server. Accessing a web page on the remote network would similarly connect to a local port, but use the SOCKS protocol to dynamically route traffic to the remote web server. Note that it is important that the SOCKS v5 protocol is used with remote DNS resolution since remote network names are typically not exposed on the public Internet. The operating system and web browser can totally handle setup, teardown, and configuration of these tunnels entirely behind the scenes with no user input being required.
4. Improved server tools can greatly simplify and standardize making secure configuration of users and services. Open source SSH server solutions permit configuration of each user and the services they are permitted to access based on their public key. Powerful, yet easy to use GUI and browser based solutions are straightforward to build upon this foundation.

SSH offers the best solution for secure remote network access, but it has been relegated to use by the technical elite. Developers, hackers, and sysadmins have used these technologies for years and recognize the value it provides as a superior form of secure communication. Only through the adoption as the defacto method for VPN access by companies like Apple, RedHat, Microsoft, and others, can this powerful technology break into mainstream usage. When it does, everyone will benefit from much more secure, stable, and easy to use networking from any location.

We’ve looked at a variety of software development processes over the years we’ve been creating custom software, including Waterfall, spiral processes (eg, RUP) and of course Agile. I find Agile particularly interesting because it is so heavily focused on value. Here’s how it works. Imagine a factory producing a widget. It’s well understood that as the factory manager, you’d like to keep you inventory down and deliver raw materials just in time. Agile functions in a similar manner in the software world. By promising to deliver frequently, involving stakeholder feedback and constantly reducing risk, it is much easier to use Agile to guide you to delivering something of the highest value as quickly as possible. In effect you are reducing your inventory by keeping your feature list very focused and not implementing things that you really don’t need. This is how the focus on value works.

Looking back to our post on the 4 dials exercise, Agile really is saying that you don’t have much flexibility with the date due to both cost and competitive factors. The date is a major factor in determining cost. If you’re spending dollars on a big team, then every day is expensive.  By fixing the date and delivering as much quality code as possible by that date, you force yourself to prioritize your huge list of features that you think you really must have but probably do not really need. And that is a good thing.

Recently the CEO of a new client called me to ask why his project with us was at day 85. This was a reasonable question since it had been originally scheduled for 24 person-days!

This client had interviewed us last summer for a position to help them develop some software they viewed as critical to their business strategy. They were very picky about whom to choose and interviewed at least 20 companies. We were honored that we got the nod.

Part of the selection process was to estimate a small project. Our team looked at a simple description, some specs, asked some questions and came back with a 24 person-day estimate. Everyone agreed that this was reasonable.

We started with a single resource, an expert lead developer who specializes in the area our partner was interested in and off he went. So far the project was pretty straightforward.

Day 85

Now we are at day 85. What happened? It’s a common tale, but a cautionary one.

First of all, the client admitted that they changed many requirements (some multiple times), including some API’s  that had to be redone and significant changes to the UI requirements.

On our end, we all knew that the development schedule had run over and that the team had not addressed the delay directly since the relationship with the client was going well.

The client also specifically stated that we had been doing a great job. They were very happy with the quality of the work and the commitment of the developer and client manager. Still, we were at day 85. Not good.

What lesson should we learn?

Is the lesson here that clients shouldn’t change their requirements? This is unlikely, given the dynamic nature of business applications. Or perhaps it is that developers should sound the alarm each time a small delay occurs, even when the client causes the delay and approves the changes? That doesn’t seem very practical either.

Here’s what I took away from the experience. It’s important to have a healthy discussion at the beginning of the project about how the project team should respond to changing requirements. Because with business applications, everyone should assume that requirements will change. It is only be seeing early versions of an application that customers can evolve their understanding of the is really required.

Assume that requirements will change. Make plans about responding to those changes. These steps will go a long way to keeping the balance between the three dimensions of projects: schedule, cost and quality.

I had a regular “client update” call from a customer last week. The project for this customer is nearing completion and so we are starting to think about the transition from development to support. In this case, the concern was whether one person could properly support the work that four developers had been doing on four mobile platforms. Among the questions they asked me was what happens when we go from a team of development resources to a support staff of one?

This is a time when clients are anxious because they want to minimize the costs of managing the risk of potential problems with the application.

Here are the questions we typically hear from clients about the transition and what we advise them.

What should support include?

The main activities that support should cover are minor changes to the custom software application and changes that are caused by updates to the operating system.

What is the payment structure for support services?

Keep the pricing model for support as simple as possible. Our clients prefer a per-hour fee structure because they only have to pay for the support they need.

Which person gets assigned to maintenance and support?

Generally, the best person is one of the members of the development team. Preferably it will be the developer who has the broadest view of the work that was done or who is most knowledgeable about the areas that will need the most support.

But then what happens if we need support in an area outside this person’s experience or expertise?

If you hire the developer to provide support, they will be familiar with the server interfaces and with the software operating environment. They have the process in place to develop and build the code and run it through tests with each change. They can also draw on their team’s capabilities when necessary to answer internal questions about how something works or what to do.

What happens if at some point in the future we need more support than this, or even additional development?

If you need it you can always ask your developer to ramp back up to a multi-person team to provide more depth.

The important point is that your developer show flexibility by meeting all your support needs and by charging you only for the time they work for you.

A short history of outsourcing

In the 1990’s organizations spent a lot of energy looking for ways to reduce their costs and so they could put more focus on their core business. Service firms arose to meet this demand to cut costs – human resources, accounting, security, internal mail distribution, catering and of course, information technology services.

As companies searched for the lowest cost providers of these services and as telecommunication costs plummeted, it wasn’t long before U.S. companies were contracting for support services in other countries – India, China, Eastern Europe, and the Philippines. Initially the prices were so low, especially in India and China, that labor arbitrage was the primary motivator. Sometimes even at the expense of quality. Or with lots of management frustration from the U.S. company as they tried to make these relationships work.

Most people would agree that the days of outsourcing internationally only to achieve lower labor costs are over. As more companies entered the business in each country and began to compete with each other for labor, costs began to rise. Now companies can achieve some cost benefit from outsourcing, but it is more modest than several years ago.

From low-cost-provider to strategic partner

In the meantime, outsourcing companies have improved the quality of their work and their ability to serve clients, so they can continue to make a convincing case for their services – slightly lower costs, higher service quality.

In a recent CIO article, Arpit Kaushik accidentally pointed out another feature of offshore outsourcing. The primary focus of the article is a caution to IT customers of offshore outsourcing to beware of exaggerated cost reduction claims by outsourcers. He advises the reader to make appropriate comparisons, to look for hidden costs, and to make the distinction between theoretical and real impact.

Towards the end of the article, Kaushik makes a comment that reveals the direction of offshore outsourcing. He cites a recent paper from a “leading offshore outsourcer” which revealed this tidbit: The “bulk of the benefit actually comes from revenue improvement rather than tangible cost savings.”

This comment reveals that the current stage of offshore outsourcing is defined more by the desire to seek strategic partnerships than cost savings alone. Cost savings will only get you so far; at some point companies have to improve long-term profitability by finding opportunities for growth and by increasing their revenue.

Certainly we’ve seen that’s the case in our business, software development outsourcing. When we first started, customers were mainly focused on labor rates.

Today they mostly ask us what we can do to improve the software they are developing and shorten their time to market. Customers want much more than skilled engineers at a lower rate. They want to know how we manage large, distributed projects, how we manage project risk, how easy it will be to do business with us.

This is a challenging but exciting direction for software development companies. Challenging because it’s more difficult to be a strategic partner than a commodity service provider. Exciting because the projects are more interesting for our engineers and foster closer relationships with our customers.

Every new outsourcing contract discussion must cover pricing. It’s always the subject that people dance around and are often reluctant to talk about.

There are so many ways to price services. Most high end professionals work hourly. Some work on a retainer or monthly for a guaranteed number of hours. Some work on a fixed-bid project basis. And a few will even work for equity. Regardless of the billing agreements, charging for services can test the trust in any buyer-provider relationship.

But shouldn’t pricing be as easy and transparent as every other part of the relationship? Don’t you want to feel confident that you’re not only getting a good deal, but that you can trust your software service provider to bring their best people and practices to the table?

You’re running a complex business and don’t want surprises. You may be pleased if your provider worked extra to meet that important deadline, but not so pleased when you receive a bill that is 40% more than what you expected.

What most customers look for in pricing is this:

• a predictable burn rate
• a simple pricing structure
• open communication and transparency when changes to a project (e.g. requirements) affect pricing

There is more than one way to meet these criteria. The approach Waverley uses (after a lot of experimenting) is to use time-and-materials pricing and a simple, inclusive hourly-rate pricing structure.

On Waverley projects we provide a team of full-time people and charge for those people on a flat-rate basis. This means that all costs are included except for obvious exclusions such travel, shipping, and special hardware or software that only you may have. Regular client management and onshore and offshore engineering management is always included. The billing rate includes insurance, computers, and a good place for our engineers to work. We can even keep the lights on through an emergency (some locations have large scale power generation facilities). Waverley continually invests to improve our service and tools. We travel to our offices to make sure all is working properly.

As I said, this isn’t the only way for you to feel that you are getting a fair deal from your outsourcer. The main thing with pricing is to be confident that your outsourcer is meeting your need for predictability, simplicity, and transparency.

Effective software development teams have defined communication paths that help guide how projects are run. One important path governs what to do when a customer finds a defect in the software. The bug gets reported, logged, prioritized. The priority of the bug determines when it will be evaluated and fixed.

But what if the rules don’t work? What if a low-priority bug is happening to a high priority person – for example, an executive under pressure to deliver who may unreasonably question the project design and has the political means to damage the overall project? Suddenly it’s time to escalate and go outside the normal rules.

This is exactly the situation that arose this week. I’ll tell you how we handled it, but first some background.

A pilot program under threat

One of our development teams is piloting a recently completed large application. The pilot is going well, at least it was until I got a call from the Program Manager for the client. This Program Manager is as steady as a rock, so if he calls and says he’s concerned, I sit up and pay attention. On this call he’s concerned, and I’m paying attention.

The issue was that two of the pilot users were having a problem with the application and were making their frustration known to a lot of people, including the CIO. Although the problem had been logged with the development team, it did not get a high priority based on its description. The development team was focused on completing an iteration and so this problem was on the list of bugs and would be considered when prioritizing the backlog for a future iteration. Standard development methodology here.

The symptom that the two users were experiencing was this: a request to the application that should normally take less than two seconds was taking 45 seconds. As you can imagine, they found this troublesome.

Redirecting the team

After I got off the phone with the Program Manager, I immediately contacted the manager of the development team in Romania and explained the situation. We agreed that we needed to immediately focus the development team and the QA team on fixing this bug.

Of course, when the developers tried to replicate the problem at their lab, they couldn’t. The application performed perfectly.

It wasn’t until they could actually gain remote access to the user’s computer that they could see the problem. What did it turn out to be? The local database for the application wasn’t indexing properly and so some queries to the database were taking much longer than they should. The developers were able to quickly fix the problem by running a simple script once they identified its source. The script was included in a future patch to be sure this problem would not resurface and we reviewed our release procedures and patch creation process to look for anything we could improve in a future release. Finally, we checked back with the two pilot users to make sure they were satisfied with the software.

What’s the takeaway?

Once the developers fixed the problem and restored the user’s application to its normal state, their job was done. As for me, I still had plenty of work to do to reassure the Program Manager, our Sponsor, and the CIO that the problem was an isolated one, that it was a simple problem and that it did not reflect any fundamental underlying issues with the application.

What does this experience tell me? I don’t think it’s that we should change our procedure for fixing bugs. My takeaway is that even if a development team has a good set of procedures and communication policy in place, some situations will arise that require escalation. In these cases, the Client Manager has to change the team’s priorities and put all its energy into fixing the immediate problem. It also points out the importance of an effective escalation path so that the client always has someone to go to when the conditions are extraordinary.

In the scheme of the whole project, this kind of escalation is unproductive. The problem the two pilot users were experiencing would have been resolved in due course. Escalating an entire QA team for 36 hours prevented other issues from being worked on, so there was likely a small schedule hit as a result of the team being quickly redirected, then redirected again when the bug is resolved. But sometimes, despite the productivity hit, it’s necessary to parachute in the swat team so that the client can manage internal expectations and politics. We understand this is the reality for our client, and we work to support them as best we can. But for everyone’s sanity and for the continuity of the project, we try to keep these fire drills to a minimum.

Picture these situations:

Three engineers from different groups talking at lunch about a difficult bug that one of them had spend most of the night fixing. Nothing wrong with that, right? Just some colleagues talking informally about their work.

The Product Manager meets with the User Interface Design Group to decide about priorities for the next release. Once again, perfectly appropriate. Part of the job of the Product Manager is to establish release priorities. Meeting with each engineering group is one of the ways this gets done.

A Sales Account Manager asks an Engineer to make a change in the next release that a customer has asked for. Oops. We just crossed a line here. We would expect the Account Manager to advocate on behalf of a customer, but going straight to the Engineer is the wrong way to do it.

The Client Project Manager calls an Engineer at a vendor firm to ask how a particular feature in the application works. The line just got crossed again. The Project Manager for the client should have a primary person, probably a Client Manager, who can field these calls for the client. It’s not that the client can’t get these questions answered, it’s just that they should be answered in the most productive way possible.

These examples point clearly to the need for defining communication paths for project teams. But how can you define them in a way that encourages helpful informal communication and that also ensures necessary formal communication?

At one extreme you could view a project team as a network with potential connections from each member of the team to every other member of the team. But if everyone talked to everyone else, life would be chaotic and the team would never reach its goals.

At the other end of the spectrum you could have tightly controlled communication that defines exactly who can speak to whom. This approach allows for no flexibility in the system and is overly restrictive.

Finding the middle

The answer of course lies somewhere in the middle. What is that middle?

Before I answer the question, let’s look at the difference between formal and informal communication. At Waverley we think of formal communication as any communication that causes a person to change work priorities, goals, or that affects productivity. Everything else is informal.

So according to this definition, formal communication includes obvious events like team meetings, client email, design reviews. It also includes any nonessential and lengthy calls to engineers by the client.

The way to find the middle is to define communication paths – not to the degree that they limit informal communication – but so that formal communication can proceed in a predictable and productive way.

The communication paths in distributed teams are controllable, even across remote locations, if everyone on the team understands and plays by the basic rules.

Early definition of communication paths will help avoid problems. As the project evolves, communication paths can evolve with them.

So how does a team go about defining communication paths? It starts by defining roles, then by looking at groups of roles, and finally at paths between groups.

That’s what I’ll be examining in my next two posts.