Sunday, December 16, 2012

Software management woes

If boat builders built airplanes, they would be falling out of the sky faster then we could scurry around and pick up the wreckage and luggage. Somehow lessons that have been learned by most manufacturers of complex items, have been left behind by all too many boat builders. Oops, Mr. Grunion, we left our lessons learned on the train station platform. Never mind Bob. Don't worry. The buyers don't pay any attention to that documentation stuff when they write the check for the boat, and when the warranty expires, it's not our problem anymore.


With the advent of increasing technology being utilized in pleasure craft, boat builders seem to have overlooked a lot of basics used by other modern manufacturers. That's not to say nothing has been learned, but a lot is still yet to be learned. The buyer always pays the price for these lessons not yet learned.

So today we have a case in point to mentally masticate on. The secret words are "Software Configuration Management," and our boat is a Magnifico 60 motor yacht.

It's a pretty vessel, and on the whole it's well built. The builder has a configuration control system, at least more so than many of the others. Although they and others have suffered somewhat in this area with the tough times, and loss of personnel.

The entire boat, lock stock and barrel is run by software. Starting generators, turning on lights, flushing toilets, and powering navigation systems to name a few. I'm okay with this, if properly documented, and designed. Couple this with critical spares, and extensive documentation, life will be good for the owner and the boat. Did I say extensive documentation? Yes, and I meant it.

I can't emphasize enough how critical this is from the onset, and why builders either don't have it, or are so reluctant to provide it.

I'm going to jump to the crux of my current problem, and then ask, and answer two hard questions.

This boat has an extensive control system. The existing software that runs it has some issues. In order to make changes, you have to have an identical copy of the software that is running in the system loaded into a PC.

When changing the software the first thing the control system does is to compare the software in the PC to it's own. If they don't match, it won't let you make changes. The good news is that you can't muck up the system by using the wrong version of the software. The bad news is that if the current version of the software can't be found, no changes can be made.

Somewhere along the way, someone made a change to the system software, and the current version can't be found, so far anyway. I have three incorrect versions. Hence the software can no longer be changed or fixed easily. So question one is, "When you have system critical software, who is responsible for documenting the changes, and archiving the software?" The dealer, the manufacturer of the boat, the hardware vendor, or the owner.

My answer is the owner in the long run, and the software should have been supplied with the boat. If it was my million dollar vessel, I would absolutely want to have control of this, or at the least I would choose who would perform this critical function for me. The entire boats operation depends on this software working properly, and being maintained, or modified when needed. It's extremely important to keep very close track of it.

Question two is, "How much documentation should come with your new high tech boat? My straight forward answer is a whole lot. In this case, the boat was almost devoid of information. An incorrect owners manual, and a collection of materials on big systems like engines, freezers, and the ilk was all that existed.

The manufacturer has been politely cooperative, in an arms length sort of way in supplying information. But it is apparent to me the info is scattered all over their system. It has taken several months to extract the data, and to compile it. Half the battle was in finding out what even existed. On the control system alone, I have accumulated over 500 pages of materials. Questions are now answered like where are the about 80 control modules located in the boat? What are modules are being used for? What are the program addresses? How do you perform manual system overrides in emergencies? What are the part numbers? In addition actual wiring diagrams, a correct owners manual, system overviews, and many others have now been gathered and printed.

A large three ring binder is nearly filled, and the attached thumb drive has all of the technical errata such as the programming software, and lots of photographic documentation. So now back to my problem with the software. If the correct version can't be found, we will have to purchase a new computer and load it with the last known version for the boat. Since the system was programmed based on the options for each boat, it is suspected that it won't be perfect out of the box. Changes will have to made to fix it. Once this ordeal is completed, the software will be saved, and the original computer will be loaded with it, and stored as a back up. The software will be backed up in two locations. One copy on on the boat, and a second copy onshore. All of these gyrations and expenses are caused by a lack of crucial software configuration management. Yep, it is a technical world, and becoming even more so. It's time for boat builders to catch up if you want to play with the big boys.

As complexity of vessels increase, it is important that the documentation level increases exponentially, and stays with the boat. Builders go out of business, vessels aren't always near dealers, and all too often the level of complexity is so high, even the dealers often don't have good capabilities to support the technology.

For the buyers, you should absolutely demand much higher levels of documentation be provided with your high tech toys, if you want to keep them running. Get it in writing when you purchase your boat that you will get all technical documentation related to your vessel delivered with the boat. No excuses. If they won't give that to you, think twice about buying it. For the boat builders, spend a week or two at Boeing or the ilk to learn how this is done. Their planes don't fall out of the sky, and the owners get all documentation required to fully to support the aircraft with their purchase.

A final note from me, the control system manufacturer gets an A+ for their gracious, and competent assistance.  The boat builder so far gets a C-. It should have been an A+ also. I'm disappointed in them, and the final fix is still yet to come.

The first photo is of Toronto's first aircraft accident. The city has gotten a little bigger since then. 

4 comments:

  1. A very interesting read. I will be following this for sure.

    ReplyDelete
  2. From my long-lost days doing mainframe computer systems programming: "If architects built buildings like programmers write software, the first woodpecker would destroy civilization."

    ReplyDelete
  3. And people ask me why I like discrete instrumentation, sometimes of the analog variety. I suspect it's because if my oil sender (for instance) is acting up, I either replace it, or clean the sending wire, or clean the contacts on the gauge. I do not have to recompile the sub-routine in C++.

    I'm not saying there isn't a place for highly automated processes, modules, sensors and actuators. I am suggesting that salt water pleasure craft perhaps shouldn't be on the same list as fighter aircraft, space stations and international networks of debit card devices.

    It's not just documentation in that sense that is lacking; it's perspective. I have considered installing a large seacock at the (port side exiting) exhaust for my new engine, the idea being that if a particularly hard heel defeated the riser at that point, the seacock would keep seawater out of the system and, eventually, the manifold. Only someone bitten more than once by water backing up would even think of this. Naturally, I thought "I will rig a solenoid and actuator so that I can't key the start without opening that valve!" Of course, a length of threaded rod with the keys left hanging on it might work, too.

    Bah. Too complex. I am leaning towards the "North Sea transverse dual exhaust" popularized by Dave Gerr. See

    http://www.cruisersforum.com/attachments/gallery/7/9/79n_sea_transverse_exhaust-med.jpg

    Good post!

    ReplyDelete
  4. Rhys,

    Rule 41: Increased complexity equals reduced reliability.

    Rule 42: Seafood restaurant place mats are never charts, even if they sort of look like one.

    Rule 43: The nicer the boat shoes, the less nautical skills.

    ReplyDelete