Topic and Location

Incident description

Flood – loss of electricity

Mississippi

This flood occurred due to floodgate control (at the dam) being overlooked in such a way as to protect the Ritzy homes built along the river way by the local elite class.

The flood suddenly overflowed the dam and the local “elite” community could not manage the volume of water crashing down upon it.

The computer room facility was barely accessible but the power was off for days. Being a production facility room power and Air Conditioning were the crisis. Several home-style generator sets were brought to the building and tapped into wall power sockets. Adequate power became available to start the computer room and minimal lighting. Large blocks of ice were brought in by pickup and placed behind the computer cabinetry. Small fans were used to draft across the dry ice to cool the computers of which lasted for a few hours at a time before replacement ice was needed.

The site met monthly production.

Monsoon – loss of electricity

Brunei

An expected seasonal monsoon arrived and flooded the coastline business structure where Shell housed its seismic operations.

Fortunately the original facility design provided for raising the floor by 1-foot for the computer room – which proved sufficient for this flood. Water seepage was detected only on the windward side of the building and was contained at the wall.

To provide for contractual commitments the data processing software, key project data and selective paperwork was temporarily relocated to the nearby hotel (3^rd floor) to isolate it from immediate risk.

The facility survived the 1-week issue.

Repeating Brown-Out

Singapore

A repeating brownout was causing data processing problems (varied degree of crash and restart) every Friday afternoon around 5:30pm.

A local machine shop operated some equipment that they powered up early Monday morning (~4:30am) and left online until Friday afternoon (~5:30 to 6:00pm).

The interruption caused an array of computer related problems on jobs that needed to run for days to weeks before completion.

Extended conversation with all personnel lead to knowledge of hearing a noise every Friday about time as various workers waited for their ride home or bus.

Visiting with the local companies led to the understanding of their power & production needs.

The processing center shifted its start/stop time and break time of night ship such that the day shift paused all production early enough as to miss the brown-out and started the night shift late enough to accomplish the same.

Monthly production increased as all other working hours realized clean power.

Flood – 1-foot above ground floor

Houston TX

This flood started as just another rain around 4:30 pm. By 6:00 pm, it was obvious that a flood was imminent and the basement and first floor were in jeopardy.

Sandbags that were available were placed at entranceway's and along first floor walls where leaks were detected. They were soon overwhelmed.

Seismic data (stored in basement) was moved via “human chain” to first floor and then to higher floors.

First floor computer rooms were shutdown, equipment that normally resided under the false floors were obtained and moved. Lower floor power was turned down to minimum areas.

In the following weeks all cables that previously lye on the concrete floor were moved to hanging trays about 1-foot up from the floor.

After considerable time of vacuuming water, the production center returned to operation with little loss other than time.

Electricity loss of City grid

Africa
Lagos

The location is a multi-floor-tenant building, which was accessed via a single elevator and of which was suddenly found without electricity.

It was summer time, on the equator, and 115 degrees F in the shade.

The cause of the citywide grid failure is unknown.

The problem is that the building engineer had not only falsified months of emergency generator test results; he had sold the onboard alternator (for engine starter batteries) so that a delivery truck could be repaired. I believe a family member owned the vehicle.

Contracts were violated as it was the major tenants responsibility to provide clean & full time electricity to the building such that vendors, contractors & visitors could use the building 24/7.

Practically, a full day was lost before the emergency system could bring the building back online.

Embezzlement

Taipei

A field manager working near China was in charge of the corporate working account for his field crew. This included operational and payroll expense responsibility. The funds were knowingly held in his personal bank account as to satisfy the local needs he often required cash to conclude business. The account had approximately $40,000 USD in it at the time of this incident.

For personal reasons and gain, he utilized the funds for procurement and debt relief to directly benefit himself.

The local authorities would take no action, as it was not their responsibility of his being a bad manager for that company.

The company had to make a decision of which meant leaving him onsite to conclude his contract or be fired and relocated home – also as per his contract.

The cost of hiring a new manager and transporting that person was also a consideration, which was further complicated by contractual obligations to return the field crew to work in a few days.

The weather was yet another consideration as it was approaching storm season and the total number of production days were few.

The final decision was to transfer the funds to him as a bonus and leave him onsite to conclude his overseas mission. Immediately upon return to Houston, his contract was reviewed and NOT renewed.

Theft by Deception

Africa
Lagos

Contractor obtained several sheets of corporate letterhead from a manager’s office of an expatriate company. The letterhead was utilized in the form of purchase orders to obtain oilfield equipment and a vehicle.

The company realized the matter upon being contacted by one of the oil field equipment shipping companies to clarify an address. This is when they determined that a 6-digit transaction was about to erupt at their bank.

Local authorities were quick to point out the company’s error in that the manager should not have had the letterhead in open/public view.

They described that the documents should have been under lock and key as within this country it was legal tender to use letterhead as a procurement tool.

About a year later, many legal fees (bribes or Dashes) later the local DA had the native African arrested for petty theft. The DA proclaimed that his crime was of stealing the paperwork and that the negligence of the local manager was the temptation for the overall act.

Some of the equipment was recovered and the vehicle was sold to cover the legal expenses of the contractor.

The legal system and assorted laws have improved.

A lockable filing cabinet was installed by weeks-end.

Prejudice

South America Caracas

A call for an emergency computer repair was issued from South America. The most expedient means of getting several boxes of parts through customs was to assign someone to the task of hand-carrying them. A local manager was selected as she could secondarily perform software updates when the system was back online.

Her view of some humans not looking the same nor speaking English well allowed her personal views to blind her judgment.

Upon arriving in a foreign country, she began to display her arrogance and decided to claim only a small fraction of the actual value of the many boxes of computer parts. The actual value was ~$50,000USD.

The attending customs agent took her information and departed to review past import records and examine the part numbers.

Upon returning he asked her to follow him and she was promptly escorted to a security holding area that within the airport on a Friday afternoon. Her increased level of tone and arrogance was met with calm and professionalism.

She was allowed to speak with the local employees; there to pickup her up and in so doing formulated a response to send back to the corporate office. Funds were wire transferred ($150.000USD) to resolve this matter, however it was several days later before she was allowed out of the security holding area.

In the meantime, a new shipment of parts (also hand-carried) was

shipped and delivered to the computer center. A policy was generated to clear all parts shipments through the division procurement officer to declare the means and amount approved for future shipments.

Sniper Shots

Saudi Arabia Oil Field

Think about this, a day at work with normal issues to resolve, dodging sniper bullets, jumping from multi-story structures to the ground in emergency evacuation moves, hiding in closets; trashcans; or anything large enough to house your body.

After trying to get to the emergency escape line (on a Saudi oil Derek) a worker used a red rag to cling to and ride a support cable down to the ground. Unfortunately, the friction burned through the rag while he was about 30 feet in the air. The crashing plunge he made to the ground was actually to his advantage as the sniper had just repositioned and was only another shot or two away from hitting him.

Keep in mind that the weapon of choice was an AK47 or similar fully automatic assault weapon.

Once home (Singapore) the native Hindu seemed normal as we sat around the dinner table.

Yet, an event occurred that really made it obvious that he was still recovering from believing that he was going to die that day at work.

The event was the taunting of him by his brother of which had dropped a spoon to the floor. The brother screamed, wrapped his face with his hands, and jumped to and fro knocking over the chair and crashing into other people and household objects.

After a few minutes of some rather “sharp” communications, we all began laughing as the attention moved on to his leapfrog dance.

Clearly, he was in no condition to return to the Saudi oil field. He remained home through the next year and finally began working at a small business in Singapore.

Earth Quake (The Big One)

San Jose CA

A computer service & repair partnership was running along between Houston, San Jose and Tokyo.

One day while on the phone (Hou-San J.) a series of wild calls were made the voice that had been muttering part numbers had stopped. I dropped the line after waiting a few seconds and then recalled a few minutes later.

The remote voice did return but was exceptionally out of character. Suddenly a set of instructions began flying across the phone line, send a fax to this VP, send email to that Manager, don’t hang up the phone for any reason, send more faxes, etc.

Some time in the process of sending faxes to Tokyo I received a call from them indicating that they had no evidence of a problem in CA and were questioning my data. After explaining to the secretary that the call was real and that I demanded that she wake her boss (VP was taking a nap) she reluctantly did and relayed the fax.

Shortly after confirming to the frantic voice in CA that the Tokyo office was updated the phone line was dropped and did not return for about 3 days.

At this time there was no BCP/DR process even thought of, and so it was observed by the lack of knowledge that when a emergency call came in it was going to be real and that no pranks would be ever utilized in business communications.

Felony Theft

Houston TX

Paperwork, paperwork, who has the paperwork.

Well one thing is for sure, you best know who has the insurance papers when handling custom designed HiPPI to FIDDI interfaces.

It was a simple task of moving from one office to another. The move was just between one building and the next. It was Saturday and there was no traffic in the parking lot.

What could possibly go wrong????

Well, the traffic changed for about one minute.

Upon emerging from the new office, headed for the next load, a strange vehicle was discovered at the corner of the sidewalk, which was being used as a staging area.

A discussion pursued with the unknown men whom had collected several items from the “staging area” and politely proclaimed that they though the items were there for them to pickup as they were The Scrap Men. As best that it could be determined, things had been returned and they departed.

The fact that there was no license plate was further disturbing.

No problem, just turn in an insurance claim and get a replacement moving. Wright – NOT!

The vendor had been shipping millions of dollars of equipment around the US, for this project alone, and had no vehicle to relieve an expense of this nature. At this point I was shaking in my boots as I did not want to be stuck with this ~$50,000 problem. Fortunately, a compromise was made between the manufacturer and the vendor and a replacement arrived two weeks later at a mere ¼ the original value. Hmmm! Is this a good time to discuss discounts, now that the real value of the board is known?

Assumed, Direct or Indirect Liability is still LIABILITY. Always obtain proof of protection via hardcopy.

Offshore “Man Overboard”

US
East Coast

Seismic production work boat. Approaching hurricane headed from Atlanta toward Rhode Island & New England.

Seismic data cable is over 2-miles long. Lots of repair underway due to storm and leaks (kerosene) from data cable.

Workman slips and was able to grab lifeline rope. Workman was underwater the entire time while off deck.

Crew Chief properly notified Captain of “Man Overboard” status.

Engines were shutdown and all hands moved to their emergency stations. Workman retrieved, via safety rope, with new respect for horsepower and moving water. No injury beyond minor scratches and bruising.

The “rehearsed” emergency procedures proved successful.

Computer Equipment

Guarantee Cancellation

Freeport, TX.

A seismic boat was outfitted with multiples of millions of dollars of computer, navigation, power protection-regulation and other equipment for collecting and processing data.

The disk drive technology was that of a removable media pack. This 3-foot tall near “washing machine” sized unit was equally heavy to its mass.

The problem was that upon entering the open seas the manufacturer would cancel the warranty of this or any other equipment of theirs.

The concern was regarding a crash (r/w heads contacting the media surface) and subsequent replacement parts.

A unique method of using oversize Air Bag suspension mounts (under inflated) allowed the unit to operate in seas so high that the waves were clearing the wheel-house.

Though full liability was passed on to the user the unique mounting process allowed the disk drive to operate flawlessly throughout its multi-year journey.

Diesel Fire

On-board in the Red Sea

As with any vehicle a boat needs maintenance. This situation involved general diesel engine maintenance concerning filters. (fuel, oil & air)

An unexpected interruption caused the mechanic and other crew members to halt their current activities and move forward within the vessel. Unbeknown to anyone the last fuel filter connection “apparently” may have not been as tight as possible or perhaps the new fuel filter was faulty. In any event, a fuel leak began to cover the back-deck with diesel fuel. At some point the fuel came in contact with an electrical device or something may have fell to cause a spark that started a fire.

Before the wheelhouse detected the diesel smoke a crew member returned to the back deck and immediately depressed the fire alarm. Automatically an SOS signal was initiated and all ship members hit the deck in a full run. A water-hose was used to attempt to guide the diesel fuel off the back deck and away from flammable items. Keeping in mind that water, diesel and fire are not a mutual mix of elements the fire got worse.

A crew member announced the Abandon Ship notice and emergency procedures began. Lifeboats were launched, key items were secured from the Skippers cabin and wheelhouse and dozens of people now migrated into the Red Sea.

Emergency procedures worked, no injuries were incurred, the crew was fished from the sea by local fishing boats (what coast guard), the boat sank and the insurance company was not happy. The daily work procedure was altered to include a watchman (secondary to the fireman) during any procedures that could expose any flammable fuel or oil to the deck area to include all maintenance operations or unscheduled repairs.

Murder -

“Eye for Eye”

Justice

Africa - Lagos

The living quarters were located some ~30 miles from the general office and computer system.

The living quarters (quadplex) used the ground floor utility & storage rooms as homes for the family that serviced the quarters and the guards that protected the property and its personnel.

One weekend morning a small disturbance was noticed near the front gate. Upon returning upstairs, (breakfast time) the cook was questioned about the number of people downstairs. When questioned, the cook notified one of the expatriates (wife of computer processing crew member) that a thief attempted to enter the compound last night. She innocently continued asking questions and the cook continued to supply data. Mistaking the curiosity for interest the cook took the wife downstairs to see the results of the incident.

Upon getting downstairs, they exited the gate and approached the trashcans. As they approached the two remaining guards moved away wherein one returned to his post outside of the gated area. The cook (attempting to please) concurrently lifted the lid from two trashcans and seemed quite surprised when the wife passed out.

The thief had been caught and local justice was prevailed wherein the thief was killed, decapitated and quartered like a cow off to butcher.

By now the additional commotion stirred other members of the quadplex. Upon determining the situation, various people scattered (some calling Europe) and others helped to carry the wife upstairs. The next few weeks were overly difficult as production diminished with the wife & husband away from the office most of the time. Several other crew members also missed time and most were very vocal in their attempts to take permanent leave from their foreign post.

The attempts to leave and request for replacements fell on deaf ears, so the same request were issued back to the US corporate office. To the best of my knowledge, the company policy of reverting to the decision of the regional office is still active.

It took about a month to get the wife back to England and into proper psychiatric treatment of which lasted for several years and inevitable outlived the remaining time of their marriage.

The local police department stopped by, I believe the next day, to take a report.

Corruption and Bribes

Traffic Police

Africa
Lagos

One day several of us gathered to plan a Friday night-out. Apparently, the selection of the day coincided with some event seen on the news and so what the heck, lets go.

The news event turned out to be that the police were looking for a criminal whom was suspected to be fleeing from a neighboring city and headed in our direction. I failed to see the logic. Later I learned that the police would be setting up several police checks (roadside) to search vehicles in the pursuit of this criminal.

When Friday arrived, additional instructions were given. After taking an inventory of the money that each individual had we were instructed to place different amounts of money into different pockets, wherein setting ourselves up to draw amounts as needed when called upon as we paid our bribes to get through the police checks. I had to wonder about driving ~100 miles to spend a few hours at a Greek dance club (ballet or something).

As we returned, after midnight, we encountered the first of three police checks. The drivers quick description lead us to understand that the higher the officers rank the higher the bribe (or dash). The driver quickly estimated what we were up against and so we made our plans to produce funds.

This, the first stop was short and not very expensive as the Privates were the top dogs. By the time we reached the final checkpoint (third) one of the passengers was really freaking out. Our “fun trip” then started to lean toward a potential night of “hell in a foreign jail”.

We arrived at the final checkpoint and all had a sinking feeling. There must have been a dozen cops with one of them having several pounds of braid, silver & gold metals and looked more like a military general than an experienced police officer. Last minute instructions were for everyone to stay calm, do NOT speak unless spoken DIRECTLY to, and recount your remaining money. More effort was invested in keeping the panic level down.

The span of three officers talking with us of which included several of us having various weapons pointed at us and for enticement they would poke the driver, later myself as well, with a pistol (45 automatic) and an automatic assault weapon (AK47). The good news is that I think the “Thompson Machine gun” was inoperable.

Bottom line, they got our money, we had fun, and we got back to the apartment.

I stayed down stairs with the guards for a while and tossed down a few additional beverages. As I told them of the venture, they laughed. I was (finally) glad to be such a source of entertainment for them, behind that locked gate.

The point of this - ALWAYS know the RULES when in someone else’s BACKYARD.

Excellent electrical power for some.

Indonesia

The company assigned me that task of setting up a seismic data processing center in Indonesia.

Generally, we knew that the power was bad and that a UPS system would be required. At that time it was not known that we would be on the 9^th floor and that the only place to locate a UPS system was in the basement and within a small room that would have to be built and air conditioned.

Upon arrival, I made contact with the building management and their engineer. We made appointments to visit the building, the building designer, the local French company that provided electricity to the block and several surrounding businesses.

The feedback provided that power outside of the “block” (Indonesian government area) was undependable and out for hours or days at a time. The French power system was excellent and completely backed up to cover 110% of the needs of the block.

Specifying the correct size of the UPS system was somewhat like playing with perpetual motion in that you had to add power to compensate for air conditioning to compensate for the heat within the UPS room to compensating for the oversize wire to compensate for being 10+ stories away from the computer.

Other than a few incidents: the air conditioner being stolen from the UPS room (yanked from window from within a secure parking garage), the unit overheated from the AC thermostat going out in the summer time and “roasting” the UPS electronics, the system provided clean power for years.

Backup - never tested
Recover - never tested

Houston, TX

A local database service provider had backup/recover scripts written for their rather old (70’s vintage) computer. The company operated for over five years before exploring the avenue of upgrading their ½” tape drives to something more modern.

I visited with them and showed them a custom controller that could be adapted to fit their exact needs and further be compatible with their application software.

In order to perform our task we had to understand their current BU process and dataset volume. Within that exploration, it was discovered that their BU tapes were blank. They had apparently operated for at least 3-years since updating the BU script and had NEVER tested the output tapes as it was too much trouble to be their at midnight when the script ran. A simple error (data density setting) caused the script to error immediately upon initiation. The script did not test for error conditions and simply skipped to the rewind/unload phase after detecting the tape density.

The manager began mumbling something about being fired, never being able to work again, and a few other things before realizing that he was venting directly in my presence.

Backup’s that are not tested are avenues of disaster looking for the time to happen.

By not testing the process, the business liability grew from small to enormous (~$100 million).

The tape upgrade project and the manager were terminated. The new manager (two months later) claimed no knowledge of this matter or the need to alter their system or their procedures.