This is the 145th article in the Spotlight on IT series. If you'd be interested in writing an article on the subject of backup, security, storage, virtualization, mobile, networking, wireless, DNS, or MSPs for the series PM Eric to get started.
In the world of information technology, we as IT professionals are presented problems of all kinds in hopes that we, the superheroes of tech, might reach into that magical hat and pull yet another rabbit from the mystical depths of unknown origin. However, in our course of daily fodder, there occasionally exists a demon that defies nature and all things holy and continues to plague us as if it can’t be killed, in spite of our best efforts! It is this subject to which I wish to pay homage for a moment.
If you have spent any significant time in the field of troubleshooting technical problems, undoubtedly there will have been several issues that prove so illusive as to defy reason. Before dismissing them, I offer the following: There might be a macroscopic or external answer that you may not have considered!
Mammal problems
My first example comes from a story that did not happen to me but to another fine technician that I worked with many years ago. It is a great example of what I would like to illustrate.
Steve, prior to becoming a computer technician, repaired typewriters. (For the benefit of our younger audience, this was the device used to create documents prior to the advent of computers.) Steve’s duties frequently took him to our nearby Air Force base.
One particular situation involved an automatic typewriter that Steve had repaired on numerous occasions. In his many efforts to resolve the problem, Steve had actually performed a full intensive rebuild of the machine, which should have resolved even the smallest of problems. Finally in frustration, Steve went on-site and requested the customer to demonstrate the problem, as he could not get it to fail for him while testing. Now, the “problem” with this particular typewriter was that it would insert extra spaces frequently while the customer was typing. The typist, a little old lady (term of endearment!), with some attitude and great determination set out to prove that she wasn’t crazy and began typing in earnest.
Now at this point, even though this is a family-friendly forum, it must be shared that this particular little old lady (term of endearment?) was particularly well endowed. Upon completing a line, she hit the return. This required her to remove her hand from home position and, evidently, would break her concentration, which ultimately caused her to lose her place. Now given the information above, coupled with the fact that she had reading glasses on, it may be obvious, as it was to Steve, that all technical problems are not found within the machine. Sometimes they may be due to external influences. As this sweet elderly customer would look up, arching her head back to read from the lower portion of her glasses, she would instead find the magical extra spaces begin to fly by as if fired from an automatic rifle, but in truth, due only to those external influences. As she squealed in delight of her exoneration, she emphatically said, “See? See?”
Steve turned about three shades of red as he realized what could not have been conveyed verbally, but simply had to be seen to believe. Imagine yourself in Steve’s place, charged with resolving such a technical conundrum of magical proportions. Would your solution be to open your mouth and utter something that could get you sued for sexual harassment? Would it be to decline any further assistance since this problem was simply out of your control? Would you blame the problem on an act of God and dismiss yourself on the grounds that the solution is simply out of your range of abilities?
Steve simply said, mustering all the composure he could, “I think I can see the problem now. Please go get a cup of coffee, and I will resolve the problem while you are gone.”
When this little old lady excused herself to the break room, Steve simply walked over and raised her chair a couple inches. When she came back and again typed in earnest, magically the problem was gone. She was happy, and Steve excused himself and quickly got out of there before he busted a gut and brought shame to his name. (While this is absolutely a true story, you shouldn’t have laughed so deeply at the expense of such a wonderful little old lady! Shame on you!)
OK. I know it would be difficult to follow such a story with anything further of value, however, this subject is real and has education value that we can further explore.
Powerless in the face of desk
Another example is one that might be more common or of reasonable nature to occur to the average support technician in the field. While working at our local rocket factory, I had a valuable opportunity to work in a high-density, problem-rich environment with thousands of employees. I found that there were a very small percentage, but very real cases, of calls that happened on Friday and happened to be very trivial.
Upon arrival, the employee had left for the day since they could not perform their duties — their PC was not working. Upon examination, we would find things like the Ethernet cable partially pulled from the wall just enough to break connection but still appear to be plugged in. Without network connectivity, they were down and unable to continue. I’ve also seen where the power cord was partially unplugged from the CPU side, with the same results. I’ve had occasion to wonder if strategically deleted files causing a failure to boot might have fallen into this category as well.
On another incident, I was enduring a chkdsk or scan on a PC that would lock up occasionally, when the solution hit me. We had tried many, many things to resolve this problem over a two-year period of time, including replace the entire machine.
On this particular occasion as I sat at the desk I had nothing better to do than stare at the wall, or in this case, at the switch console (surge protector with separate on/off switches for each device) for this PC. This switch console had a neon light in each switch to see if it was on. I noticed, and happened to pay attention, to the fact that the lights would all change brightness and even dim almost all the way out at different intervals. It hit me that the power might be seriously fluctuating here.
I fetched my Fluke 87 Multimeter and used the min./max./avg. setting to monitor the power for the rest of the day. To my amazement, the power fluctuated from as low as 89 Volts to 120. The mystery was finally solved! Other machinery in the area was influencing the power, and when it would dip low, the PC would hang and require reboot. I have also seen other instances where microwave ovens or personal heaters have caused PCs to fail in various ways due to power fluctuation.
Sabotage!
The final example is the one that takes the cake, at least from my own experiences.
One of my direct reports was charged with automating a nightly export from our MRP application that would automatically fax orders to all our vendors to replace items used from our warehouse in that day. He had written a script that would perform the export, manipulate the file, and produce the faxes and automatically fax them. He had tested it many times, refined it and it worked pretty much flawlessly.
Since this was such a critical process that needed to be done in the wee hours of the morning every morning, we had another employee log in remotely to verify that it had been done, and if not, then come in and do it manually. We found it odd that there was a sporadic and occasional case where it would fail and she would have to come in. There was usually some grumbling the next day and she would usually go home early to compensate, or get overtime. My employee would troubleshoot and continued to try and resolve any problems to eliminate these failures.
After a year of this, he finally came to me in desperation and stated that he could not find anything wrong with the program and that it worked for him 100 percent of the time when he would test it. We had some suspicions but did not really believe there was any malice or foul play, but finally ended up taking a camera that we had and placing it in the server cabinet and directed it toward the screen of the utility PC that we had performing the script. We discovered that the employee was coming in remotely and interrupting the script during normal function!
We now understood the problem but not the reason. I assumed there was some ulterior motive — perhaps getting overtime or something else. Now that we had the equipment out and set up, I moved the camera to my own workspace for two nights. Upon viewing the tape, I was absolutely caught off guard when I observed the result! The lights were left off, but the door was open a crack and the light from the hallway illuminated just enough to observe the employee and a male figure enter and close the door. It was now very clear what was going on and that the problem was not in the script or the PC that ran it!
In this case, the employee gave notice before I had to act on this information and the problem solved itself, but not until after at least a year’s worth of consternation over the problem.
Consider external influences
So in conclusion, a wise troubleshooter will eliminate all the obvious and internal points of failure, but be willing to expand to the macroscopic and environmental possibilities that can create problems or participate and manipulate them. Don’t be surprised to find external influences pressing space bars or perhaps even worse! I’ll bet you didn’t know that information technology encompassed such a wide spectrum of technological issues, did you? Now you know!
What external influences have you observed in your own troubleshooting adventures?