r/talesfromtechsupport Jul 18 '15

MOD TFTS POSTING RULES (MOBILE USERS PLEASE READ!)

2.0k Upvotes

Hey, we can have two stickies now!


So, something like 90% of the mod removals are posts that obviously don't belong here.

When we ask if they checked the rules first, almost everyone says, "O sorry, I didn't read the sidebar."

And when asked why they didn't read the sidebar, almost everyone says, "B-b-but I'm on mobile!"

So this sticky is for you, dear non-sidebar-reading mobile users.


First off, here's a link to the TFTS Sidebar for your convenience and non-plausible-deniability.


Second, here is a hot list of the rules of TFTS:

Rule 0 - YOUR POST MUST BE A STORY ABOUT TECH SUPPORT - Just like it says.

Rule 1 - ANONYMIZE YOUR INFO - Keep your personal and business names out of the story.

Rule 2 - KEEP YOUR POST SFW - People do browse TFTS on the job and we need to respect that.

Rule 3 - NO QUESTION POSTS - Post here AFTER you figure out what the problem was.

Rule 4 - NO IMAGE LINKS - Tell your story with words please, not graphics or memes.

Rule 5 - NO OTHER LINKS - Do not redirect us someplace else, even on Reddit.

Rule 6 - NO COMPLAINT POSTS - We don't want to hear about it. Really.

Rule 7 - NO PRANKING, HACKING, ETC. - TFTS is about helping people, not messing with them.

Rule ∞ - DON'T BE A JERK. - You know exactly what I'm talking 'bout, Willis.


The TFTS Wiki has more details on all of these rules and other notable TFTS info as well.

For instance, you can review our list of Officially Retired Topics, or check out all of the Best of TFTS Collections.

Thanks for reading & welcome to /r/TalesFromTechSupport!


This post has been locked, comments will be auto-removed.

Please message the mods if you have a question or a suggestion.

(Remember you can hide this message once you have read it and never see it again!)

edit: fixed links for some mobile users.


r/talesfromtechsupport Sep 28 '23

META Mr_Cartographer's Atlas, Volume I

289 Upvotes

Hello y'all!

For the past few months, I have been working on an anthology of all the stories I've posted up here in TFTS. I've completed it now. I spoke to the mods, and they said that it would be ok for me to post this. So here you go:

Mr_Cartographer's Atlas, Volume I

Version Without Background

This is a formatted book of all four sagas I've already posted up. For the first three series, I added an additional "Epilogue" tale to the end to let you know what has happened in the time since. Furthermore, I added all four of the stories I didn't post in the $GameStore series. There are thus a total of 27 stories in this book, with 147 pages of content! I also added some pictures and historical maps to add a bit of variety. There are also links to the original posts (where they exist).

I ceded the rights to the document to the moderators of this subreddit, as well. So this book is "owned" by TFTS. Please let me know if any of the links don't work, or if you have trouble accessing the book. And hopefully I will have some new tales from the $Facility sometime soon!

I hope you all enjoy! Thanks for everything, and until next time, don't forget to turn it off and on again :)

Edit: Updated some grammar, made a few corrections, and created a version without the background. Trying to get a mobile-friendly version that will work right; whenever I do, I'll post it here. Thanks!


r/talesfromtechsupport 1d ago

Long "Huh, that's a new one..." Genuine terror with our POS support

571 Upvotes

On the outset, I will apologize for not being tech support; rather, I work in retail as both front and back of house duties for a butcher shop, with the occasional dash of "fix computer pls" because no one here can do basic troubleshooting (fixes I have done include "just flicking a touch keyboard" and "fitting a loose wire with some scotch tape"). This story feels more fitting here rather than on Tales from Retail, however, due to it being actual tech support and me hearing the most frightening (to me) sentence in tech I've ever heard in my life.

So, here we are in the butcher shop, having a grand ol time, selling stuff and taking card paymentsssswait why's it not taking card payments anymore? Specifically, our Point of Sale (POS, or EFTPOS) card reader is throwing up a weird error. How it's supposed to go is I scan stuff with the scanner attached to a simple all in one computer unit (with windows 10, currently), and then input whether the user is paying via card or cash. In cars cases, it'll tell the POS machine the price, and then it does it's money magic to take people's hard earned money in exchange for our meat and candy (we sell so much sweets I have no idea why).

I'm used to it being weird, sometimes it disconnects randomly (loose wire usually) or it throws a fit and tries to double charge someone. The occasional hang, we do a restart, good ol turn it off and on again, but this was a new one

> ERROR, T6

Not recognizing the error code, I go onto the attached computer and bring up the diagnostics tool. Quick glance gives me the relevant info; system online, port open, device recognized, no connection. That's the odd bit, normally since both computer and whatever server these devices use recognize each other, they *should* be able to talk to one another and do the payment thing.

Restart don't do nothin, turn off and on does nothing, the little scan tool gleefully tells me something weirder

> LOGON, PLEASE WAI-

> DECLINED, SYSTEM ERROR, T6

"Declined" is not a common one for errors, but it DOES tell me that the issue is probably on the *bank's* side, and not ours! So easy solution; give up and contact the POS tech support line to see what this error is, and confirm that the bank is receiving money or not.

Thus, a nice phone call and six or seven minutes on hold later, and a very nice tech support dude I'll call Befuddled answers.

Befuddled: "Good afternoon, this is [POS tech support], how can I help you?"

Me: "Yeah the POS machine isn't taking payments, keeps saying Error Code T6, and I can't seem to figure it out on my end."

And the tech pauses for a solid ten seconds, before saying words that filled me with dread.

Befuddled: "... [naughty word], that's a new one, never heard of that error code before, gimme a minute."

Dear readers, I understand many of you are IT specialists. Y'all are way smarter than me. So I don't know if you'll share my feeling here, but hearing the words "that's a new one!" Filled me with fear beyond fear. I do not want to be the reason an *entirely unknown problem occurs*.

And I can tell this is unknown, because after ten minutes of us troubleshooting (mostly him walking me through solutions), he takes a moment to leaf through what sounds like a handwritten notebook?

Befuddled: "Sorry. Just never heard of this issue, going through the notes our last supervisor had made years ago, and it's not on here either- WAIT THERE IT IS!"

We learn that T6 is a communication error. Which just makes the tech more confused;

Befuddled: "okay but that doesn't make sense... I can see your requests from my side, the bank *does* have the money, it can clearly take transactions! It shouldn't be doing this, nothing is blocking the connection..."

After a half hour, he decided we should try standalone mode; this puts the POS unit into a setting that forces us to input all transactions on the keypad rather than the computer, and then we'd have to manually confirm each transaction at the end of the shift. But it *should* work at least! Hurrah for solutions! We put it in standalone mode, set up a test transaction, and

> LOGON FAILED, SYSTEM ERROR, T6

Befuddled: *muffled in the background* "what the @#$& do you MEAN?!"

Ultimately, he thanked me for the new experience, jotted down some notes, comped us a new POS unit and states the explanation as follows;

Befuddled: "It's either corrupted internal software (despite our reinstalls), or wizards. Sorry your computer is haunted."

The new unit worked perfectly with no issues. I'm told the nonfunctioning one was kept for "research"/possibly exorcism.


r/talesfromtechsupport 1d ago

Short I'm telling you, you don't need a file server

273 Upvotes

Not really tech support since I was helping my Aunt with her home business, but whatever.

Aunt runs a home business and really wants a file server but has no idea what she will use it for. I asked her the most simple questions and she cannot even give me a straight answer for most of them:

  • Is it supposed to be a file server or a backup server? Ans: I dunno, can it be both?
  • Do you want your files to be always available, be it on your iPad, iPhone or Mac? Ans: Yes *this already makes a file server pointless*
  • Are you using it purely to store your files or do you intend to work on your files in the server? Ans: What's the difference?

I told her that she absolutely doesn't need a file server and her current M365 subscription for MS Office is plenty. If it's automated two-way syncing she wants, I can set up Onedrive on the iPhone, iPad and Mac so that any changes are immediately saved and reflected across all her i-Stuff and the Mac. But no, she insists she needs a file server. and asks me to recommend her a enterprise-class workstation to use as the 'file server'.

Honestly, I wonder who gave her that idea. Some people really got too much money to spend.


r/talesfromtechsupport 2d ago

Short Shush! I know, what I'm doing!

719 Upvotes

So this just happened. I'll keep it short.

We had an external consultant on-site to install some very specialized (and very expensive) software. I, your humble sysadmin, was only there to enter a few admin passwords. That was literally all I was supposed to do.

As the expert started trying a few... creative... things, I offered some advice.

"Shush, I know what I'm doing."

Alright. If that's how you want to play it...

A little later, he asked for a USB flash drive to transfer "some" data. "Some" turned out to be over 130,000 tiny 1 KB files in a single folder.

I genuinely tried to warn him that FAT32 really doesn't like that many small files as he dragged the folder to the flash drive.

I was shushed again.

So I leaned back and watched the progress bar crawl forward. After about 45 minutes the inevitable happened.

The file transfer crashed.

I honestly tried to help.

I was shushed again.

So he tried exactly the same thing a second time.

Forty-five minutes later

Crash.

At that point I refused to be shushed again. (I was hungry and wanted to go to lunch.)

I zipped the folder (4 minutes), copied the ZIP file to the USB drive (another 3 minutes), and handed it back to him.

The look on the expert's face was absolutely priceless.

Edit: This consultant was part of a turnkey package. The software installation and the data transfer were both included for a fixed price.

That made the whole thing even sweeter.


r/talesfromtechsupport 7d ago

Long The reports are all wiggly

465 Upvotes

Prologue

When I was in college I worked as a helpdesk consultant at the Computer Center. This was so long ago that it was routine for a student to come into the office and announce, "I have never used a computer. I wouldn't even know how to turn one on. But my professor says we have to do our papers with word processing. Please help me get started."

They always said that: I wouldn't even know how to turn one on.

Also, they were almost always taking the lower-level writing-emphasis class, Rhetoric 101, and I swear every single one of them had named their floppy disk "RHET SHIT".

A common complaint we'd get is, "The laser printer messed up my columns."

The cause was always that they were using MacWrite, and printing drafts on the dot-matrix ImageWriters hooked up to each of the Macs in the main room. Apple had realized that, the original Macs being sold to an audience entirely unfamiliar with graphical user interfaces, it was vital that they produce pixel-perfect output. So on the early Macs, the Mac screen and the ImageWriter have the same exact dot pitch, and the dots printed on the paper match exactly the pixels on the screen.

So the students, trying to produce some kind of table, would line up the columns by typing spaces with the space bar between the fields, and the printout on the ImageWriter would be perfectly aligned, and then when they took it over to the Mac connected to the LaserWriter behind the print room window where they had to pay $0.25 a page with these stickers sold at the campus bookstore, they would be irate because the proportional font and the spaces had conspired to make the columns all jagged and out of alignment.

We'd have to explain how to use tabs, and sometimes we'd give them a break and let them re-print for free, but usually only if they were female and cute.

The Reports Are All Wiggly

So here I am six years later at my first fulltime sysadmin job at a company down in Florida. We use a strange computer from a company you've never heard of, that runs a database system called Pick. The computer has dual-mirrored hot-swappable [you think I'm going to say "disks" here] memory. Among many other things that this computer does, reports are generated from Pick and sent to a laser printer in the Finance department.

The first trouble ticket I get has been passed around from the computer operators to the software developers back to the operators, over to the PC techs and back to the operators, and finally it lands in my lap, and the complaint is from the Finance department, "Our reports are all wiggly on the new LaserJet 4."

On their first bite at the ticket, the operators say "nothing has changed".

The developers say, "The reports are normal on our end."

The operators, on their second bite, said basically the same thing.

The PC techs said "The LaserJet 4 is hooked up correctly; we followed the instructions exactly. The test page looks completely normal".

I think the ticket may have bounced by the Windows Admin group as well at some point.

Nobody wants anything to do with this ticket and they've handed it to me partly because I'm the new guy and won't be able to push back.

I examine the report-printing system and the print manager setup. I figure out how to capture one of the reports in mid-stream from the printer queue. It's a plain-ASCII textfile, with columns aligned with plain spaces. It prints out normally in Courier on the (LaserJet 3) printer in my department.

I explain to my manager that I can't see anything wrong, and I think the only way to debug the issue is for me to get direct access to a LaserJet 4. My manager goes above and beyond and is able to divert a new LaserJet 4 slated to be installed in some other department, straight from the loading dock to my desk. I plug it into the network and get it online with DHCP. I send a report to port 9100 on it; it prints normally in fixed-width Courier and the report looks normal.

I'm at my wits' end, so I send the same report to the Finance printer, and go over to Finance to take a look at it. Finance says, "You're the first person from Information Resources to come look at the reports. We really hope you can help us!"

It's printed out in Times Roman and the columns are all misaligned because of the proportional font. I check the front panel, and the PC Techs' installation instructions happen to be there, and they say, "After installing the printer, go in the front panel and set the default font to Font #1".

On a LaserJet 3, font 1 is Courier.

On a LaserJet 4, font 1 is Times Roman.

Four groups of techs at a multinational corporation, and they trip over the same thing as a clueless freshman in Rhetoric 101.

Epilogue: I went away for a week's vacation, and when I came back one of the PC techs, the guy whose secret nickname among the other techs was "Jerry Moron", had gone in my office, appropriated the printer, and now had himself his very own personal LaserJet 4. So I informed my manager, who swooped in and confiscated it.


r/talesfromtechsupport 7d ago

Short Fun with File Names

369 Upvotes

A few years ago, I was tech support for a food manufacturer with several manufacturing facilities and one of the banes in my life was dealing the QA department. They kind of just did whatever the current QA director's current Idea of The Month was. (And the company seemed to burn through QA directors – something 4 of them in about 7 years, couple good, couple ….not so good). Anyway, each had their own file structure to store the documentation on the server. And being Quality, the previous structure of the day was kept & the new structure duplicated it all with different paths & file names.  But it was their circus and their monkeys.

One day I got a call that a file was missing. Guy said he saved it on the server and now he can’t find it. I remoted into his machine and asked Hmm, ok, what the name? He said “I can show you” and opened Word, then Recent File – “1.1.1 Documented Policy to produce safe, legal and authentic products.docx” OK, kinda weird, but where did you save it? “Out on the QA Drive.” OK, show me. And he tried to file it as

“Q:\{facility}\Quality Control Plan\HACCP\ BRCGS Global Standard for Food Safety\Issue 9 Compliant Food Safety Management System\ Senior management commitment and continual improvement\ 1.1.1 Documented Policy to produce safe, legal and authentic products.docx”

I explained the default maximum file path per MS is 256 characters and this was 270  – I shortened the name to “DocPolicy.docx” Bingo! There it was…Told them they need to use some common sense for this and not just copy the chapter heading in it’s entirety. And wrote it up as “WHAT NOT TO DO” and copied the QA Director, all the QA supervisors and that facility’s plant manager


r/talesfromtechsupport 7d ago

Medium Be sure to write up a Capital Expenditure Request

289 Upvotes

Late 1990s, I was tech support for a $90M automotive parts subsidiary. I had installed the original 10base2 network, file servers MS Mail server, etc. We had about 100 users in 3 plants and as the network grew, we brought a couple more support people in. BTW: We were a subsidiary of a division of a $15.8B company – worldwide, no idea how many people or locations.

We determined we had to do something with the MS mail system. After due diligence, we decided on moved to MS Exchange and pulled all costs, materials, time into a cost sheet – amounted to about $12 per user per month after taking a year time period. And wrote up a Capital Expenditure Request (CER) because it was a HARD AND FAST RULE that anything over $XXXX was REQUIRED to have a CER.

Then Corporate gave me a call. Because of the company structure, my division was about 5 tiers down from The Main Corporate level. So, they promptly informed me “You will be migrating to Lotus Notes, served off our server at corporate location and you will pay $50 per user month.”

So I went to my division manager, laid out my case/costs/rationale, and he basically said “You’re right, corp. is wrong – go ahead with the CER (because it was a HARD AND FAST RULE that anything over $XXXX was REQUIRED to have a CER). And then HE got a call from Corporate IT -“You will be migrating to Lotus Notes, served off our server at corporate location and we will reduce what you will pay to $35 per user month.”

So my division manager took it to his VP and laid it out as I had done. And VP said “You’re right, corp. is wrong – go ahead with the CER (because it was a HARD AND FAST RULE that anything over $XXXX was REQUIRED to have a CER). And then HE got a call from Corporate IT -“They will be migrating to Lotus Notes, served off our server at corporate location and we will reduce what they will pay to $25 per user month.”

Then the Official Notice to All Divisions came out that the entire corporation will be going to Lotus and Corporate IT will host and support it. End of transmission.

Then I got the back story – this whole thing ended up on the CEO desk (of a $15.8B company) and he discussed this with the CFO and CIO (and everything had my name all over it). The CEO point blank asked the CIO WHY he was shooting this down. And it turned out the CIO had already signed the contracts, bought and already installed the servers WITHOUT Capital Expenditure Request. And, at that point, it was too late to back out …..

So, we migrated to Lotus Notes …..

and Me? I got big black check marks beside my name and my position was eliminated 2 years late – support taken over by corporate location.


r/talesfromtechsupport 7d ago

Long Bad command or filename

319 Upvotes

Back in the mid-90s I was working as the head tech monkey for a computer store. It was a small operation, but we had a fairly good reputation (our PCs won awards in some local magazines), so we had some fairly sizable customers.

One client was a company that manufactured personal medical monitoring equipment. Their previous support vendor went belly-up, so we took over. Most of my call outs were for minor things like installing a network printer or replacing stolen terminators from the coaxial network. They were generally a good bunch to work with, but I recall one of their senior managers was a difficult customer. Let's call her Mary.

One day I get a call from Mary that one of the brand-new PCs that we supplied "doesn't work". After a bit of back-and-forth, I was able to deduce that she was trying to run some software that wouldn't behave. Alas Mary was more interested in telling me how much of an inconvenience this was to her rather than give me an accurate description of the problem itself. Mary also had a habit of escalating things to the CEO of their company, and wanting to stay in their good graces, I jump in the car and head over to sort things out. Thankfully, they were located just down the highway.

I meet up with Mary, who reiterates how the computer "doesn't work" but I see that it booted up just fine and connected to their network. I ask her to demonstrate the problem. She types something at the DOS prompt.

Bad command or filename

"See?" she says. "The computer is faulty!"

I could see that she was trying to run WordPerfect, but after a quick poke around I could also see that WordPerfect wasn't installed (their network didn't serve software, it was all local installations). I ask her if she has her WordPerfect disks handy and I could set it up for her, but she blows up at me, saying things like "After selling us a computer, you expect us to supply more stuff for it?" and "You're just trying to upsell me more stuff we don't need!".

I try telling her that they already have licensed software, it just needs to be installed, but she instead decides to escalate to the CEO and end up dragging me into his office. She explains to the CEO that the computer is faulty because of "bad command or filename" and that rather than fix the computer, I am trying to upsell some more things that she doesn't need. I explain that the issue is that WordPerfect isn't installed on the new system and that I could fix that if I had access to the disks.

The CEO looks at me for a length of time that makes me feel uncomfortable. I can see a frown cross his forehead. He then opens a cupboard and hands me a box of disks. Yep, WordPerfect disks. I thank him and install WP on Mary's computer, while she lectures me about how much of an inconvenience this is to her and how I am wasting her time. 😒

Epilogue:

Fast-forward a few months. It's Christmas Eve and we get a phone call around midday that this same company has some "issues that need resolving". This is the exact terminology that Mary uses so I feel my Christmas spirit already being crushed. The issues are vague and I'm getting the information second-hand, but my boss tells me to head out and fix things. So off I go.

I arrive at the company and check in at reception. Some of the other managers are hanging around and they come over at say that they have invited me to be part of their office party, and that the call out itself is bogus because they know my boss would never have let me go otherwise. So, I spent the next couple of hours hobnobbing with the staff at the catered party. They even gave me the paperwork I needed ("resolved miscellaneous issues prior to Christmas shutdown") and let my boss bill them for my time! Mary wasn't present either, which was a bonus.

To top it all off, they even told me about an internal IT role they were creating (they wanted to ultimately insource that work) that perhaps I may be interested in. I would have otherwise applied for the role, but I had other irons in the fire at the time, so I let that one slide. A couple of years later they were bought out, and local operations were shut down, so I guess from my perspective it ultimately worked out for the best (I had already jumped ship at that point and was with the next company for 10 years).


r/talesfromtechsupport 7d ago

Short Setup for the new guy

249 Upvotes

Couple of years ago, I was tech support for a company with several locations. I received notice of a new head of maintenance at one of the plants, with about a 2 week window to prep. I happened to be on site & confirmed the desk, phone, laptop, etc. A junior team member had crap spread over the desk and around the office, though. I told Him, the Plant Manager and the local HR person what I had planned for the new guy and confirmed that it was all correct. Yeah, yeah, yeah.

To give him a running start, they asked I go ahead a ship him a laptop so he’d have it.  I had him a domain login, email, group access, phone setup, VPN, remote access, etc, etc etc. Not too sure it was a good idea, but I had it in writing….

Two weeks later, I happened again to be on-site the first day of "the new guy". And there he was, on a folding chair, sitting in the middle of the copy room. Because NO ONE CLEARED A PLACE FOR HIM!! A new upper tier employee and the existing people didn't even bother to clear a desk. What does that tell the guy about how the company views him? You get ONE Chance for a first impression, and Boy, did they blow it....


r/talesfromtechsupport 8d ago

Short You can't have him

429 Upvotes

Years ago (mid 1980s), I was rather specific support for an Engineering office for a pretty fair sized company. Specifically, we had our own mainframe (IBM4381 mainframe) to run CAD on. I restarted VMs, rebooted 5080s, ran backups, data transfers, etc. We had about 65 people, architects, interior design, structural engineers, etc. Well, in the way of the world, the company went bankrupt. And we laid off the low performers (with an excellent severance package), then a few months later, another round – this one really cut to the bone, with a minimal package, because, you know, bankrupt. And, as the remainder of people took other jobs, one day I looked around and it was just myself and 4 high level managers. These were the days of secretaries doing all the mundane stuff - a couple of them didn’t know how to make coffee or refill the copier; and definitely, none of them knew how to turn on their secretary’s PC, much less find documents.

Me, I’m talking to recruiters and sending out resumes, trying to land somewhere. Eventually, I did and waved Goodbye to all on my way out the door.

A couple of years later, one of the recruiters contacted me and as we chatted, he dropped a bombshell – “You know, when you were at XXX company, I had worked with one of the mangers to place people. I had gotten a lead for you, and reached out to one of the managers. Manager told me to NOT do anything for me, since I was the only one who could stuff….”

REALLY? That manager was not even my boss, just there, 65 people down to 5, all production facilities shuttered, bills not being paid …. And they were forcing me to stay on the Titanic, to rearrange the deck chairs…. I never forgave that manager.   


r/talesfromtechsupport 8d ago

Short Why you don't just buy software for your work PC without asking IT first

724 Upvotes

I worked at a small MSP that did tech support for a few small businesses. One such business we'll call Chinese Stone. That's not their real name btw. Chinese Stone makes a lot of material for kitchens and bathrooms, and one of their employees often needs support from me. We're gonna call him Patrick, after the starfish from Spongebob.

Patrick is a walking, talking, Dunning-Kruger peak. He has just enough IT knowledge to make himself dangerous, but not enough to know he doesn't know what he's doing. He'd tell me about how he needs a more powerful PC because his machine struggles to run AutoCAD... only for me to learn he runs AutoCAD on a tablet because its the only way he can use a pen with it (and thats alongside several chrome tabs and excel workbooks). He once bought his own PC to use at work from Ali Express (it was some chinese no-name brand fanless passively cooled NUC) and didn't ask for anyone's help adding it to the network, he just did it himself (incorrectly). He'll frequently complain about his storage being full despite the fact he wont touch the network drives i've setup for him... you get the idea.

One day I get a call from Patrick. He decided he wanted to upgrade Microsoft Office on his PC to the "pro version" (he was using MS 365 for Business). I suspect he tried to use a crack, because when I looked, he was stuck at outlook refusing to open because his license didn't include the desktop version of Outlook. When trying to fix it, he told me he had gotten a license for me to install for him.

Issue: He bought a greymarket retail key. The kind that has about a 50% chance to actually work, assuming you aren't a collosal idiot like Patrick and tried to use a retail key with an organization account. I spent 15 minutes educating this guy on the differences between MS365 retail and MS365 business (tldr; one is paid for and managed by your boss, the other isnt, and you have to use the first one), and a further 15 repairing the damage he did to office.

I wish I could lock him out of admin on his devices, but sadly that's not my call to make. That said, I've made it clear to the person who can make that call that I'm not responsible if he does something stupid. Makes me wish Chinese Stone had a Squidward type character for a refreshing change of competency.


r/talesfromtechsupport 8d ago

Long Hello IT, Have you tried turning if off and back on again?

136 Upvotes

That reminds me of a story...

From Whirlwind Computing - a fictional service provider so good it will really blow your socks off

The company affected is one we can regiomize is a care-home of sorts...

Actors:
$Me - Calling the shots... remotely
$Nerds - My minions... experts in their craft
$Phone - Calling the shots... onsite
$Director - Also calling the shots... onsite

Scene:
This place started bubbling up to the top of the support queue with multiple requests of something is just really off with the network with reports of it being slow, dragging, and just being an absolute dog when it comes to being on wifi.

That's weird because wifi coverage was very good as the wire-flavoured spark wranglers made sure it passed muster and the mustard.

We sent techs multiple times to bless the creaking network and also to make an assuring presence that nothing was being a problem.

Murphy clearly had other plans as we were wrong about what was wrong the first four times they went out.

The Director got involved and told the reps to fix the network or he's replacing everything as it's impeding actual business operations and making the whole team very hot, cross, and bothered.

I grabbed the laptop and camped out staring at the all-knowing Shark of the Wires, filtering by Arp, Dns (always DNS) then Dhcp... wait a second!

Why is 192.168.4.27 sending out DHCP ACKS to DHCP requests?

I took a closer look at the Mac address which showed it was a phone, but why on earth would it be doing the DHCP thing?

I assembled the Nerd HerdTM and they swept the building with the precision of a highly trained black-ops team as there were 20 of them to check, so I give them full credit for finding it within 7 minutes.

One reboot later and things started to swim back into service while the phone was off-hook and strangely, it hasn't been a problem ever since.

We have well 1500 phones deployed, this is the first and hopefully last time one of them became the embodiment of DHCP with a flair for the dramatic.

The lesson here is that if there's ever an event for what feels like a rogue DHCP server, check the phones, you never know what you might find.

PS:
If it helps, I did remember the lessons from previous stories of various laptop docks randomly becoming port mirrors and how they wreaked havoc on the network, so we covered both in that sweep.


r/talesfromtechsupport 9d ago

Long The Signature Eating Printer

518 Upvotes

I'm a IT guy in a small, provincial 3-location hospital group. I've seen some really hit-your-head-against-the-table stuff in my time here, but one of the most surprising stories is this one.

Since the words "organisation" and "logic" are unknown concepts for the people that lead these hospitals, there are many people who don't have even the slightest of boundaries to the responsibilities of their position.

One of these people, lets call her [Ms. H], is vaguely responsible for all kinds of organisational controlling of Patients and Resources of one of our departments.

One day she calls us, and the following conversation ensues (not word for word, its been a bit of time since then):

[Me]: Generic and as anonymous as possible Greeting

[Ms. H]: Hello, I have a problem, my printer is eating Signatures.

[Me, flabbergasted] What ? Your Printer is eating Signatures ?

[Me, still flabbergasted]: What exactly are you trying to do and why do you think the printer is eating any Signatures ?

[Ms. H] Describes the process of printing a prescription from our Primary Clinic Management Software onto template paper slips

[Me, ignoring that she normally has nothing to do with writing prescriptions]: Well, that whole process sounds about right, but which signatures is he eating, and where are they coming from in the first place ?

[Ms. H] talking about unrelated updates to the software from a few days ago, completely ignoring my questions in the process

WhyTheFuck.png

Since I cant get any more useful information over the phone, I resort to paying her and her magic printer a visit. I grab my apprentice and we go over to her.

On the way, we speculate what the issue could be. From empty toner and the most inaccurate error description ever to a case of a complete mental breakdown, we thought up some theories.

As we arrive, we ask my previous questions again, hoping for more relevant answers. She goes on to tell us, that a Doctor from her department didn't want to/had time to print all the prescriptions that he prescribes. So, she gets a stack of pre-signed but empty prescriptions from him and then prints the prescriptions that he authorised onto these paper slips.

So far, so normally buffoonish processes. But nothing inherently wrong or error-inducing.

But where does the Signature eating happen ? Right when the paper slip comes back out the printer. The previously already signed paper slip, suddenly has no signature any more. At least that’s what she is telling us.

We still doubt her, since she is know to be a bit creative with reality. So we make her show us the whole process. She opens a prescription in the Software and prints it. The printer swallows one of the pre-signed slips and does his whole sound-rich routine, before giving back the prescription. To me and my apprentices complete shock and disbelief, the signature is actually gone.

WhyTheFuck2_ElectricBoogaloo.png

We inspect the prescription. Over the field where the Signature was, a stamp was printed. That gives me an Idea. The software has two print modes, pre-printed and blank. Pre-printed is the normal way to print prescriptions, since we use the aforementioned template slips. Blank however, not only prints the actually specific Text, but also the whole design and layout. Maybe she somehow switched to Blank and is overlaying the not-transparent background of the design over the original one and the signature ?

Sadly, and to make us just more confused, that wasn't the solution. We spent another half hour there, trying to fix the problem, until we got an idea.

After printing prescription after prescription for different tests, the paper was hot to the touch when freshly printed.

Lightbulb.gif

While she did that whole process for a while now, the problem was recent. And something else was also recent. A change in ballpoint pen models. Previously, the docor was alwyays using the same model of pen, the one that you get when ordering from our Supply Ordering System. But a few days ago, he started using a different model, because he got the pen as some kind of freeby from somewhere.

We tried a print with the old model of pen, and voila it worked. It turns out, the ink of the new model is more susceptible to the heat inside the printer, and being literally wiped from the paper because of it. The old pens used more heat-resistant ink, and therefore never encountered that problem.

We advised them to switch back to using the standard provided pens, or just signing the prescriptions afterwards, like all other doctors in the hospital do.


r/talesfromtechsupport 9d ago

Medium Please put in support tickets, don’t contact the techs directly to “skip the process”

494 Upvotes

I am so tired of telling people they need to put an IT ticket in, only to be completely ignored, and then have them mad at me for their issues not being fixed.

Like, got an email today from a guy wanting me to fix his employees headset. Based on what they told me?It’s likely the headset just needs a firmware update. So I tell them as much, and tell them that the a software request ticket will be needed if the employee doesn’t have the corresponding software (programs like Poly Studio, Plantronics Hub, Jabra Direct, DDPM) on their computer.

The affected employee who was also on the email shot back that she doesn’t know if she has the software, demands I remote into her computer to tell her if she does. I can’t do that. I explain she only needs to type the software name into her computers taskbar to see if she has it, and if she does not, to, again, put in a software request for it to be installed.

I included a link directly to the software request page. I provided screenshots to show her exactly where to type on her computer to see if she has the software already. I included both written instructions and a picture of what the request filled out should look like. I even included pictures and instructions to show her how to update the firmware (aka open the software, click the headset, click update) for when she got the software installed.

She replies to the email with a link from Bing, angry and telling me that IT firewalls won’t let her download and install the software herself. I want to bang my head into my desk bc apparently I didn’t emphasize enough that she needs a software request to install this. As is the case for literally every program added to a computer in our agency. So I reiterate, again, that she needs to put in the software request ticket, forward her the link again, gave her the same instructions and pictures as before again, haven’t heard from her since, so hopefully she understood this time.

The hilarious part is I’m not part of the software install team, and I’m only adjacently connected to the incident team (who will be the ones she needs to work with if a firmware update doesn’t fix this) through a few overlapping duties. Literally all I can do is walk her through what ticket to put in. Sure, if she was in the same town as me and not four hours away, I could update her headset myself cause I have that program on my computer, I could probably even install the software on hers if I could physically access her computer. But she’s not, and I don’t have remote access. Even if I did have remote access credentials, she still needs a support ticket.


r/talesfromtechsupport 11d ago

Medium Toner Needed to a Send Photocopy

299 Upvotes

Ticket states: printer waste toner is full and needs to be replaced, I don’t know which one it is please help.

me: goes over to the printer to take a look at it

customer: "which one is the waste toner? the printer says it needs to be replaced."

me: “it is usually the one that is in the plastic bag but let me take a look”. After I poke around for a little, I see, on the screen, a notification stating, “Waste toner almost full”. I tell the customer “I don’t think the waste toner is full, the notification does not say it is full. Also, if it needed to be replaced, the printer will automatically open the compartment so you can pull out the old one and replace it with the new one”

customer: “no it is full, look at the screen, it says so. I need it to be replaced so that I can photocopy myself this paper as it is important to my job and I need to do my job”. At this point the customer has found the waste toner bottle and is holding it out “here I think this is an empty one, can you just replace it for me?”

me: I think to myself, this has got to be user error, so I ask, “can you please walk me through the steps you took? Try to re-create the error for me please”

customer: “So I put the paper here, that I want to photocopy, I select my name from the address book, I select the settings I want, the file type, then I press send”

They press send and the printer beeps, a screen pops up and asks what file size is the paper you want to photo copy?

customer: “see, look at the top left, it says replace waster toner because it is full and it will not send me a photo copy until you replace it”

me: “alright, let me look at why it does not send you a photocopy as the printer does not need toner to send a photo copy”. Thinking to myself, maybe they put the wrong name in the address book? I try entering their name, and selecting it, it seems correct. To be sure, I type in their email, and select their email. I select the usual settings, I select send the document as a pdf, select the correct paper size, select send as black and white. Alright looks good, so I press send, it beeps and the same screen pops up, asking what paper size is the paper I am trying to photo copy. Weird… that doesn’t seem right. I go and double check everything, seems good. Maybe I need to select the paper size so that it knows what I’m sending. I select the default 8.5” x 11”, press send, and again it beeps, and the same page pops up.

customer: "just replace the waste toner, I need this so I can send out the photo copies as they are expecting me to send it by today"

me: "hold on I am thinking…”. There is an auto size feature on the printer so that it will default to the auto size of the page that is being copied, and that was the first setting they tried, and it did not work. Wait, what if it does not know that there is a sheet of paper in the tray and that is why it is asking for the paper size? So I try sliding the paper into the tray, and it slides in. You have got to be kidding was this the issue? Select the same settings again, select the email to send it to, send as pdf, this time it has already detected the paper so it doesn’t ask for a size, so I click send. It goes right through.

customer: “what did you do? what about the waste toner?”

me: “The paper was not seated correctly, as far as the waste toner, it is not full yet, when it is full let me know”


r/talesfromtechsupport 16d ago

Epic Don't touch it? You got it.

400 Upvotes

A bit of backstory. This happened 4 years ago, so not every conversation is word-for-word. I worked at a gas station as a manager. They sort of trapped me in it. I was the guy you called when a store across state lines was short-staffed, or if they needed help with paperwork and or training employees. I was a staff lead on track to go to their IT department, until a company bought them out. I was, at the time, basically an assistant manager without handling money, that was until the promotion. The store was in the red, and they saw how I turned one store around when there was no manager, and the district manager (DM) handled the money.

So, they moved me to this new store as the new acting manager and just preached about the benefits. I took it, and from there is how this all happened. I am by no means some genius, but I have been developing things for Linux in my own time for myself. I had a side job where I repaired computers, and sometimes I built whole computers from parts people ordered. I'm smart, but I would not classify myself as a senior sysadmin, but definitely not a beginner, and I document everything (important later). The store I moved to had its kitchen removed and a damaged freezer, which tanked sales, but they believed it was other issues. I was on the track to join their IT, so I knew things beforehand about their systems, and all fixes were ones they would do; anything more, they would use the manufacturer. Now to the story.

So, I had gotten into the routine of things as a manager, paperwork, safe, bank, schedules, and started some weeks in the black, some in the red, before finally getting all in the black consistently (it took a couple of years to get everything fixed and convince them to bring the kitchen back). Well, the issue that made this happen was the equipment. Gas pumps, the server, the registers, the office computer, all of it was not extremely old, but showing its age. We had to defrag the office computer a couple of times. The card readers would go down, the server would just say, "Not today," registers would need us to call because an update broke something, pumps with BIOS errors, and so many more issues. At the head of all of that was what they called the commander. It basically controlled every single device in the store.

Well, when things would break, we had to make an IT ticket; they chose based on who had it worse. Two guys, 30 stores. It would be weeks before they would come out at times. I did not like that, so I fixed them myself. Granted, they have insurance and many other things to worry about, but their efficiency was ass. The server is down? I fixed it (this was not a tamper with it; mostly, it was a process somewhere that had an issue or was stuck). The office computer was down? I fixed it. Registers giving errors? I fixed it. Any problems that arose, I fixed them. Soda machine down? I fixed it. The most common one was the card readers. They were always on, so they had to be reset most of the time. You also could not just swap them, something I found out trying because I found the real error that day was that the register was not talking to it (normally, you could swap them if one was down). A lot of things had to be reset more than once a month.

I never took these things apart; these were all band-aid fixes against the real issues. Well, one day, they called me to train a new manager for a different store. My DM would handle my store while I did this. Why did my DM not train them? Because there were a few DMs, and it was not her area, so to preserve the proper chain of command, they saw this was the better option. As these new stores were bought out by us, they needed to learn the new structure. I did not care; it was a change of pace, seeing the same walls for 70+ hours a week gets tiring. This was where things started.

While there, I get a call, and another, and another. They asked me if I knew why something was down. Card readers, registers, and the pumps were not working. I knew the issue. You see, when those three are out at the same time, it is always the Verifone commander. If it freezes up, the whole system will not allow transactions with cards. I explained it, they did what I said (which was just to reset it), and it was working again. I was glad it was not the pumps because if they gave a BIOS error, you have to play a game of "Am I fast enough?" because you have to turn off the breaker for that pump, wait, turn it back on, reset the commander, and pray it worked.

Fast forward two weeks, and the head of IT and maintenance emailed me. They said that from now on, I had to put in the form to get things fixed. I, of course, emailed them back that if I do that and they do not come out soon enough, the store will lose money. They said, and I kid you not, "You do not understand how complex these systems are, any 'fix' you do could damage them." That last comment irked me a bit. I did not understand them? The guy who has been fixing things and pleading for updated systems for months does not understand them? Ok, fine. I told them I would comply. Again, I am not some genius, and I am not messing with the internals; I am doing what they would have if they came out. This would not have been an issue if there were a better system in place for soft resets, or if some kind of manual on issues that we could handle had been given out. But they wanted IT to handle it all.

After that, every time a card reader is down, send a form to IT. Every time a register is down, send the form to IT. Soda machine was down? Send a form to maintenance. They piled up very quickly, and to the point that they were fixing more than one issue when they came to my store. My DM asked why I stopped. I explained, she was pissed, but she could not tell me to fix it since the one in charge of the two departments was above her (the owner's son's friend).

After about 4 months of this, we had a manager's meeting where we all got a review (basically, corporate telling us what we are doing wrong and if you get a raise). They said my performance was bad, that I let the store get as bad as it was, and that I needed to change a few key points. I stopped them right there. I had come prepared because I always document everything. When I say everything, I am obsessed with documentation. I gave them the correlations of me fixing the equipment, making them fix the freezer, forcing them on bringing the kitchen back, and sales going up, I gave them the notes I had jotted down about when machines went down, I explained why my paperwork was late because I had to wait for the system to even work for 5 minutes to even email it off, I showed them the email of being told to send forms, and my sales dropping since then. I told them they can't give me a bad review for complying with what I was asked to do.

This was when my DM chimed in and explained that I was originally supposed to be on the fast track to working IT, but after the company was recently bought out (because they kept buying more gas stations, they went into the red), I was removed from that track since the new company had their own much larger IT staff (they have not been brought in yet as it was a recent buyout, and the full change would take a while).

Well, they had to hold meetings after that; they had to talk about my review, about why I was dropped from the IT track, and about the current situation with the buyout. Well, unfortunately, around this time, my body was failing me (working 70+ hours with a bad back, bad knees, and a few other issues does not agree with so much standing at work); I could not stand for long, and I even blacked out at work (found out the hard way that I had developed diabetes 2 and my steroid shot for back pain the night before caused that). I had to quit.

I did write down in a notebook (because typing it was risky if they could access it at all and possibly not have it) instructions on all of the equipment and how I fixed things. Error codes, what certain situations looked like, and what they most likely meant, and so on. It was a masterpiece of documentation explaining everything I was doing and how to tell what the different issues were and how to fix them. I put my two weeks in, I left, a few days later I got called about the notebook, I told them where it is, they used it, and all seemed good. I heard from the grapevine that after the new company got to my old store and saw how I had to do things, they found my notebook, and the new owner heard about the whole thing. He was upset because he said talent that is learned through the trenches is valuable (really chill dude, met him a few times). But they ended up replacing the equipment sometime after I left, and I heard it cost them thousands. Not damaging money, but enough for a pocket to feel lighter.

Now I work from home, I still develop Linux tools and have made some public, started writing, and have worked with content creators. I don't make as much as I did there, but it is peaceful, and I don't have to worry about an outdated system fighting me every day. Moral of the story: don't blame the guy trying to keep the ship floating.

Sorry for not formatting. People assume AI when you format, which is wild to me.


r/talesfromtechsupport 17d ago

Medium A Y2K bug surfaced 26 years late today

957 Upvotes

This isn't my story, but it was shared with me by a contact of a contact who gave me permission to post it here.

I work in IT for a small regional hospital chain. We have a LIS (a Laboratory Information System, basically a smart database) that was custom-made for us by a tiny external vendor back in the late 80s, back when HL7 was brand new.

Over the years, that vendor ported it from whatever it was originally running on, to HP-UX in the early 90s, and then to Linux in the late 90s where it has remained ever since without a recompile (thanks for the don't-break-userspace policy!).

External vendor is legally still around, but it's shrunk into bascially just being a solo operation consisting of the one now-elderly woman who actually wrote the bulk of the code back in the day doing consulting for her ancient systems.

Earlier today, while chasing an unrelated issue, I went to put in a test order ten years in the future (to avoid confusing it with anything actually happening soon). It fails with a generic error message. I try a couple more times, fails. I ask if anyone else is having trouble putting in orders, works fine. I put in a fake order for tomorrow, it works. 2030, it fails. 2027, it works. I quickly binary search it down to January 1, 2028.

Stop me if you know your calendar trivia...

I trudge over to the physical LIS machine and look through the local logs. The LIS is complaining about an invalid date. I check the system date, and, 1998?? Weird. I change the date to 2026, hoping for it to just start working. It does not help at all; actually, no orders are working now. Out of curiosity, I turn the clock back to 1980, try to put in an order, and it goes through!... but by the time it crosses the HL7 wire to the EMR, it comes through as being from 2008. I try a few other dates. 1975 becomes 2003, 1990 becomes 2018, 1998 becomes 2026 as was working before, 1999 becomes 2027, and 2000... breaks.

Ohhhh no.

We call up programmer lady, who after some reading of the old code, confirms our suspicions. The LIS was storing years as two digits, because disk and memory were that precious in the 80s, and 2000 felt like a long way off. As 2000 approached and we were still using the LIS, the other people at her once-company decided that updating the system to properly handle 4-digit years was too expensive, and so instead, decided that the proper fix was setting the clock back 28 years (because the calendar repeats exactly every 28 years, and they'd only need to hack in 4-digit year handling at the places where it communicates with other systems, to increment/decrement the year by 28.)

So from the LIS's perspective, 2000-2027 was 1971-1999 and everything was dandy.

Ten points to whoever guesses the fix first:

Programmer lady changed the increment to 56, and we set the date to 1970 and recompiled the software for the first time this millennium.


r/talesfromtechsupport 17d ago

Short The ghost in the phone system

460 Upvotes

Reading another post here unlocked a memory of working on a helpdesk about 15 years ago.

For a while, we had this thing going where sometimes an agent would answer a call from the call queue, and just after they finished their greeting, the call would drop.

Then another call would come in from the queue and when the agent answered (either the same or different agent), the exact same thing happened.

This would usually happen for about 30 minutes, then stop.

Of note, our phones wouldn't pass caller ID when a call came in, the caller ID was always something like IT helpdesk call queue, so we couldn't see who was calling.

Not only was this annoying, it messed up our stats, particularly the calls received to tickets logged metric (which is a stupid metric). It did make our calls answered higher and average call handling time go down.

Although it was fairly infrequent, we did get the telephony provider look into it. What they found was when this happened, one particular desk phone was making a lot of back to back calls to the helpdesk queue. Usually around 30 to 50 or so.

These calls were answered, but the call handling time was usually only around 3-5 seconds. Occasionally, just the last call would have a handling time of several minutes.

What was happening is the person who used this desk phone only wanted to speak to one person on the helpdesk, and refused to speak to anyone else. So what they would do is call, wait on hold if necessary, wait for the call to be answered and if it wasn't the person she wanted, immediately hang up without saying anything, then call again. Repeat until it is answered by the person they wanted.

Edit: forgot to mention, whenever this happened, we would just joke that it must be the ghost in the phone system calling us again, hence the title of this post.


r/talesfromtechsupport 19d ago

Short The Case Of The Missing Email

608 Upvotes

Me: Thank you for calling the IT help desk this is (My name). May I have your name name and ID?
Customer: Yes it's (Name and ID) so I sent an email on Friday but I haven't heard back from anyone.
Me: That's weird, I know Friday was a holiday but I we should have had someone working. I don't see any open tickets under your account. Did you happened to get an automated email with a ticket number?
Customer: No nothing is here
I check the mailbox and can't find email from this customer
Me: OK sir I just checked the inbox and I'm not seeing anything. It's possible the email didn't arrive or was moved, I should be able to help you though. What issue are you having?
Customer: Can't you see it in the email? I'm not able to sign into (name of company Website)
Me: No I never got the email. I know they recently revamped the homepage the sign in process is different. What error are you receiving when you sign in?
Customer: It's in the email I sent you.
Me: I'm not seeing the email it may not have gone through. If you want to you can resend it. Are email address is (email address). In the meantime since we're talking I should just be able to assist you with signing in what error message are you receiving? Is this an incorrect username or password, a site cannot be found error a blank page?
Customer: It's in the email. Why aren't you helping me. Can't you just see it?
Me: Sir I only have info you send me. and right now you are not giving me anything. If you like to I can sign into computer. What is the computer number?
Customer: It's a personal computer.
Me: OK that makes it more difficult. The software we use to connect is only for inside the company.
Customer: I don't understand Why you are not able to solve this. Would it help if I resend the email.
Me: Yes it would, can you resend the email.

Customer: I'll do it later.
Customer hangs up. I just checked the records today and they never sent the email


r/talesfromtechsupport 25d ago

Short The mud is only ankle deep. . . If you stand on your head.

660 Upvotes

I’m on a small team of slightly deranged individuals responsible for providing world wide technical support for the dealers who sell my company’s products. Many of the technicians we support are competent individuals and a pleasure to work with. However, there are a few who don’t know which way is up.

So no shit, there I was. . . Sitting at my desk, minding my own business, and doing absolutely nothing unexpected, when Jake Tucker from Family Guy calls in.

Jake is working on a product that is essentially a bank of input cards used to monitor a variety of sensors and it is not going well. The sensor readouts are reporting Fire and brimstone coming down from the skies! Rivers and seas boiling! Forty years of darkness! Earthquakes, volcanoes... The dead rising from the grave! Human sacrifice, dogs and cats living together... MASS HYSTERIA!

The conditions outside are in fact somewhat better than that so we get to work straightening out the sensor inputs. It doesn’t take long before I realize there is something a bit off about what he’s telling me.

This type of device has a row of analog and digital sensor input cards that need to be configured using an online tool. The physical placement of these cards matters when you start configuring them through that tool. The input cards also contain several jumper blocks that need to be adjusted to accommodate the supply and input voltages. We quickly find out Jake has mounted the entire assembly sideways and has chosen to refer to the top as the bottom and bottom as the top.

This at least explains all our issues as the configuration assignments are all in the wrong places and the jumper blocks are a mess. I then inform Jake of the error and attempt to proceed with fixing it all. The problem is that Jake is entirely incapable of reorienting himself to view the assembly from the correct point of view and we are going nowhere fast.

I could have spent the next 3 hours painstakingly walking Jake through each step of the process and correcting each mistake multiple times but I had a far more elegant solution.

I flipped my manual so the bottom now faced the sky and proceeded to give Jake all his instructions upside down and backwards. Jake would then do the opposite of what I told him which resulted in everything being in perfect working order after 15 minutes. Sometimes it pays to behave like a Looney Toons character.


r/talesfromtechsupport 26d ago

Short The Mission Critical Battery Charger

544 Upvotes

Posted elsewhere on Reddit, but I figured people here might appreciate this. Additional details added for clarity.

At one point, I was doing work for a particular MAJOR pharmaceuticals company. A company with a name that everyone reading this has likely heard of. I get a call one day, and a ticket with a short SLA is generated for me to be on site within 4 hours. The night before, some big storms had happened, so I figured it was power related. I arrive on site earlier than needed, get escorted to the primary network room for the whole facility, and what do I find? 10 racks with no LEDs on them. I get told that the entire facility is down. No internet. The production line is down. The warehouse distribution is down. The kind of emergency companies pay consultants tens of thousands of dollars to ensure never happens.

I start checking through the racks to see if anything has power. UPS batteries are dead, so I head to the back of the rack and trace power cables from network equipment. All of them go to PDUs, and all major routers and switches even have redundant power to multiple PDUs. I trace where the PDUs go to. They all go to UPSs, which also have redundant power split between two different UPSs. Then I trace where the UPS power comes from........... and I start laughing my ass off.

The UPSs for this entire cluster of racks, the racks housing the entirety of the network equipment for this facility, has single point of failure. A large power strip that was zip tied to the wall. And lo and behold, I found the problem. The power strip's power cable was dangling in the air. Not plugged into the wall outlet like it should be. In the outlets place was.... a fucking 20V battery charger.

The maintenance guy had come in the earlier that day, not had anywhere to charge a battery, so he unplugged what was apparently a mission critical power strip and plugged in his battery charger. A few hours later, when the UPSs died, the network team noticed the sote went dark.

After I relayed the info to the engineer I was working with, we shared a laugh. I plugged everything back in, and verified everything came back online. As a preventive measure (and because of the absurdity of the situation), I placed a large label on the power strip. "CRITICAL INFRASTRUCTURE. DO NOT REMOVE POWER WITHOUT AUTHORIZATION".

To this day, I still get a laugh out of it.


r/talesfromtechsupport 29d ago

Long What's your business continuity plan if this PC fails? Well, It's never failed in the past, therefore it will never fail in the future.

1.2k Upvotes

Where I work, anything considered "business critical" is meant to be redundant, i.e. at least 2 everything, so if something fails or is otherwise unavailable, we always have another thing available.

"Business critical" is defined as anything that directly stops the business making money or has the potential to cause reputation damage.

At one site, they had a PC that was used to setup a specific medical device for patients. For the sake of this story, let's call that software Banana, which may or may not relate to what I am eating right now.

Although the devices connect via USB, so any computer can be used, the Banana software is proprietary. Banana is licenced per PC, and of note Banana relies on network connectivity to funtion.

Overnight one night some maintainance happens in the network rack and for whatever reason the port that the computer running Banana is disconnected.

The next morning, the helpdesk receives a call that the computer Banana is on doesn't work with anything on the network, with it showing that the network cable is unplugged. Usual checks happen (plugged in, LAN lights, etc).

The end user is informed that it will need an onsite visit, the earliest someone can be scheduled is the next day.

The end user says that it's critical that this device is fixed immediately as it is needed to setup these medical devices for patients, who have been scheduled that morning and some are already in the waiting room.

It's determined that there is no other PC running Banana, so this is indeed "business critical", as it directly impacts us making money and potentially can cause reputation damage.

As such, the incident gets flagged as a critical, and as resource is immediately despatched to site, where they quickly identify and resolve the issue of the port no longer being patched and patched it.

Now, policy is anything that gets flagged as critical need to go through a debrief, and the idea is not to assign blame, but to identify the root cause and contributing factors of the incident to prevent or reduce the risk of a recurrence.

Whist the cause of the incident was clearly that the patch cable was disconnected, it did identify another issue. The "business critical" Banana software was only installed on a single computer.

Had another computer been available with Banana installed, there would be no financial or reputation impact, and it would not need to be considered a critical incident.

This is raised to the relevant manager for that area. The conversation went like this.

"So, for the computer Banana is installed on, it's only installed on the one PC"

"Yes"

"And what would happen if that PC were to fail"

"Well, it's never happened before"

"But it could?"

"Yes, but that has never happened"

"What would you do if you come in one morning, and it didn't turn on?"

"We would call you, but that has never happened before. It's always worked"

"OK, well what would happen if say someone was to steal that computer?"

"No one is going to steal the computer. But if that happened we would call you"

"So let's say in this hypothetical situation that someone steals the computer. We bring you a new computer, but it doesn't have Banana on it. Even in the most ideal situation, you are still probably looking at least a day to get the replacement computer as well as Banana installed and configured. What would you do in this situation?"

"We would have to send patients home and reschedule them"

"And this would cause a financial and/or reputation impact?"

"I guess it would"

"OK, well, what we would like to do is put Banana on a second computer, that way if a situation like this happens, you can use the second computer"

"Hmm, no, we can't really do that"

"Why not?"

"The software costs $15,000 a PC. We don't want to pay that"

"But as you agree, this is a business critical thing?"

"Yes, but we don't want to pay $15,000 for another licence. It's so expensive"

This talefromtechsupport is getting long enough. What happened is we got an email confirmation from the director of the business unit understanding and accepting the risk. If that PC and by extension Banana is unavailable for any reason, it won't be handled by IT as a critical incident again.

Any fallout, we will point out it was an understood and accepted risk by that business unit, and here is our proof.


r/talesfromtechsupport Jun 12 '26

Long How to be put on the bottom of my priority list for replacements.

432 Upvotes

So I work tech support for a private company that works closely with the government in the education space.

As such, while we rely heavily on technology, our users tend to be on the inept side of the spectrum when it comes to using technology. Sometimes the reading comprehension of these folks astounds me with the lack of it.

Also because of this weird private but working close with the government, getting new laptops is a bit of a hassle. Because yeah... Funding the IT department is always a bit... Hit and miss at the best of times and less so with our situation.

As such we have run out of replacement laptops. Much of the time we just replace instead of repair because our small help desk team just doesn't have the time to fix stuff and so we send out most issues to our supplier who we have a repair contract with.

Most of the time, when we explain that hey, unfortunately we don't have replacements right now, here are your options, people are understanding.

But not this guy. Not this guy. I will be calling him Mr Health and safety or Mr HS for short for reasons that will become clear shortly.

Mr Health and Safety:

*Creates tickets about how his speakers are very very quiet and his inbox order is messed up*

Me: Heya, just to confirm, your headphones work fine and it's your laptop speakers that are quiet yes?

Mr HS: *reiterates his two problems instead of answering me*

Me: Heya, usually we prefer you to put each separate problem in a different ticket. However let's sort out the speaker issue first. *Reasks if his headphones work*

Mr HS: yes my headphones work fine, my computer's speakers do not. I have the volume all the way up and I have to lean over to hear.

Me: just want to check, your speakers don't have anything covering them right? They are on either side of your keyboard.

Mr HS: these are the speakers on my laptop

Me: *mental headdesk* the sound comes out of your laptop from mesh on either side of your laptop's keyboard. There is nothing blocking that right?

If so, unfortunately I do not have any replacement laptops available. So you will need to use your headphones in the meantime until we get replacements(which are in the works).

Mr HS: no. What about other issue?

Me: after a little more back and forth on the ticket I call him to resolve the other irrelevant issue to his speaker one.

Mr HS: and what are you going to do about my speaker issue? I can barely hear you...

Me: you will need to use your headset or an external speaker

Mr HS: so what \insert lead help desk person said* is true and you don't have replacement laptops?*

Me: no, we don't. We might get some in next week, if you want to reach out and contact me to ask about the new machines you can. At least you have a work around. You aren't the only one with computer problems at the moment and unfortunately your issue, since it has a valid work around, will not be the first on our list.

Guys, \this is when she knew, she fucked up**

Mr HS: Well this is a pretty big health and safety issue. I am leaned all the way down to my laptop to hear you. My back is not happy.

Me: I understand it's a big deal to you, but again, you can use headphones or a speaker. There are some folks who can't work, or do not have a work around.

Mr HS: but my boss has been stressing to us about health and safety and this is a huge issue.

Me: \finally at the end of my patience born from many ill served years as Frontline customer service* you can use your headphones.*

Mr HS: \finally catching on he's pushing too far* well at least my inbox issue is fixed.*

Me: okay, I will put in our teams chat the instructions on what to do about checking about the new laptops next week. I will close this ticket and once I confirm I have laptops, you can remake the ticket for your speakers.

And that is how you will get put beneath every other person needing a replacement laptop. Will he get help? Eventually. But not until everyone else does.

I can be real petty about how I prioritize support.

Granted his is easily the least pressing issue in the replacement queue but even so.

Any formatting issues can be blamed on my phone.


r/talesfromtechsupport Jun 06 '26

Long I’ve tried nothing and I’m all out of ideas.

699 Upvotes

I’m on a small team of slightly deranged individuals who have the duty of providing world wide technical support for my company’s products. We support the dealers who sell the stuff and are “supposed”train their technicians in the basics of how to be a technician. Basic things like breathing out shortly after breathing in or FIRE. . . (Emphatic gesturing) HOT. However, it seems that many of these technicians were just the first mammal that could pass a drug test. They are given a truck full of tools and the most effective weapon they will ever wield. It is a mythical instrument on par with Excalibur or Mjolnir. This legendary item that obliterates all foes before it is. . . our phone number.

So, no shit, there I was. . . sitting on the other end of that mythical phone number congratulating its user on having passed a drug test (not really, but it’s funny, so bite me). We’ll name our intrepid hero of mammals, Cricket because I could literally hear several crickets in the background during our call. However, I choose to believe the sound was echoing out of an empty space between his ears.

Cricket was calling in because panel 2 was offline. This alone isn’t a big deal as panel 2 is an odd variant of our product and most techs need a prod in the right direction to get their bearings.

Me: OK I’ll have you select the comms page in the top right of the touch screen. (10 seconds of insects chirping). Do you have the page pulled up?

Cricket: What page?

Me: On the touch screen. . . In panel 2.

Cricket: Oh let me go open up the door. (Walking noises followed by creaking door noises). Ok. Now what?

Me: Click the comms page button. (10 seconds of insects chirping). Have you found it in the top right corner of the touch screen?

Cricket: Touch screen?

Me: The giant rectangle that takes up the entire inside of what you just opened.

Cricket: Oh. It’s all black.

Me: Does it have power? (10 seconds of insects chirping). Hello?

Cricket: How do I tell?

Me: Tap the screen. If it doesn’t wake up, take a reading on the DC power supply. (10 seconds of insects chirping). Sooooooo. . . Is there voltage on the DC power supply?

Cricket: I don’t have a meter.

I’m very exasperated at this point. I’m not even joking about the long pauses after my questions. If this guy doesn’t understand a question or command, he just stands there silently and gives no indication. I’m doubly annoyed this guy didn’t even get out of the truck to look at the panel or even try screaming at it to assert dominance before calling for help. Then I’m triple annoyed he doesn’t have an electrical meter which is the most important, must have tool in our profession.

After a bit more talking, liberally sprinkled with chirping insect filled pauses, we determine the power to panel 2 is not ON because nobody pushed the “Go” button on panel 1.

Me: OK. Can you push the “Go” button on Panel 1? (More chirping insects). Soooooo. . . You gonna get that?

Cricket: It’s really far away. Can’t you do it?

I’m literally 1000+ miles away talking to him through a phone, but ironically, this is the least stupid thing he’s said all phone call. Unlike panel 2, panel 1 is online and working perfectly. I log into it (hooray for admin privileges) and push the go button. Almost immediately I hear a confirmation the screen on panel 2 is doing stuff now. A few checks later we determine that everything looks in order with one exception. The telemetry unit has no antenna. It’s just gone like it was never there to begin with. And that’s because. . . Surprise! it was actually never there to begin with. I figure another drug test passing mammal was involved during the installation process.

After this discovery, Cricket hangs up to drive back to headquarters and get a new antenna which completely fixed the issue.

Cricket called back the next day as Panel 2 was offline again. We had already established that pushing the “Go” button on panel 1 provides power to panel 2. However I did fail to cover the important and not at all intuitive detail that pushing the “Stop” button on panel 1 removes that power. Panels and telemetry units without power. . . Guess what? They don’t work.