We Told You So - AVL Designs, Inc.

Posted

When I was creating this post and video, it was obvious that only one title would do: “We Told You So!” Please don’t mistake this for arrogance. What you are hearing is frustration. We have been telling people this for a lot of years.

I had to go out last week on a site visit for a client for whom we designed a variety of systems some 15 years ago. The design included dimming and lighting controls. We did video. We did audio. We did electronic acoustics. I don’t remember everything we designed for them, but it was a lot. Absolutely for sure, as we do with all of our clients, we always warned them “You are getting all this new stuff and, if you’re lucky, it may last 10 years.” Sounds dismal, doesn’t it? They just got new. They haven’t even had time to play with it and now we are giving it an expiration date? Well, if we are going to do our job right, we have to tell them that! With upgrades, and plain old “wearing out” a day will come when things will begin to fail.

So, as the end user you have to think about a lot of things. Number one, we always tell people when they get an audio console, DSP or a lighting console, which has memory capability, they have to make incremental backups to USB drives. And by “incremental” we mean at least weekly.

So, over 10 years time that will be 520 back ups! “That’s too much work” you say. Well, how much will it cost you if you have not backed it up and with time it, most probably, will fail and no longer be usable? The smart person who follows through and makes the backups will be able take that file and load it into another product of the same type. This will get you up running again.

That best-practice was the good way of looking at things, right up until Covid. The pandemic added a new urgency to those methods, but we’ll talk about that in a minute.

So anyway, let’s get back to the client I was just telling you about. First of all, the dimming system we recommended for the job was substituted to save money.

Sadly, they bought a product that we did not want to allow on the project. Unfortunately, New York State’s equal bidding process never takes into account serviceability and long-term reliability of equipment. Experienced consultants know the likely pitfalls and will guide their customers toward better decisions. Nobody asked us when the substitutions were made on this one.

So, as a result, the dimming system they have is now failing. To add to it, all of the lighting in their room is still incandescent because it was installed 15 years ago and LEDs pretty much didn’t exist at the time. (The ones that did exist were pretty poor.) This is now an auditorium where at any moment they could have a single-point failure, taking out all the lighting in their room. Any single point failure we mean the dimming rack control modules.

A typical dimming rack has lots of dimming modules and they just do what they are told. They get their signals from a brain ( control module). And if that brain fails, you are toast! All of your lights are nonfunctional.

That is one of those events that we were warning you about. We have always told people to buy the rack and buy a spare control module, as well. There are 96 dimmers in it. The chances of all of those failing simultaneously is relatively slim. The chances of the control module that runs the whole thing failing is a lot higher. So, please, just buy a spare.

At the time the cost of a spare control module was a few thousand dollars on a multi hundred thousand dollar installation. Seemed like smart move to us. You put it in a box, store it somewhere, have it available for a time, like now, where bad things are happening all over the world.

Of course, they didn’t do any of that. The other thing they did not do was make any backup to their audio console and, therefore, they weren’t saving backup files of what is in their audio console.

They now have an older digital console that is failing. It’s sad.

And there is the added problem of signal, which is how it gets from point A to point B. We’ll talk about that a little bit more in a minute. To recap, you have these single-point failures: the console dies, audio system is down, dimmer rack, front end dies. All are down.

As well as the digital signal processors that tune their line arrays. Same thing here. Speakers are going to last a long time. Amplifiers are easy to replace, and you don’t have to have the exact same amplifier that you had when you put the system in. The DSP is very specific and requires a file load. If the DSP fails all the precise tuning of the stem is gone.

They didn’t buy a DSP either because the idea of having backups and support is not something that school districts think about for auditoriums.

So, they have this equipment failing and now that it has become a critical situation, so they came to us.

They told us they had a $30,000 budget to do something but, here is a new wrench in the plans, there was Covid. With the supply chain debacle, most of the things that are needed to remedy their situation cannot even be purchased. There was the China problem, the people who went out of business problem, the “you can’t get parts for things” problem. So where it should have been doable, now our clients were faced with only $30,000 they could spend and $200,000 worth of problems.

With all of those obstacles in the way, it was time to figure out workarounds. And we’ve been figuring out some pretty crazy workarounds for people lately to help them get past some of these things.

Workarounds are nice, if there are any. Sadly, there will be no way out for a lot of people.

It is time to get back to some of the things we have been telling people for years and, from this point on, everybody should take this advice more seriously.

Get backup equipment.
You just have to buy it.
Do not delay.
Buy critical components.

Here is a real life scenario: there is an audio console with eight physical inputs on it and digitally up to 164 inputs coming in from stage boxes. I was out at another job site the other day where their stage boxes failed. That means that they have no physical connections at their console. They have stage boxes and they can’t get the stage boxes to talk to their console. (not all stage boxes speak the same digital language )

So those who have depended on only one way to get their system to work truly are in jeopardy. A catastrophe is lurking around the corner.

When the digital revolution began we would tell people “If you’re going to put in a digital console, leave your analog snake somewhere. You might want it.” I think I actually said, “Someday you might want this.”

Well, someday is now. If they listened to the advice, I could have walked in with some old whatever, like a Mackie 1604, and at least have gotten sound out of a system for them. Now they have no way to get signal from the stage to the booth because they wanted their analog snake gone. “It’s old technology. We need to get rid of that.”

Enough ranting about that. So when you look at the single point failures in audio systems, you’ve got stage boxes, you’ve got consoles.

Let’s say you own a Behringer wing, which is working at the moment, and I back up files. The question is can I rent or buy a wing if I have to?

So, the first thing you want to think about is using equipment that is available on rental from people nearby who rent equipment. But unfortunately, in the bidding market, they can’t use that as a parameter for why you do or do not accept a substitute product. So, you might end up with, let’s say a lighting console that fails and in the future your files won’t go in any console you can rent. Now, being that things are DMX, in most cases you can usually get things up and running if you happen to be using CAN that can be a little more difficult if your console doesn’t work well with the version of ACN that somebody may have installed.

But if you don’t have any of those options, you are just out of luck. You have no lights. So, at that point you’re getting out candles or you’re getting out flashlights, which is lots of fun.

In many cases the average high school we do could have like $1.6 million worth of new equipment going into a project. And the backup equipment they ought to own is let’s say $35 – $40,000 but they don’t buy any of it. And then something like Covid comes along, which no one ever expected. In this scenario, manufacturers may or may not even survive.

Now let’s talk about that a little bit because in the middle of Covid (in our world we are still in the middle of it) we have multiple installations going on right now that are stalled waiting for key products to come in that can’t be delivered.

So let’s just say you had a very specific audio console that needed to be this many inputs to work with these kinds of stage boxes and let’s just say the stage boxes showed up but the console didn’t. And the stage boxes are Dante and the console you were going to get was Dante.

But now you can’t get that. So, now you have to buy another console somewhere or get a console that is Dante but maybe you can’t find one. And now you’ve got cat 5 wiring, which is great, but you own a bunch of stage boxes that won’t work with your console.

In the current climate, one of the new things is a new wrinkle beyond just backup. It’s the “which standards am I going to work with?” wrinkle. So if I’m going to work with old school DMX, which to be honest I think in a lot of applications is smart, if you don’t need tons of channels. You won’t have to worry about a console specific ACN network and whether or not your system will talk to it, you can just hook up DMX and use anybody’s controller to get something to come on.

With audio analog snakes, they’re a great idea. Have analog backups running from your stage to your, your stage rack where, worst case scenario, you patch a bunch of XLR’s together and you can actually get sound out of things.

So, it’s kind of interesting looking back at the many years we’ve told people this and everybody ignored us and now we’re being faced with multiple people contacting us. Very recently it’s been starting to happen with “I got this problem” and “I’ve got that problem.”

I was just at one the other day, their digital signal processor on their PA system died. They never saved any of the files. So, they don’t know what any of the filters were when it was fine tuned by a bunch of experts years ago and they can’t get the DSP that was in their system anyway. Even if they had saved the files, they are still toast.

So if you own something that is using whatever filters, patching, whatever it might be, here is your homework (due immediately.) Create Hard copy Excel spreadsheets of settings. They’re a pain in the to do. Most consoles and equipment will not spit them out for you in any fashion. But having that information such as “how is my dimmer rack patched?” in a pinch is pure gold! How is the DSP set up in my system? Hard copies are paper copies. And I know that’s like it, it’s like, they’re saying this ? “You must be a dinosaur talking about paper.”

Trust me, it makes sense. I had one of our who clients contacted us where their DSP failed at a site down in Washington DC. We had to dig out a Windows 95 laptop to open the files, but we had copies of the files that they didn’t have. So we were able to send them a hard copy spreadsheet of all the parametric EQ settings, delay settings, all the, all the tricks that were used to tune their sound system so they could get a new DSP and get it up and running again.

Time to Be The Champion

Please, I cannot stress this enough. Be the champion. Backup, backup, backup. Back up equipment, data hard copies, these are the things that save you and get your venue running when everybody else is scrambling.