Storage... another "hidden" cost of doing business...

Having been a systems admin for one of the largest imaging companies on the planet, I admit having a tendency to apply a bit of overkill, but I have a legit claim to do so... in that the owners failed to heed my warnings of having redundancy, and when we experienced a complete and TOTAL storage failure, they found out I wasn't BStin', and was right all along.

A couple of background points/rules...

In reality, unless data exists in 3 physical locations, it does not exist. (Of which, one copy should reside in a different geographic location.)

Data failure will generally occur at the worst possible moment, and is directly proportional to the value of the data.

All that being said, data integrity is not cheap, but is not as expensive as one might think.

With the advancements in drive technology, we're seeing HUGE drive capacities... while really cool... it also means that when you get drive failures... you're also looking at huge data losses.

There are data recovery services that can do amazing things... but the cost for data recovery is based upon the amount of data that is recovered. e.g. large drives are gonna cost large dollars to recover.

I would rather have a LOT of smaller drives that I can make multiple copies to, and reduce my data loss exposure.

Single drives, especially external units, are very much a convenience, but are also subject to heat and physical shock issues. (I had a couple of clients who have brought in drives that were fine when they left their house, only to have dead drives when they arrived for their sessions... sudden stops in traffic sent drives flying out of the seat, and into the floorboard.) So safety packing for transport is a smart thing to invest in... even if it's a cardboard box with foam or packing peanuts.

I prefer to look at multiple drive arrays... NAS, SAN and DAS.

NAS units are relatively cheap, but come at the cost of performance.

SAN boxes aren't as cheap as NAS boxes, but the increased performance pretty much accounts for less time copying files and tying up your system.

DAS storage is probably the more expensive of the three, but lend themselves to generally the best performance and easily integrate backup and archive solutions. (That 3rd off site copy)

Whether you choose to continue with individual drives (internal or external) or another storage solution, one thing I would seriously encourage you (and anyone else who's in your situation) to do, is to make a comprehensive plan of a backup solution that you can get into a routine of doing... no solution will be any good if you don't execute it on a regular basis.

Thanks, Max.

I am currently running a 2TB Glyph and 2x2TB NAS for storage and backup
As I'm doing more mobile work with the HD24 (24 tracks @ 44.1/48k), I need additional storage and backup.
I've also got a UPS for the NAS, the Glyph, the machine, and necessary peripherals.

Which is why I was thinking of adding a couple of 2TB internal AND external drives. Give myself a little more extra working audio storage, and even more backup capability.
Even the NAS systems really aren't cost-efficient right now... my 2x2 costs the same as current 2x1s I've been looking at.
Eventually I'd like to get something w/ 8-16TB in an array, but that will be a bridge further down the road.

So for now, I'm just looking at 2TB IHD/XHD drives... probably a couple of each.

My recording drives are a max of 500GB on purpose. These are the most stable drives. 1.5 TB is the max size I am willing to use for storage again because of stability and longevity.

I do this for ALL of the reasons Max discusses. I do utilize Glyph drives as well as NAS servers and a few other enclosures for non-critical recording sessions. One thing I didn't see when I skimmed Max's post, it is important to power up the storage drives periodically to keep the spindles lubricated.

Ben,

Here's the deal.. since we're running pretty much the same workflow...

I track remotes to the HD24, then move it to a local internal drive... so that I can work on it in SloStools. After that first session, I drag the session to 2 Glyph drives, then to the server DAS RAID 5 array. From there I copy the session on the main RAID 5 array to a second DAS RAID 5 where it gets backed up to DLT, in both incremental and archive backup routines.

It's not until I make the final mix that I'll delete the raw session from the HD24... just in case I run into an Oh Shit moment.

I run two internal 160Gb -10k rpm drives, that I hope to sometime in the "near" future, upgrade to 250gb or 300gb, 10k spindles. (They do fill up a tad too quickly)

Using a couple of external drives for alt safety isn't a bad idea... good on ya's for at least going through the exercise... but I would seriously rethink spindle size... 250-500 is a LOT less risk, considering the consequences of "oh shit factor" from heat, physical shock and performance.

And as John mentioned... exercising the drives at least annually is a good idea, although I have NOS spares that have come up just fine after sitting for 36 and 48 months. (But then too... these are indeed drives that have never been fired up.)

Max & John,

A bit of a detour from Soap's original question, but do either of you gentlemen trust optical discs for archival long-term storage yet? DVD-R or BD-R in particular?

In my other a/v ventures a Blu-Ray burner would be of some value backing up assets of video editing projects.
In addition to having the option of burning the final video project to Blu-Ray it would be a nice bonus if it was reliable enough for archiving.

I have always been very leery of DVD-R for archiving, since it doesn't take much of a scratch to cause errors on a disc so densely packed with black dots. Multiply that x5 for a 25GB BD-R and I'd be extremely uneasy about that whole thing unless they're somehow less prone to error. But stockpiling SATA drives isn't necessarily bullet-proof either.

I have an empty 9-bay chassis from a 7:1 duplicator sitting around. [needs a new power supply] Load it up with SATA drives for a RAID? Trust a BD-R (which I will need eventually anyway as I move to Hi-Def cameras)?

Thoughts?

I usually make an archive copy of the redbook disc but I don't trust that as true backup. Optical discs are pretty easy to mar or break unless you have dedicated lockers or trunks. They also take up significant physical space in relation to the amount of data they hold. This last bit is why I archive on magnetic tape drives. Eventually I am sure I will move to SSD for archive but I want to see some more longevity studies first.

Thanks John,

I'm not ready to trust SSD either. I've had CF camera cards go bad too.

I always archive numerous copies of the finished project, whether it's CD or DVD. But whenever possible I like to archive all the assets of the project - the original miniDV tapes pulled from each camera, the redundant recording medium (or two), the uncompressed 'capture' files of that footage, the Final Cut Pro files, the associated render files, and even graphic design for the packaging. That way I can 'unbake the cake' if the client wants to change something next year, instead of starting from scratch. As you well know, a couple of long video projects will fill your drives up pretty fast. Add several big, multi-layered, hi-res, Photoshop files, plus the uncompressed audio files and it can get out of control quickly.

Unfortunately, I've had a couple problems with hard-drives crapping out and lost a significant amount of data forever. But I don't recall having a DVD-R failure YET, just handling with extreme care and stored in a typical DVD case. But I know it's just a matter of time. I do need to follow Max's example and find another physical location for a third copy.

Bottom line I guess... as usual, there are no short-cuts.

Yeah... optical is still sketch in my book... not because of the medium... but because of the writers.

My understanding and somewhat dated research is that the writers are limited in the energy that the laser uses to burn the data, and over time, the media can actually "heal"... thereby loosing the data.

The optical media is subject to flex/warp just like a vinyl record... so as an archive material, they should be pretty stable in a relatively stable environment, such as a bank vault, but probably wouldn't survive a fire in a typical 1-2 hour rated home safe. Something like a 4-6 hour rated safe they might fair up... but typical tape backup would survive in a 1-2 hour safe... though it might need to be moved to a new shell.

Safe deposit boxes are the real deal... as long as the bank is relatively healthy, financially.

I highly recommend checking the the National Archives ( [[url=http://[/URL]="http://www.archives…"]The National Archives Records Management Information Page[/]="http://www.archives…"]The National Archives Records Management Information Page[/] ) and see how they do things... They generally plan for obsolescence transition of data every 10 years. e.g. they plan to transition all electronic data to a new archive system/media every 10 years, to ensure that the hardware is available to keep the data movable to the next generation of storage hardware.

Great posts about data safety here, esp Max's. Been nodding my head in agreement as to data safety, storage procedures, etc. FWIW, I do almost exactly the same as Max has listed; in my case I do my live tracking on external HDs into labeled folders. (I have several Western Digital 180 gig HDs in enclosures that run off my laptops or JoeCo black box recorder.)

Once a live gig is captured and comes back to the studio, I immediately transfer it into an internal HD on my DAW, and then go back and re-label/append the first copy as "Backup". (I NEVER erase the original copy until the client has a final copy (or copies) in hand, and the cheque has cleared the bank. ;-)

At this point, I also make a third copy on another external/inhouse/acrhive HD - which will be overwritten and added-to as work progresses on the version living on the internal HD. By the time the client is getting work-in-progress/temp copies for review, I've got three versions of the project, along with the CDr safety we make on the gig, along with a second SD stereo chip copy, done off the same 2-mix bus as the CDr.

Once the project is finally completed, with the client signing off on it all (usually once we're making CDr or DVDr copies of the finished master), then, and only then do I start erasing backup copies on the live-gig HDs.

Once the extrnal HD is full, I take it offline. Even with 1T and 2T external HDs now, I still fill them up every 5-6 months with various completed projects. THey're not used for production work, more for just bulk data storage. I get them offline ASAP, and put them in sealed clear plastic bags, with labels on them for fast/easy reference whenever I need to re-visit them.

Even so, with all of these precautions, there are STILL occasional heart-stopping "oh shit" moments (I'd say about 1x per year) no matter how safe we are, or what we do. Recently on a live gig, I had a CDr almost run out, and the backup SD chip go bad, all while the HD hiccuped on one of the multitracks. (How safe can one be, eh?) I'm inclined to think that some kind of supernatural power intervened to screw up just that particular moment, but we still got it all, albeit a stereo safety for a few seconds, to "cover" for the damaged multitrack.

I feel that one can never be too careful about this. As soon as your guard is down, or you get pushed to do something different than your normal, safety-based workflow; BAD things can happen. Don't get tempted to do shortcuts! Similar to what Max said: it happens at the worst possible time/proportional to the worth of the data".

Been there, done that, don't wanna ever do it again.

Joe, how are you liking the JoeCo? I've watched them since they first came out for form factor size alone and haven't been able to bring myself to cycle out the HD24XR.

TheJackAttack, post: 392343 wrote: Joe, how are you liking the JoeCo? I've watched them since they first came out for form factor size alone and haven't been able to bring myself to cycle out the HD24XR.

ditto's

Well, I don't know how long the HD24XR is still going to be around, aside from used on Ebay, etc. (looks like its no longer in production....?) One of severeal reasons why I passed on it and went with the JoeCo.

I'm 100% happy with the JoeCo. One of the best purchases I've made; It's nearly bulletproof/idiot proof. I wish I had a 2nd or 3rd unit sometimes. I'm still using the original firmware, although I'd like to update it; just haven't had time to do so, and worried that it might change an already great-working system.

I just use it to record the tracks live onto an external Fat32 USB Hard drive, then bring it back to the studio, copy it onto an internal drive, and off we go.

It really doesn't do all that much, aside from seamless tracking to a USB hard drive. You can use it for playback, but beyond that, (at least with the firmware I have), it doesn't do a whole lot more than its advertised to do; no ovedubbing, punch-ins, etc. I like the "Playback lockout" feature, even though I don't that many bands with it patched into a FOH or monitor console. I work with mic pre's split off the main snake, but it's still a great feature "just in case". And, I'm sure traveling bands that use it for live recording love the ability to playback the previous night's performance for sound check, even if the band's sleeping in.

You can slave/sync up to 3 of them together, I believe, for more tracks.

Sounds like another nice "gutless wonder" type device...

A device like the JoeCo doesn't need to do much, but what it does do, it must do well... and it sounds like it do what it do, VERY well.

Exactly, Max! thumb

The most important lesson I learned at NBC was redundancy. And you need backups to the backups. Master? Which master are you talking about?

I think you were having a little solar activity Joe? Otherwise you must have been working with my assistant? Mr. Murphy? He's always there to help. I told him he should help everybody. He must have listened to me?

And that's why God created us.
Mx. Remy Ann David