Jump to content

where to store/backup all my massive raw video data on a budget


black
 Share

Recommended Posts

You won't find any cheaper solution than storing it on 3 or 4 TB hard disks, those are the cheapest per GB of data. Use those for backup / storage for different projects, connect and disconnect whenever you need to. If you got a tower computer, you might already have empty slots for disks, if not you will need to connect disks through usb/firewire/esata etc.

 

If you want more storage and more convenience in using it, use an old machine and add disks to that one and build a raid array - or do it the simpler way and get a NAS device, like a QNAP / Netgear NAS.

 

There aren't really any other options on a budget.

Link to comment
Share on other sites

I guess buying lots of external HDD is the only way for RAW

 

Well, if you're just looking for something to dump your footage to, yes, an external drive will be fine. But for archiving, backup or even working with the footage, it's less than ideal. I'd have an archive/backup copy of the files on a RAID 10 array and then copy files to a RAID 0 array for when I'm actually editing.

Link to comment
Share on other sites

I use bare hard drives and a dual hard drive dock to read them.
This way you only have one "reader" and one power supply, as opposed to having several external hard drives of different formats and with different power supplies.

http://www.everythingusb.com/startech-usb-to-sata-standalone-hdd-duplicator-dock-18699.html

You can also get these anti static hard drive cases that look a lot like a betacam tape case.

http://www.wiebetech.com/products/cases.php

Also, with the raw footage you might want to look into compressed raw formats for archival, or even some sort of prores depending how much you need to push the footage.
Link to comment
Share on other sites

Hmm, I was wondering, if it is based on experience what some "experts" are stating, that a hard disk will go bad if it's not in use and just stored away in a dry place.

I mean, I re-used a 20 year old scsi hd from 1992 (from an old mac IIci) the other day and the thing was still working. The has-to-be-in-use theory I think, only applies to SSDs because as far as I understood it they always need a certain charge to hold the data, and that charge will fade away over time. (How long? I don't know.)

Link to comment
Share on other sites

What I heard is that you should plug them in every year for a day or so, not sure how accurate that is, but anyway hard drives keep getting bigger and better, and it's very likely that in a few years time you'll be able to copy a bunch of old hard drives to a single one.
I've had loads of backed up projects on Zip drives, CDs, DVDs, Blu-Rays that I recently copied to a single hard drive. It's very likely that in 10 years I'll be storing all those hard drives into either a new format or an extremely fast and cheap cloud solution.
Link to comment
Share on other sites

I just fired up a couple HUGE SYSTEMS HugeMediaVault DualMAX (SCSI-160) arrays a couple weeks ago, that likely hadn't been fired up since around 2004 or 2005 at the latest.  That was one of those "fingers crossed" kind of moments, plus hoping I remembered the proper cabling and terminator setup, plus hoping that the SCSI-ID dials hadn't been fiddled with in all these years and two cross-country trips.

 

Back-up is an enormous problem.  Hard drives are the only cost effective solution that are also fast enough.  For the individual or for facilities generating tons of data.  Tape technologies haven't kept up and they've always been a bad solution anyway, both hardware and software wise.  At one time "experts" expected optical discs to be viable.  LOL.

 

There is no truly longterm backup solution for digital data.  This is one of the biggest issues for longterm film archival and one that the MPAA might have truly screwed the pooch on when that d-bag Jack Valenti was in charge.  Nothing has been invented that has a chance of outlasting archival film, properly stored.

 

 

edit: I can't imagine the hell it would be trying to find a working exabyte drive, or metrum, or the crummy software that was used to back data up back then.  Getting ahold of an old machine with a SCSI or SCSI-II interface and the right version of an OS.  That stuff was awful.  ZIP drives didn't last a year because the drives themselves were crud and they'd just start corrupting any disk you put in.  Is Iomega still around?  Oh look, they're Lenovo now.  I know never to buy any of their stuff, ever.

Link to comment
Share on other sites

My first and only guess would be linux.  Linux tends to be the best and most common OS choice for embedded, purpose-built device scenarios.

 

They just self-contained all of the points of failure you have with a dedicated PC + array.  By having the host PC inclusive to the arrangement they did remove the software+hardware related problem you get with a media only solution but you could very well be in a position where the media and data is good but the integral host PC has failed at any of the plethora of potential points that they do fail.

 

The saving grace with new drives and continually moving your data forward for fear of losing it (not only to failure but of planned obsolescence), is they get faster and smaller and cheaper, requiring fewer of the new to eventually landfill the old.  It's just a dreadful, repetitive and wasteful process.

Link to comment
Share on other sites

Yes. The Backblaze storage pods run Linux.

 

Basically, this is what I first suggested for you: running disks in raid-mode in a normal computer case. Linux and FreeBSD works great for software raid setups in raid 0, 1, 5 and 6 modes. They are also among the cheapest solutions, since you don't need the newest hardware and it might work with old hardware that you already have.

 

However, it takes more knowledge to run a setup like this though. Buying an off-the-shelf NAS is for most people the more convenient option, mainly because it doesn't require as much technical knowledge to setup the NAS, and because it is easy to replace a failed disk - it's usually indicated with a light or in some simple way in the maintenance interface.

 

Personally I've used FreeBSD file servers for over a decade now. My current setup is an array with seven 2TB disks, which give me 10 TB effective space and I can lose two physical hard disks in the disk array before I lose data. Read and write speeds over the network usually range in the speeds of 75-120 MB/s, so it's quite close to a normal local disk in the machine. For smaller files it isn't as fast though.

Link to comment
Share on other sites

Hmm, I was wondering, if it is based on experience what some "experts" are stating, that a hard disk will go bad if it's not in use and just stored away in a dry place.

I mean, I re-used a 20 year old scsi hd from 1992 (from an old mac IIci) the other day and the thing was still working. The has-to-be-in-use theory I think, only applies to SSDs because as far as I understood it they always need a certain charge to hold the data, and that charge will fade away over time. (How long? I don't know.)

 

The older a hard disk is, the more likely it is to fail, whether it is in use or not.

 

The zeroes and ones are stored on magnetic platters, and on a hard disk that isn't in use, the directions of the magnetization will weaken over time, eventually so much that your data can't be read properly.

 

And then there's the mechanical issues with a hard disk. The question is not IF a hard disk will fail, but rather WHEN. During my span of computing I've probably had around 40 hard disks. At least around 25-35% are dead by now. Some have had dead controllers, some have had the read/write heads hit the spinning platters, some of them stop spinning.

 

If you got important data, don't keep it in one place. And if you got important data, keep on moving it to new physical storage media every 2-3 years or so, depending on redundancy. The old storage media won't last.

Link to comment
Share on other sites

You can also try a Drobo, it's foolproof and simple to use, and you're supposed to keep replacing the smallest hard drives with bigger ones, and it takes care of everything for you. It never looked very professional to me, but it does look nice and simple.

Link to comment
Share on other sites

I've got four full-size ProMax towers with $16,000 worth (in 2002 dollars) of 10K RPM SCSI-160 platters that I'm half tempted to try to spin up now that I know the two DualMax RAIDs are still functional.  I just need to buy some earplugs first, lol.

 

What's sad is I've got a single half-height drive quietly spinning on my desk with the same amount of space as what I get with the ProMax array that I bought for $79.  It doesn't have quite the same throughput but it's really, really close.

 

All these fast SCSI systems were for the Cinewave-HD I build in 2002 (that also still works).  Uncompressed 422 used to be hard.

Link to comment
Share on other sites

You can also try a Drobo, it's foolproof and simple to use, and you're supposed to keep replacing the smallest hard drives with bigger ones, and it takes care of everything for you. It never looked very professional to me, but it does look nice and simple.

 

Don't know if I'd call Drobo foolproof... http://scottkelby.com/2012/im-done-with-drobo/

 

Personally I'd rather go with Synology, QNap or Netgear, they seem far more reliable.

Link to comment
Share on other sites

 

 

There is no truly longterm backup solution for digital data.  This is one of the biggest issues for longterm film archival and one that the MPAA might have truly screwed the pooch on when that d-bag Jack Valenti was in charge.  Nothing has been invented that has a chance of outlasting archival film, properly stored.

 

 

edit: I can't imagine the hell it would be trying to find a working exabyte drive, or metrum, or the crummy software that was used to back data up back then.  Getting ahold of an old machine with a SCSI or SCSI-II interface and the right version of an OS.  That stuff was awful.  ZIP drives didn't last a year because the drives themselves were crud and they'd just start corrupting any disk you put in.  Is Iomega still around?  Oh look, they're Lenovo now.  I know never to buy any of their stuff, ever.

 

I watched this film that has been linked on these forums as well about the transition from film to digital. Key point there was that big studios pack a working reader + computer along the archival medias. 

 

I've had poor quality CD-Rs brake in two years (my friends first ever music production batch). In the days of CDR-R's I personally used high quality CD-R's for my storage needs and those discs do still work. How ever I would never use DVD or Blueray disks for archival purposes. Unless it is a third medium beeing used along say tape and HHD's

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

  • Similar Content

    • By pintowar
      2 SSD Crucial MX100 of 512GB. Used but in perfect working order. See real photos and health and performance tests. MLC memories, more durable and of higher quality than the current models of Crucial.
       
      Price: 280 €
       





    • By CMerlin
      Hi, I'm new to this forum, after a while of reading I decided to put my first post.    I have always worked with the Canon 60D, and premiere pro CS6/CC. Now I bought Panasonic GH4 and i'm a little lost with the workflow.   My PC:   i7 3930k Asus P9x79 WS GTX 670 2GB 32GB RAM 1600   Storage:   C: 256GB Samsung 840 PRO (Os / App) 
      D: 2x Caviar Black 1TB RAID 0 (Media / Projects) 
      E: Caviar Black 1TB (Export ) 
      F: 256GB Samsung 840 PRO (Media Cache) G:Caviar Green 1TB (D: Backups)  H Caviar Green 500GB (Music, movies, etc.)    Dell U3014   I tried to work with 24p Cinema 4k. Goes well, but applying any effect, it is very slow. When I export to h.264 4k 24p, takes a long time, the CPU goes only to 27% and the RAM to 11GB ... I do not know why not use more.    Here are my questions:    1. Which would be the best workflow for working with 4K files? Proress, DNxHD ...? Proxies?      The final product will almost always exported at 1080p, only in some cases to 4k for now. So if there are two workflows would seem interesting to me also.    2. After the editing, workflow to move to color grading in Resolve?    3. Better combination for external backups?    Any other advice will be of great help.    Excuse my English.    Thanks        
    • By villafanephoto
      Hello, Everyone.
       
      I am debating whether to jump into the Canon 5D Mark III Raw video capture world, but as I have been researching the topic and equipment further I am now wondering what kind of real world computer performance in necessary in order to edit Raw video?
       
      I have a Mac Pro (2009) with 16gb RAM and four internal HDs (500GB for OS and three 2TB for storage).
       
      Until now I've only edited ProRes in Final Cut 7 on my machine, and I've never had a performance problem.  Moving into Raw shooting, however, I wondering if this will change my computer performance needs dramatically.
       
      I'm planning on editing either in ProRes or Cineform in Final Cut 7, but I wasn't sure if the higher quality files and larger sizes of shooting Raw would force me to change my setup dramatically?
       
      I will most likely need an external backup system to handle the larger storage needs, but I was wondering if investing in a RAID system is necessary?
       
      I was also wondering is there were any dramatic performance need difference between editing with ProRes vs. Cineform?  Does one require more power to run smoothly?
       
      I was also wondering where my money would be better invested in updates?  If I want better performance would it be smarter to add more RAM?  Or would I be better served by buying a RAID system?
       
       
      Any input on how others shooting with Raw feel is greatly appreciated.  Please don't hesitate to add comments and insights beyond the scope of my initial questions here.
       
      Thanks so much.  I look forward to your responses.
    • By villafanephoto
      Hello, Everyone.
       
      I am debating whether to jump into the Canon 5D Mark III Raw video capture world, but as I have been researching the topic and equipment further I am now wondering how much data storage and backup will be necessary for larger files.
       
      With estimated calculations of Raw footage being about 7GB per minute, I wanted to ask others who are shooting Raw how much storage space are you finding you need?
       
      Terabytes seem like they would get chewed up quite quickly, so how many are people working with?  4TB?  16TB?  32TB?  And for what size projects?  5min?  30min? 90min?
       
      Futhermore, data storage seems like it could vary dramatically depending on what editing format you end up converting the Raw footage into.  Up till now I've only used ProRes in Final Cut, but I am also seriously entertaining cineform.  It seems to have excellent feedback for a compressed file format, and would probably save space  compared to ProRes, as the Raw>DNG>TIFF>ProRes conversion for editing would require a lot of intermediate files, and storage space.
       
      Finally, I also wanted to ask about computer performance when shooting Raw.  Since I will probably have to invest in an external backup system to get an adequate amount of storage space, I have started to entertain using a RAID system.
       
      Do people find they need a dedicated RAID setup for editing speed now that they shoot Raw instead of H264?
       
      Or is a RAID setup really overkill at this point from a editing performance standpoint?
       
      Thanks so much.  I look forward to the responses.
    • By simondm
      Hi, 
       
      I was wondering if any of you uses any type of cloud storage to backup important video files in the cloud?
      I was using Dropbox but now that I've reached my free gigs limit, I'm not sure I should go for the paid version. 
       
      Especially as I recently discovered Copy, from Barracuda.
      They offer 15 Gb for free (+ another 5 Gb if you follow this referral link: https://copy.com?r=yx223v )
       
      I've heard people who referred a lot got up to 1 Tb for free...they say it's a limited time offer, so you might want to hurry.
       
      At the moment, I'm only saving the important files for every project. This is just in case my main backup (on a physical hard drive) fails at some point. You never know, better be careful!
       
      Is anyone else using a similar solution? I'd love to know! Thanks
×
×
  • Create New...