SR Posted July 23, 2018 Share Posted July 23, 2018 So I finally understood what RAID was thanks to this stupidly easy-to-understand video. My photography work (my current line) has gotten to a point where each projects (raw, PSDs) easily reaches 100GB to 500GB. Most of my video projects till now have done using the NX1 and GH5. And I am working on a few short scripts that I'll be shooting with the preordered BMPCC4K (first time working on raw or prores). With those in mind, I was trying to figure out a way to move and work on data effortlessly. So this video was tremendously helpful in understanding RAID, which has always intimidated me, perhaps because the information I found till date were so fragmented. I was thinking of getting the Synology 8 bay NAS DiskStation DS1817+ (Quad-core 8-bay NAS with AES-NI support, with over 1,179 MB/s reading and 542 MB/s writing performance), but I'm not sure what speed I'd need to handle 4k and prores. Also, I was curious on what RAID solutions each of you are using and how the experience has been. Quote Link to comment Share on other sites More sharing options...
Tone1k Posted July 23, 2018 Share Posted July 23, 2018 What system do you cut on? That Synology Nas is a good unit. Fill it with 8 drives in a raid 10 config and it will be plenty fast as long as you install a 10gbe card in it and use a 10gbe to thunderbolt adaptor. Using the 1gbe connections will not give you the speed you need. Do you need to share the data on it with other users? Robert Collins and SR 1 1 Quote Link to comment Share on other sites More sharing options...
kye Posted July 23, 2018 Share Posted July 23, 2018 4 hours ago, SR said: So I finally understood what RAID was thanks to this stupidly easy-to-understand video. My photography work (my current line) has gotten to a point where each projects (raw, PSDs) easily reaches 100GB to 500GB. Most of my video projects till now have done using the NX1 and GH5. And I am working on a few short scripts that I'll be shooting with the preordered BMPCC4K (first time working on raw or prores). With those in mind, I was trying to figure out a way to move and work on data effortlessly. So this video was tremendously helpful in understanding RAID, which has always intimidated me, perhaps because the information I found till date were so fragmented. I was thinking of getting the Synology 8 bay NAS DiskStation DS1817+ (Quad-core 8-bay NAS with AES-NI support, with over 1,179 MB/s reading and 542 MB/s writing performance), but I'm not sure what speed I'd need to handle 4k and prores. Also, I was curious on what RAID solutions each of you are using and how the experience has been. Remember that RAID isn't a backup solution. Yes, it protects your data if one of the drives stops working, but if your files get corrupted / you delete something important by accident / you get a virus / etc then it's still your only copy of the data. The backup copy should not be connected to the computer (if it is then it can also be corrupted by viruses or software malfunctions) and ideally it would be somewhere protected so if you get robbed or if there is a fire then you still have the data. The solution I'm working towards is having a RAID 5 system for the 'live' version of my data and archiving onto single hard disk drives as backup. I just bought a 12TB drive and that is my 'live' drive and I have a couple of 4TB drives as backups (ie, I backup projects by date and when the first drive is full I then backup to the second). My MBP isn't powerful enough to edit my 4K source footage so I make 720p proxy files and edit with these from the SSD in my computer so the speed of the drive with my 4K footage on it isn't a big deal for me at the moment. In terms of your data rates for the Pocket 2, BM lists these: Quote 4096 x 2160 CinemaDNG RAW - 272 MB/s CinemaDNG RAW 3:1 - 129 MB/s CinemaDNG RAW 4:1 - 97 MB/s Apple ProRes 422 HQ - 117.88 MB/s Apple ProRes 422 - 78.63 MB/s Apple ProRes 422 LT - 54.63 MB/s Apple ProRes Proxy - 24.25 MB/s 3840 x 2160 CinemaDNG RAW - 255 MB/s CinemaDNG RAW 3:1 - 122 MB/s CinemaDNG RAW 4:1 - 92 MB/s Apple ProRes 422 HQ - 110 MB/s Apple ProRes 422 - 73.6 MB/s Apple ProRes 422 LT - 51 MB/s Apple ProRes Proxy - 22.4 MB/s 1920 x 1080 CinemaDNG RAW - 66 MB/s CinemaDNG RAW 3:1 - 32 MB/s CinemaDNG RAW 4:1 - 24 MB/s Apple ProRes 422 HQ - 27.5 MB/s Apple ProRes 422 - 18.4 MB/s Apple ProRes 422 LT - 12.75 MB/s Apple ProRes Proxy - 5.6 MB/s Storage rates based on 30 frames per second. The highest there is for 4K60P which would be 272 x 2 = 544MB/s which the specs you quoted easily matched. It's a bit of a separate topic but I think that lots of people are going to get the Pocket 2, and after testing the various codecs (and seeing the file sizes!!) will realise that a lower quality codec is absolutely fine for what they're doing. To put it in context, YouTube 4K is something like 35Mbps, which is a hell of a lot less than the 2176Mbps of the Pocket 2. Obviously you want to record at a higher bitrate than YT streams at, but I'm not sure that you need 62x the bitrate!! SR 1 Quote Link to comment Share on other sites More sharing options...
Ivko Pivko Posted July 23, 2018 Share Posted July 23, 2018 If you're on Windows you can create a RAID as an internal drive with storage spaces, then have a Synology for backup in home and have backblaze to backup that internal Storage RAID drive. With backblaze you can backup an USB drive which could mean that in Mac you could leave the Synology and get a drobo as a backup solution, but for editing I think it would be slower. Quote Link to comment Share on other sites More sharing options...
SR Posted July 23, 2018 Author Share Posted July 23, 2018 6 hours ago, Tone1k said: What system do you cut on? That Synology Nas is a good unit. Fill it with 8 drives in a raid 10 config and it will be plenty fast as long as you install a 10gbe card in it and use a 10gbe to thunderbolt adaptor. Using the 1gbe connections will not give you the speed you need. Do you need to share the data on it with other users? It's a windows laptop (i7, 960M, internal pci SSD, etc). It's fine for video work I've been doing so far, but I'm planning on moving to a desktop by the end of the year. Will do on the 10gbe card. Thanks! 5 hours ago, kye said: Remember that RAID isn't a backup solution. Yes, it protects your data if one of the drives stops working, but if your files get corrupted / you delete something important by accident / you get a virus / etc then it's still your only copy of the data. The backup copy should not be connected to the computer (if it is then it can also be corrupted by viruses or software malfunctions) and ideally it would be somewhere protected so if you get robbed or if there is a fire then you still have the data. Good reminder. I'm definitely not planning on using RAID for 5 hours ago, kye said: The solution I'm working towards is having a RAID 5 system for the 'live' version of my data and archiving onto single hard disk drives as backup. I just bought a 12TB drive and that is my 'live' drive and I have a couple of 4TB drives as backups (ie, I backup projects by date and when the first drive is full I then backup to the second). My MBP isn't powerful enough to edit my 4K source footage so I make 720p proxy files and edit with these from the SSD in my computer so the speed of the drive with my 4K footage on it isn't a big deal for me at the moment. Have you run into any problems using RAID 5 so far, any disk failures? 32 minutes ago, Ivko Pivko said: If you're on Windows you can create a RAID as an internal drive with storage spaces, then have a Synology for backup in home and have backblaze to backup that internal Storage RAID drive. With backblaze you can backup an USB drive which could mean that in Mac you could leave the Synology and get a drobo as a backup solution, but for editing I think it would be slower. I've not looked to cloud solutions so far. How much data do you store on cloud backup? I imagine it'll be quite hefty for video. Quote Link to comment Share on other sites More sharing options...
no_connection Posted July 23, 2018 Share Posted July 23, 2018 Unless you intend to make use of beefy NAS for editing from you could run some local SSD storage for that and have NAS for actual storage or "safe keeping". RAID 0 of say 4 M.2 in a PCIe 16x card don't have to get crazy expensive. The downside is not having all the projects at the same place at the same time. So only current project being "online" at the time. I don't think many uses RAID5 any more, at least not in server. You end up always using two parity disks or more, so RAID6 If you gonna spend some real cash hop over to FreeNAS forum and have a look around. If noise is not a huge issue grabbing a used server and some disks for it might not be super bad if you find the right one. And when you have 64GB+ RAM for intelligent cache you get decent performance. The upside of using ZFS is data integrity and snapshots. So no more corrupt data and virus damage. Quote Link to comment Share on other sites More sharing options...
kye Posted July 24, 2018 Share Posted July 24, 2018 9 hours ago, SR said: Have you run into any problems using RAID 5 so far, any disk failures? I'm not on RAID yet - once I outgrow my current HDD then I'll need to go there. Just a single drive at the moment. 4 hours ago, no_connection said: I don't think many uses RAID5 any more, at least not in server. You end up always using two parity disks or more, so RAID6 RAID 5 means that if you have one drive fail then you're ok, but if two drives fail you lose your data. RAID 6 means that you need to have three drives fail to lose your data. In server environments I can understand that, but for video? Considering that if you have a drive failure then you just order a replacement drive and a few hours / days later then you're back to safety again, I don't see that as being too risky for data that you've already got a backup of somewhere else. You could even back up the array when the first drive fails as an extra layer of protection. Drives are quite reliable these days - every computer without RAID is only a single failure away from disaster but the people aren't running around screaming in panic! Quote Link to comment Share on other sites More sharing options...
webrunner5 Posted July 24, 2018 Share Posted July 24, 2018 7 hours ago, no_connection said: If you gonna spend some real cash hop over to FreeNAS forum and have a look around. If noise is not a huge issue grabbing a used server and some disks for it might not be super bad if you find the right one. And when you have 64GB+ RAM for intelligent cache you get decent performance. He is not kidding about the noise and heat also. A few years ago my son and I had like 4 servers running 24/7 in a spare bedroom. I am talking the big ass rack mount ones. The IBM ones we had had 16 fans each in them. There had to be 50 fans going at once, and they all had 7200 rpm SCSI drives in them also, 6 in each of them, not counting our Normal computers in there. It was pretty damn intense to say the least. You did not talk on a cell phone in there I can tell you that, or need to turn the heat on in that room in the winter. The electric bill was pretty intense also! ? I like Raid 10 and Raid Raid 50 the best. But it takes a lot of drives to do Raid 50. But Raid 5 is a good one also. None of them is fail safe. And you need hot swap drives to be even safer. I don't ever want to shut off a Raid configuration if I am having problems. It may never be able to get to the boot sector on a re start! And then as they say, you are in deep Do Do. Even a limping drive is better than, a holly crap, no drive detected warning one! I don't know, it might just be cheaper and safer to put all your data on 4K Blu Ray disks and be done with it. Lot safer media to archive with. They have like a 2200 year shelf life! They have 100GB ones out. Quote Link to comment Share on other sites More sharing options...
no_connection Posted July 24, 2018 Share Posted July 24, 2018 9 hours ago, kye said: RAID 5 means that if you have one drive fail then you're ok, but if two drives fail you lose your data. I think there is a decent article about this on the FreeNAS forum somewhere, but the short story is that with larger HDD the chance of data being corrupt or another disk failing while you rebuild the replaced one is too big. And since with RAID 5 during rebuild you don't have any chance to correct for errors or corruption it might very well fail or give you broken data. Quote Link to comment Share on other sites More sharing options...
dslnc Posted July 24, 2018 Share Posted July 24, 2018 Have been using various raid 5 nas in the past ( without hdd chrashes ) for backup going forward I only use raid1 enclosures (2xhdd) which give the security of having an exact duplicate should one of the hdds fail - no rebuilding needed for it to work. With hdd prices being quite low I am ok with using 50% of the total raid capacity as ‘security/parity’. And I am far more comfortable using big hdds with raid1 than raid5/6. A 2tb hdd can take xx hours or days to rebuilt in a raid5/6 setup. Imagine a 8 or 12 tb hdd... ssd are not built for raid5/6 either. Using those raid configurations with ssd will shorten the life considerably ( accordingly to raid experts) Quote Link to comment Share on other sites More sharing options...
slonick81 Posted July 24, 2018 Share Posted July 24, 2018 Random thoughts: 1) RAID should have consistency check procedure that you can assign on schedule (so it won't interfere with your work), set up priority (you may get some urgent work right in the middle of self-check), and get readable output (so you don't need to guess which drive became corrupted, at least). Most of cheap disk enclosures, as well as many cheap quasi-RAID cards don't have it and you'll notice data rot only when you fail to read it. 2) HW RAID consistency != FS consistency. I've seen some systems showing healthy RAIDs with badly corrupted NTFS/ext3 at the top. If you have a classic RAID that provides block device and OS managing your FS then better be checking both (RAID/block device and FS) from time to time. 3) FS consistency != data consistensy. Say "hello!" to network errors, software crashes and those funny encrypting viruses. It's not RAID's direct responsibility but better keep in mind. 4) Rebuild time. If we have classic RAID for video editing purpose (i.e. no database patterns with 4K random IO) plus HW RAID SoC or CPU/RAM for SW RAID with decent performance then rebuilding speed is limited by write speed of new disk (that replaces broken one). 5) RAID doesn't replace backups. I'm somewhat a ZFS fan. At work I have 120TB NAS under FreeNAS and Proxmox node for FTP and some random VMs, both running on ZFS - so far, so good. NAS was originally built 3 years ago and grew from ~40TB to current size and there are some projects that lived there all this time - no lost/corrupted data. Got 700MB r/w on 10Gbe according to BM utility, performance is mostly limited by HBA (LSI 2008 based SAS card - quite old design for tight budget). Quote Link to comment Share on other sites More sharing options...
dslnc Posted July 25, 2018 Share Posted July 25, 2018 Interesting article: https://www.zdnet.com/article/why-raid-6-stops-working-in-2019/ Quote Link to comment Share on other sites More sharing options...
kye Posted July 25, 2018 Share Posted July 25, 2018 16 hours ago, no_connection said: I think there is a decent article about this on the FreeNAS forum somewhere, but the short story is that with larger HDD the chance of data being corrupt or another disk failing while you rebuild the replaced one is too big. And since with RAID 5 during rebuild you don't have any chance to correct for errors or corruption it might very well fail or give you broken data. This is interesting and I can see the logic. In that sense RAID 6 is what you want. 14 hours ago, dslnc said: Have been using various raid 5 nas in the past ( without hdd chrashes ) for backup going forward I only use raid1 enclosures (2xhdd) which give the security of having an exact duplicate should one of the hdds fail - no rebuilding needed for it to work. With hdd prices being quite low I am ok with using 50% of the total raid capacity as ‘security/parity’. And I am far more comfortable using big hdds with raid1 than raid5/6. A 2tb hdd can take xx hours or days to rebuilt in a raid5/6 setup. Imagine a 8 or 12 tb hdd... ssd are not built for raid5/6 either. Using those raid configurations with ssd will shorten the life considerably ( accordingly to raid experts) I'm being really pedantic here (and may well be telling you what you already know) but RAID 1 will need rebuilding if one of the drives fail. Yes, the disk will keep working in terms of having your data still available, but if you have RAID 5 and a drive fails it will also have data available too, so the RAID needs to be rebuilt in both of these situations. Rebuilding is the process to return the array to a state where it has one or more redundant disks. In terms of SSDs, do people run them in RAID? Is this a thing that is commonly done? I would imagine that for large data rates you might have the drives in parallel without any redundancy perhaps? Quote Link to comment Share on other sites More sharing options...
slonick81 Posted July 25, 2018 Share Posted July 25, 2018 6 hours ago, dslnc said: Interesting article: https://www.zdnet.com/article/why-raid-6-stops-working-in-2019/ I meet this idea from time to time, and it's reasonable, but somehow isn't proved by real life. I mean there is RAID5 made of several 2TB drives in this article as example and controller needs to read 12TB of data to rebuild it, so there is a significant chance to get unrecoverable read error during rebuild process and lose some data. But it also apllies to simple data reading. Like every 20TB of data read you should get such error, no matter is it raid6, raid1 or single drive. Then (in case of ZFS, as example) the system will detect it as checksum error, mark array as degraded and try to rebuild it if possible. But 20TB of data read (not stored) is really small amount of data if we talk about video editing. In my case it's like getting footage for film or small serial and transcoding it to proxy format. Without even starting to edit. Or there is a resilver process twice a month, when the whole array is checked for consistency. It should degrade just from this procedure. So I should be swapping drives and rebuilding array all the time. But nothing even close in reality. And it makes me paranoid. I even started to keep some test data set with externally calculated checksums and verify it by myself from time to time - but it's fine so far. Quote Link to comment Share on other sites More sharing options...
no_connection Posted July 25, 2018 Share Posted July 25, 2018 I think the read error rate is exaggerated. My 4x 4TB RED in Z2 gets scrubbed every month and is almost full. After a few years of service I have yet to encounter any read error from disk or corrupt data that ZFS picked up. That is a lot of TB read just for scrubs and still no error. However if you are at the point where one drive already failed there is a very high change of other drives being close so the extra safety is at least for me well spent. BTW ZFS resilvers on the fly whenever there is a checksum error. Scrub is only "needed" for data that is not touched regularly. HDD does the same internally too and detects weak spots in magnetic strength, so it it very likely that scrubs actually helps and prevents data from becoming corrupt. Quote Link to comment Share on other sites More sharing options...
SR Posted August 7, 2018 Author Share Posted August 7, 2018 This sounds like it could be a big deal for RAID, if this new mass produced 4tb ssd is affordable. Plus, it comes with a 3-year warranty. https://news.samsung.com/global/samsung-electronics-starts-mass-production-of-industrys-first-4-bit-consumer-ssd Quote Link to comment Share on other sites More sharing options...
webrunner5 Posted August 7, 2018 Share Posted August 7, 2018 1 hour ago, SR said: This sounds like it could be a big deal for RAID, if this new mass produced 4tb ssd is affordable. Plus, it comes with a 3-year warranty. https://news.samsung.com/global/samsung-electronics-starts-mass-production-of-industrys-first-4-bit-consumer-ssd Yeah, "affordable", that is the magic word LoL. This is a pretty good idea if and when the price comes down. Being new you are going to pay the price as they say. Quote Link to comment Share on other sites More sharing options...
Dan Sherman Posted August 7, 2018 Share Posted August 7, 2018 1 hour ago, SR said: This sounds like it could be a big deal for RAID, if this new mass produced 4tb ssd is affordable. Plus, it comes with a 3-year warranty. https://news.samsung.com/global/samsung-electronics-starts-mass-production-of-industrys-first-4-bit-consumer-ssd Sata based SSDs are on the way out in consumer devices. M.2 SSDs are smaller and multiple times faster because they are not limited by the very slow for 2018 sata protocol. depending on how computer literate you are, I would advise against a consumer grade NAS. If you have the skills its far more economical and flexible in the long run to build your own NAS running FreeNAS. SR 1 Quote Link to comment Share on other sites More sharing options...
Shirozina Posted August 8, 2018 Share Posted August 8, 2018 If you are moving to a desktop PC just use RAID 0 on internal disks and regularly save to a backup as it will be cheaper and less hassle than an external setup. I use 4 x 2tb which gives me enough read and write speed for 4k and an 8tb ext HD for daily backups. If I have a drive fail I would just replace the bad drive, rebuild the raid and copy back the data to it from the backup which is probably a lot easier, faster and safer than trying to recover from the more advanced RAID configurations. RAID 0 currently enables fast write and read speeds and large storage capacity for less money than SSD or NVME storage. When the latter 2 drop in price significantly we can consign RAID to history...... SR 1 Quote Link to comment Share on other sites More sharing options...
Dan Sherman Posted August 8, 2018 Share Posted August 8, 2018 8 hours ago, Shirozina said: RAID 0 currently enables fast write and read speeds and large storage capacity for less money than SSD or NVME storage. When the latter 2 drop in price significantly we can consign RAID to history...... Everything you said up to this point was right on tarkget. The primary use case of RAID is not speed or capacity, its Redundancy. That's what the R in RAID stands for, speed and capacity are nice benefits though. Also Raid is becoming more prevalent, because as we need to store ever increasing amounts of data, the risks of loosing it to errors increases. Hence the reason why new RAID schemes that increases redundancy are always being worked on. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.