As many of you know, I’m a bit of a film buff—and my DVD collection contains thousands of titles. So I’m wondering: is it feasible to build a media server that can hold that many DVDs on a hard drive array (I figure I’d probably need 15-20 tb)? If so, how would I go about it, and what other equipment would I need to operate / organize the server? I also have several hundred CDs I’d like to include.
My AV receiver is a decent Denon (4306) with ethernet connection, so I can hook into my home theater, I believe.
Is this practical? What kind of cost would I be looking at? Is there software that can coordinate all this? Because I have to tell you, I’d sure like to put the discs into storage and just use the media server to have movies / music on demand from a home network, which would free up a lot of physical space.
Ideas? Suggestions? Contractor bids…?
Something like this?
Yeah. But turnkey systems tend to be way more expensive. And I’d need 4 of those to include everything.
How does it work, do you know? Can I build my own high speed RAID array to handle what’s needed?
Check out AVS forum. I believe there are separate forums on home theater PC’s and media servers, with the media server section as a subheading under the Digital Audio and Video Devices forum. There is info on both doing it yourself as well as commercially available solutions. Just from reading about this subject previously, doing it yourself is not easy and it takes a lot of time and effort to set up and maintain a media server. Computers and A/V equipment don’t always merge real well and it may be best to go with a product whose integration has already been tweaked and confirmed
Practical, sure. All you really need is a PC at the TV end to replay the digital video and a PC at the storage end to hold it all. These can be the same PC.
Building the storage server shouldn’t be too hard. 300 to 500 gig drives are out there, and you only need 30 or 40 of them. So you’re going to have a big-ass line of drives and enclosures strung along there and will be spending thousands and thousands of dollars.
You’re real problem isn’t the hardware – it’s stripping all those digital files from the DVDs and moving them into your digital system. You’ll have to personally put each disk in a drive, run the stripping software, and copy the ginormous file to your hard drive array. Then repeat, thousands of times. It will take months.
Ugh.
Well, that sucks. Maybe I’ll just keep putting them all in by hand, or daisy chain a few of those 400 disc jukeboxes. Only problem is, they’re huge.
Jesus. First no jetpack, now this. The futurists have sold us a bill of goods, man.
I have to agree with Robert. My own collection of film (VHS and DVD’s) is modest, probably only about 30 archive boxes or so. But I have close to then thousand sound recordings in all formats—78’s, 45’s, LP’s, CD’s—and while I have fantasized about converting them all to CD (for space) the upshot is that I don’t want to be doing it all the time.
You may get started on this, but, after a couple of months, you will be asking, why am I doing this?
Just go rent a cheap storage unit. That’s what I did. It’s worth it. Time is money.
Jeff, what counts as “feasible” in your mind? Figure, for a start, that disk costs $1 a gigabyte. 20 terabytes is 20,000 gigabytes, so you’re talking $20 grand in disk space alone.
On the other hand, a quick amazon search yields a 400 DVD changer for less than $300. So you can do 4000 DVDs for $3 grand.
I haven’t looked to see if this is controllable by USB or as a SCSI media device, so this is only a back-of-envelope sortg of thing, but that’s the kind of money you’re talking.
it looks like you’re headed in this direction – I have a friends who has one of those with 6 450 GB Raptors from Western Digital – he’s using it to serve high quality avis though. Not sure now cthulhu’s idea handles imaging the dvds, but I think these racks are definitely part of the hardware piece.
You just need to find some desperate illegal no longer able to cut it in the apple orchards. Jose, here’s your workstation in a tiny basement room. Every time you have copied 100 DVDs, you get three shiny dollars!
Yeah, the downside is the time spent ripping and/or imaging all the original disks.
The upside of the time delay is that if you start with, say, four 500 GB drives for a terabyte of RAID 6 (redundant) storage, you can go on and start doing the ripping. At the rate disk drives are developing, by the time you fill those up it’ll be time to graduate to four 1 TB drives, and when you fill those the 2 TB units will be coming on line. In each case you just pull disks one at a time, plug in the new ones, and wait for the system to rebuild before repeating the process. (We now pause while the chortles from amused IT managers die down.)
Do NOT try this with consumer-grade motherboards and power supplies! You are definitely going to have to spend some serious money for the support hardware—an industrial server-grade chassis, with motherboard and power supply. You might also get started figuring out Linux. Windows is not gonna do the job for you.
It’s definitely doable. $5K will get you started in a meaningful way, and if you do it incrementally you’ll only moan at the individual charge-card bills rather than howling in agony. The other advantage of doing it incrementally is that if you decide to give up at any point you haven’t paid out for the full system.
But it is pioneering in a very real way. Sounds glamorous, but remember that pioneers have to deal with hostile tribes.
Regards,
Ric
Might want to contact Will Collier if you haven’t already. If I remember correctly he was trying to do something similar. Good luck. If the link doesn’t work it was a Vodkapundit post on Aug. 12th, 2006.
The other way to go is to ditch the jewel boxes and use sleeves – you can go this route and at least accomplish the space-saving. There are software solutions out there that let you assemble the digitized data for each dvd into a handy computer catalog…
Actually, these are the units I’ve seen that are best for the sleeves
You can find 2 1.6 to 2 TB Network Attached Storage (NAS) devices at newegg for $1500 or so. That’s roughly 400 DVDs. These are easy to set up, pretty much hook up a network connection and go. You will want a 100 Mbit ethernet router and cards.
As someone pointed out by the time you filled one up capacity may have gone up nd prices down on these NAS devices. They are raid arrays inside the box.
So… it looks like Robert has the right idea… the missing piece is the service to do the archiving… surely those overseas shops that handle huge scanning projects have thought along these lines? My experience with ripping dvds suggests that you’d probably have to live with something like a 90% success rate… but maybe just archiving straight images is a much more straightforward process, still, aren’t a lot of dvds engineered to frustrate that?
Ain’t input a bitch?
You should hear Ted Turner go on about it.
Yeah, multi-terrabyte RAID arrays are possible, but you’re talking full 42-U cabinets and associated power and cooling. And you can have a zippy-de-do-da database full of pointers as a front end. But you still gotta feed the monster.
What you really need is a lot of dwarves, I think. Unfortunately they tend to be really pricey these days, the arrogant little bearded bastards.
You might consider an intermediate step. High density DVDs. Recorders will be out soon and you can probably get ten regular movies on a single lDVD. The disc media won’t cost much. Total cost perhaps $2K.
If you put all your stuff on hard disks it will be expensive. And hard drives fail so you need some sort of redundancy or you could lose hundreds of movies at once.
The downside: Putting the stuff on HD DVDs would be slower. You wouldn’t have immediate access to a given movie – unless large changers for HD DVDs become available.
In five years ten terabyte rundant storage will cost about what one terabyte costs now. And maybe less.
K – good point, but at the same time, HD-DVD and cousins will ensure that average disk size keeps expanding as well… seems like the ultimate solution is to have just one copy of these things and let people own access instead of the physical media and get the media served over broadband. In 5 years let’s hope that advances in VOD and connectivity obviate this problem…
I’ll do it for fifty bucks.
There’s a couple of these bad-boys in the room upstairs from me as I type. (Needless to say, I’m at work… :-( ) A bit spendy for the typical home theater user, though…
mojo: Don’t forget the other problem – they smell of cabbage.
Ric (first among several) nailed it: Start with a configuration that allows you to claim “success” without having also to claim “completion”.
Ease into it, and the technological side doesn’t even have to be too daunting, just redundant disks on a normal machine. And be aware of the fact that you’ll likely have several false starts, as you change your mind during implementation of your nefarious plans, given new information gleaned along the way.
1) Have you an actual notion of a feasible budget? Like, expecting a big payday in court from some psycho chick? On the cheap, a 1TB SAN pack runs like $600. $600×20=$12,000 for 20TB. Someone else named $20K for 20TB disks – I say that would run on the cheap, oh, $300 per 750GB SATA, yielding $6000/15TB, $8K/20TB, $400/TB.
Is that vaguely reasonable to you? If not, then as with monkyboy’s Balloon Fence, further investigation is unnecessary. If so, we can proceed, noting that costs will spiral from there, maybe 2-4x or even more, considering chassis, switches, cabling, software, labor, real estate, power, cooling, redundancy, and so forth.
2) Blu-ray drives go $1K each, disks cost ?, will hold up to 100GB, maybe twice that. At least the jukebox becomes imaginable. Start ripping now.
3) This would scale better if shared among some number of paying users. Say $5000 for the juke (or less, will fall), $50 per disc now crashing below $1/disc within a year, for 100-400 discs. This commences at $10-20K, or in a year under $5000 (not including network gear, etc.). Bad – but for 10 users at $45/mo, pays for itself in 1-4 years.
Just right now, you might swap a Blu-ray drive into an existing jukebox and start burning ‘em in your spare time. By the time you get very far the discs will be cheap enough. I don’t know the jukes you’ve seen but they might have replaceable magazines so exceeding the 400-disc capacity would weigh less in the balance, even with your current DVD count. You would just have 1/4 or so online at a time.
PS Array rebuilds…teehee! If you go that way, yes indeed, buy the premium gear.
Oh and if you are serious about doing this, I would be happy to help.
Fuggit. Rip ‘em. You can hook this up to a Mac or a Pee Cee, but for God’s sake hook it to a Mac.
Fibre Channel baby!
yours/
peter.
Will Collier’s initial bleg was here, with a status report here.
It cost him nearly $600 and his soul (in that Microsoft products were involved), and he concludes, Like they used to say on “Jackass,” we must insist that no one attempt to recreate or reenact this stunt at home…
As for me, Mr. Moneybags, I’m still boggling at all the space I’m saving switching away from those bulky VHS tapes.
I reiterate that storage is the way to go, having stored 20’ x 20’ x 10’ of sounds and written word and personal papers in the past few years.
The two main prior impediments are gone—the object as sole source of information (pretty much negated because of the internet), and the object as source of pleasure or education (wait, I want to watch “Touch of Evil” right now) is offset by age and patience. What I mean is, you may THINK you want to re-read “Man in the High Tower” RIGHT NOW, but in fact you can wait until the weekend and move a few dozen archive boxes to get to it. And you will enjoy it all the more when you do.
The alternative is to have your living space completely cramped. And in 20 years, you won’t be able to move without knocking something over.
I have thousands of movies at my disposal. I can’t just decide on the spur of the moment which one I want, find it, put it in the player and sit down and watch it—but then I also don’t have to store them all. Netflix does it for me.
Right now I keep the DVDs in sleeves, which fit into 5 aluminum-hard plastic lock boxes (they hold 510 DVDs each). That route replaced 7 bookshelves—and only cost me 1/4 of the bottom of the hall coat closet. The biggest problem with that—and it ain’t much of a problem, really, unless you’re as lazy as me—is that, because they are stacked, I often have to lift several of the cases to get to what I’m looking for. And while 510 DVDs aren’t that heavy, they’re heavy enough when all you really want to do is eat summer sausage, drink beer, and watch the unrated version of Caligula.
Still, I think I’ll stick with that for now. I’m not a wealthy guy, so I guess I can wait until General Mills starts putting TB hard drives in cereal boxes.
Hell, the first digital watch I ever looked at cost a thousand dollars or so. Now they’re packaged with Happy Meals.
Noise/cooling need to be factored in. Depending upon how noisy the fan motors are, you may want to put up some kind of sound barrier. The sound barrier may complicate cooling requirements.
Do you have a whole house surge suppression system?
Sound barrier…I’m thinking, mile-high dirt berm? Giant helium balloons covered in solar panels for a power source?
Screw that noise, Jeff. Just wait until movies are released on memory instead of media, and you’ll be good to go.
I’ve got a 2GB USB memory in my pocket right now; cost me $40. In a couple of years, you’ll have DVD-capacity USB memory out, factory-writelocked and written, for just a couple of bucks in hardware cost. Of course, it might also require USB 3.0 (not invented yet, as far as I know) to get the data out fast enough, but it’d be fairly immune to the problems DVDs have: getting scratched just in the handling.
I think what you’re going to see with high-density DVDs is they’re just going to move the norms for resolution way up.
And in that event, of course, you could have a bunch of USB hubs daisychained, and copy, for instance, 30 films at a time. You’d just have to plug ‘em in, have a script that copies the contents of all the USB drives to your main drive array, and go.
Problem, though, is not only storage but software. You’re going to want video library software, so you can organize everything. Probably something like iTunes, only for video.
Hard drive prices continue to drop (more accurately, capacity per dollar continues to grow), so keep the dream in mind while watching prices. Right now you can get a 1TB FireWire drive array for $550; that’d let you get what you want for under $10k. Of course, you’d have to daisychain 20 of them together, but look for the capacity to rise more for the price.
The problem is when those movies move to memory, you’ll have to buy them again. And the time to rip them is going to be a chokepoint. I don’t think the time to do that is ever going to get short enough. I’ve backed up a couple of my favorite movies, just in case. Takes about 15 to 20 minutes to copy the image to the hard drive. So if you spend one hour every night you should get 4,000 DVDs done in 3 years or so. The incremental approach would be wise.
My experience with ripping dvds suggests that you’d probably have to live with something like a 90% success rate… but maybe just archiving straight images is a much more straightforward process, still, aren’t a lot of dvds engineered to frustrate that?
The failure rate is probably during the burning to DVD part. You could always do an MD5 hash for both the DVD in the drive and the image on the hard drive, if they match, it’s perfect.
Not if you have hubs. If you’re copying in and out of hubs, you can do movies several at a time, which means you can just issue the copy command for, say, a couple of dozen memory modules and then walk away. It’ll take the same amount of time, but with a lot less involvement. With USB, you can daisychain hubs together, so (this is a current-day forinstance) you could daisychain four 7-port hubs together and have a couple of dozen available USB ports. If you can do them in groups, you’d simply run a copy script and let ‘er rip. You’d probably want to do them in groups of, say, twenty or more at a time so you can have it copying for an entire day at a time.
Or you could simply run multiple multiport USB PCI cards, which would speed things up a bit. If you’ve got enough PCI slots, that is.
Plus, I think the throughput is going to jump again; USB 2.0 is currently 60 MB/s, and I’d expect USB 3.0 to do a bit better.
And yes, you’d have to dish out for the movies again. Of course. Otherwise, you’d going to have to get a DVD jukebox that’s remotely controlled by the server, so you could autoload DVDs. Probably such a thing doesn’t exist, yet, although DVD storage jukeboxes do exist. Just try to get a price on one without having to contact the dealer, though.
If Jeff wants to keep his movies on DVD, he’ll probably better off daisychaining DVD jukeboxes, if you can even do that. I don’t see how the Sony 400 disc unit, for instance, lends itself to that mode of operation.
Right now I have the Sony, but it’s mostly filled with CDs (I keep maybe 25 movies in there, most of them my son’s, so he doesn’t get his sticky fingers all over them). I think you can daisy chain two or three together; still, the units are rather big and bulky and need to be connected out by the home theater system.
Eventually I want to be able to run the entire media server from a wireless keyboard attached to a home network that I keep on a sliding tray under the coffee table.
A boy can dream, right?
I hear Wal-Mart, who I was surprised to learn sells 40% of all the DVDs in the US, is moving to downloads (with the purchase of the DVD, you get a key to download the movie to your hard drive; eventually they will sell simply the downloads). But, as with the failure of the eBook, I think the fact is, people like to have physical copies of the media. I’m no different. That way, if something goes wrong, or your hard drive crashes, you haven’t lost anything but the uploading time. You still own the flick.
Incidentally, I was an early Netflix member. Very early. But for me, the problem was that by the time the movie arrived, I wanted to watch something else.
Plus, as someone trained in lit, I have been conditioned to go back and constantly review/re-examine narratives for subtleties I may have missed early on. This enables me to comment on them intelligently.
And part of the baggage of that training is that you own a whole lot of books, films, etc. Libraries simply don’t work, because the prigs who work there frown upon the marginalia you add while reading. And Netflix wants shit returned before they’ll let you use some of their other stuff.
If I had the time, I’d scan some of my theory books to show you what I’m talking about. In retrospect, I should have done so while I was having fun with “Thersites”; might have saved him the trouble of convincing himself he was dealing with somebody below his disciplinary pay grade.
Heh. I’ve had that happen a time or two. It was less of a problem when we had our account set up to allow us up to three movies in the house at a time.
As it is, if I’m not really sure I’m going to want to see the movie, I don’t order it.
Oh well, sure—if you want to comment intelligently…
My brother does HT sales and installation in the Denver area; I could ask him if he knows of anything like this, but the likelihood of it is that even the HT guys are all playing direct from DVD media.
The PowerFile site doesn’t seem to fit the solution described right now, but I know they used to sell relatively inexpensive jukeboxes that could do this. They required a host computer, of course. I used to use one for archival data storage.
Something like this sounds pretty ideal. “The starting price for the new Kaleidescape System is $9,995″… though kind of high.
But the key to the whole thing, and adds some future proofing, is that it can source to multiple viewing locations and offers a nice interface so you’re not fiddling with a HTPC if you’re not into that. I have no clue how much expansion would be required in this case. That was from a quick Google search.
These are only 110$ for 400GB this weekend.
http://www.newegg.com/Product/Product.asp?Item=N82E16822152046
If you rip your DVD’s to XVID or DIVX you can easily put one movie per 1GB.
Thats 400 movies for 110$.
Buy the biggest, quietest tower you can find (Antec maybe) for under 200$.
Ah, heck, the easiest way is to just download a movie via torrent file whenever you decide what you want to watch & just keep the file on you system. Or, delete it & redownload it whenever you wish. No sense in keeping something on file taking up space when you can download a movie in compressed avi format (usually less than 800MB) that you can decompress & watch at your leisure. Depending on the number of seeders, it can take anywhere from 45 minutes to a day to download.
Illegal as hell, but a lot easier than ripping thousands of CDs, of which a couple hundred you’ll ever watch again. And, yeah, since it’s illegal I’m really not endorsing it (ahem).
Jeff,
These things are practically free nowadays.
Or save your money and buy the disk.
You can always find a free tool to do what you need on older hardware. It’s not easy, but that Elive distro can make your old PC look and run like something out of the movies. I have it running on an old Toshiba from last cenury and it works great.
MythTV is a linux PVR distro that looks pretty cool.
Of course, you’ll have to get fat, dress like that asshole “I’m a Mac” guy from the Apple commercials and grow a goatee to use it, but hey, whatever works. I managed to get by with a Real Doll and a soul patch.
20TB is only 40 500GB drives, if you only want a giant virtual disk with no redundancy.
But that’d be stupid, with all the effort put in to encoding the DVDs, so call it 60 disks, plus another 5 or so for spares to swap in.
About $175 a pop for the drives, makes juts over 10 grand for the drives. Then you gotta talk enclosures, cabling, power, space…
(However, if we use the 1 gig per movie XviD/DivX model, and your thousands of movies is only, say, 3,000… that makes 3 TB of space, which is only 9 500 gig drives to get RAID5, and you could probably fit those in one tower, for a total cost of around 3-4 grand, including the computer guts and a nice RAID card.
For more ease, and a lot more money, there’s always an XServe and some XServe Raid units…)
And, of course, the MacMini has lead to all kinds of interesting developments in HTPC stuff for the Mac. There’s a new beta product that handles the DVD ripping, cataloging, and viewing. Looks like it’s a legal/licensed product too. Hopefully the integration with FrontRow will be improved.
The DVD images in this case, or most cases, will still need to be stored in some massive drive array. RAID5, plenty of power, lots of cooling, etc. And by “cooling” people aren’t just talking about loud fans. You’ll need to deal with increased air conditioning costs and maybe some minor construction to allow some hot air to vent to the attic directly above the unit.
Of possible use:
18. Handbrake
http://handbrake.m0k.org/
Handbrake enables you to stick a DVD in your DVD drive and have the contents of that film stored to your hard drive in a form that can be read by pretty much any media player. I often use it to put a few movies on my laptop for travel purposes, so I don’t have to worry about keeping track of DVDs while on the road.