160TB Server with Linus! (From Linus Tech Tips) – Smarter Every Day 222

160TB Server with Linus! (From Linus Tech Tips) – Smarter Every Day 222

– Hey, it’s me Destin. Welcome back to Smarter Every Day. There’s a lot of stuff that goes into making this YouTube channel and some of it’s more
complicated than others and there’s a lot of
behind the scenes stuff that you don’t see. This is one of those types of things. I wanted to learn some stuff
and I called in an expert and he happens to be a
personality here on YouTube. His name is Linus Sebastian
from Linus Tech Tips. So the thing you might not
realize about Smarter Every Day is I have a problem and it’s because of this thing right here. So as you know this is the
V2511 high speed camera. This is the workhorse of
how I do slow motion video on the channel. But the problem is you
got two types of memory on this camera. You have volatile memory on the camera. Think of it like RAM on your computer so if you shoot something
you got a certain amount of RAM that can save on this local camera and if you ever lose power, you’re done. There’s non-volatile memory on top and this persists if
you lose power, right. So this is 512 gigabytes of memory but the problem is after every shoot I have to do something with this, so I have a problem. Let me show you what I did
to try to fix this problem. So about, I don’t know, a year ago, no, six months ago I sent
this particular, come on. Come on, come on over here. About six months ago I sent this tweet out to a guy named Linus Sebastian. He does this thing where he puts, I don’t know, servers in people’s houses
and I called him here. So Linus, I need your help, buddy. This is pretty serious. – I’m here. All right, yeah this is really serious. This is bad. – First let me show you my hard drives. Can I show you this? – Yeah, okay.
– Okay, let me get the hard drives and show it to you. – I don’t know if I can handle this. (Sebastian laughing) – I’m sure you’ve seen people
with more than this, right? – No, I think this is even
more than Justine had. – Okay, so this is not
all of it, there’s more. Do you want me to just stick with this? – No, I wanna see, bring on the carnage. – Okay, let me go get the old stuff. – So let’s start doing some
inventory here, I guess. 434 gig, so we’re at two
and a half terabytes. That’s a one terabyte. Three and a half. – So this is the old stuff. I call this the deep vault. – Five and a half. This is an eight terabyte drive. – Yeah, it is.
– And it’s literally ducked taped
to two other drives. – That’s how I roll, man. It’s pretty serious. And so, I literally go to the spreadsheet, look at this. So I have an entire spreadsheet. – Go have a look at this. – So what I do is I just come in here, I Control + F and I’ll say, like, Mars rover, and so if I go to hard drive A27, so go find A27. (Sebastian laughing) This is how it works, man. – [Sebastian] Oh, God. – [Destin] So A26. 21, okay so somebody’s moved A27. (Sebastian laughing) – [Sebastian] So we’re finding a flaw in the system already here. – We’re hosed, we’re looking for A27. So A27 is not where it’s supposed to be. So you see my problem, right. So these aren’t even indexed yet.
– We all see your problem. – [Destin] These are– – [Sebastian] (screaming)
You just dropped it. – [Destin] Yeah, I did. But it was on carpet so we’re good. So it was my understanding you shipped a 45 Drive server here. – Yeah.
– And we’re gonna install it and it’s gonna
make this problem go away. – It might not be enough. – Seriously? – Have you added up the capacity of this? – I have not, no. – ‘Cause I’m at like 25
terabytes here already. – Yeah.
– I shipped you a 160 terabytes server. So here’s the thing, you could make the argument
that our data hoarding really doesn’t make a ton of sense because we could just
grab the necessary footage from a YouTube upload, splice it into our new
video when we wanna refer back to it and bippity boppity, off to the races. Now, with Destin, his case was a really
interesting one for me because he actually
does not have the luxury of being able to go back
to the previous video and grab it without making any
very significant compromises, because this kind of high speed footage the data rates are so
incredible and the amount of runtime for what could be a very, very short clip wouldn’t make sense to upload in the YouTube video at all. So he’s gotta pick and choose his battles. So he’s throwing out a ton of frames whenever he’s showing you, whether it’s a hummingbird’s
wings in realtime, he’s throwing out a ton of
data in the actual video that you guys watch, except for the small segments
that he decides to show you. So everything else here could be necessary if you ever wanted to go
back and look at something in more detail. So we gotta get this server rolled out. – You’re genuinely worried, aren’t you? – Yeah. – My tweet said you would
fix my problem, dude. – Wait, your tweet– – Help me Linus that you’re my only hope.
– Your tweet says I’ll fix the problem so now it’s my liability if it doesn’t, is that right? So you put words in my mouth. – You never actually replied to this. – No, I didn’t. – You don’t even care. You don’t even watch Smarter Every Day. Let’s just fix my problem, I don’t care. – [Sebastian] All right,
so let’s come over here. So Destin was– – [Destin] Yeah, we installed it. – [Sebastian] Yeah. – [Destin] Any more light always is good. The nice yellow light. – Oh God, he just tripped over the drives.
– Sorry. It was only like 20 terabytes or whatever. – So this is great. We actually had this
server shipped down here. It must’ve been like– – [Destin] It’s a while back. – Two months ago or something like that. It took me a long time to get down here. I’m sorry, because clearly the problem is, I would’ve gotten on the next plane– – [Sebastian] It’s a big deal, dude.
– If I’d known it was this bad.
– Yeah. – Now, have you installed
the drives in here yet? – [Destin] No, I haven’t. – And that’s why this drill is here. – [Destin] Yeah, we’re
gonna take this out. – Undo them out, got it. (machine drilling) – [Destin] Cool. – Look at that. (cover banging) (Destin laughing) – [Destin] Yeah. – I love this video so far. – [Destin] Are you being sarcastic? – The shenanigans are real. No, I love it. This is great. So Seagate actually
shipped you 12 terabyte instead of 10 terabyte drives, which is what I thought they had sent you. – Okay. – So that’s another 30
terabytes of raw capacity. – You just rip them open like that? – This is good. Yeah, yeah. You just kind of shuck them. – Like corn? – Yeah. Yeah, yeah, basically. – I think you think I’m joking but I actually don’t know all this stuff. – So this is just a friction
mount here on the side. – It’s a spring? – [Sebastian] But give
it a little more pressure than you’re comfortable with. There you go. – That’s it?
– That’s all there is to it. – So this is easy. – This is easy. So the way that RAID works
is it’s writing the true data and it’s also writing what
are known as parity bits. So if I had, let’s focus on these
four drives right here. I’m running this in, let’s say, a RAID 5. – Okay. – That means that for every
three pieces of real data, and this is a
oversimplification for the sake of it being easier to understand, I’m writing a parity bit that is, think of it as kind of like
an algebraic expression where this parity bit is on the other side of the equal sign of these three. – Got it. – So if I were to lose any
one of these four things I can rebuild the other one. – By resolving the equation. – It’s a way of rebuilding
what you had lost, right. – [Destin] Right. – But we are not running parity in a conventional sense here. So we’re using a software called Unraid that rather than striping the parity bits so that they exist across multiple drives it actually uses an entire single drive, or two drives if you prefer to be able to lose two drives to write
all of the parity bits to. So that comes with some
advantages and disadvantages. The disadvantage is that it’s meant mostly for archival storage. It’s not as fast to write to. – Okay. – But the advantage is that in the event that you set it up with
a single parity drive and you lose three drives, let’s say your house floods or something and it floods up to here, all this data is still good. – Okay. – Whereas if you had
the parity bits striped, if you lose more than your threshold, all the data is gone. – I see. So let’s say just for
sake of easy numbers, let’s say they were 100 terabytes here, which there’s more than that, but let’s say there were
a 100 terabytes worth of hard drive space, that’s not how much server
space I’m gonna have? – Nope. – How much will I have? – You would have anywhere from 80 to 90, depending on what you prefer. – And so that other part is
just for the parity bits? – That’s right. – This is a big deal for me man, I mean. – No, it’s great. You need it more than anyone
else we’ve deployed one of these to so far, I think. – Are you serious? – Yeah. – It feels like it. – This is a dire situation. – [Destin] This is another video for you but this is like a really significant, I’m not gonna say emotional, but there’s a lot of
stress tied up in this. A lot of my life has been protecting this and I’ve got some in like
a safety deposit box. How long’s it gonna take
to slurp all the data? – [Sebastian] A long time. – [Destin] Really? – You’ve got a big, big data hoarder problem here. (Sebastian laughing) – [Destin] This isn’t data hoarding. (Sebastian laughing) – There it is. (Sebastian shouting with excitement) – [Destin] Did it work? – Oh, yeah. We can do this later but I just wanted to make sure all the
drives are showing up. Three, four, five, six, seven, eight, nine, 10, 11, 12, 13, pal, perfect. – [Destin] Is it good? – And the SSDs are here too. – [Destin] Is it good enough to high five? Okay, just making sure. They’re all formatting, wow. You can hear them all clicking. – [Sebastian] Yeah, they’re going. – This is nothing. You do this all the time. You don’t care about this at all. – [Sebastian] Oh no, I think it’s cool. I think it’s cool. I’m sharing your cool experience. (Destin laughing) – Okay, so once we set up our RAID here, even in the event that
we actually did lose more than two drives, because it’s on RAID
the only data you lose is the data on the drive
that physically failed. So that’s cool. And in addition to that, because we’re using Ironwolf Pro drives they come with a five year warranty and they come with a
data recovery warranty. – Thank you. I appreciate it. – Oh, you’re very welcome.
– This is a big deal. This is Linus from Linus Tech Tips. They do great videos like
they hooked me up here. This is a big deal. Thank you for that. – My pleasure.
– And your partners were 45 Drives and Seagate. – And Unraid. – And Unraid. So thank you to you three
companies that did that, really appreciate it. Go subscribe to Linus. Linus does a bunch of, like everything, right? Everything from reviewing keyboards to like everything.
– Pretty much. – You do everything.
– To touring the world’s only commercial quantum computer manufacturing facility. – Really? – Yeah, we did that. – Go check his stuff out, it’s amazing. I like your channel. – Thank you. – Yeah, I’ve watched your
channel for a long time. Big thanks to Linus for
coming out to Alabama. This was a huge problem I had, this data storage issue. Thanks to Linus for working
with these companies, his contacts, not mine. You had 45 Drives who made
the enclosure and the server, Seagate who donated the hard drives and Unraid is running the
software on that server. I’m very thankful and I’m also interested in seeing how he does in the next video. I’m gonna do something
kind of interesting, I’m gonna pluck Linus
out of his RGB Keyboard gamer system, sandals
over socks kind of world and I’m gonna introduce him to this man. – I’m Luke Talley and at this time in 1969 I was a senior social
engineer at IBM in Huntsville. – Luke Talley is an amazing individual and I want to just
juxtapose Linus who’s up on all the new tech with
Luke who had to figure out how to make memory for computers. I want to put those two together
and just see what happens? And it is fascinating. I hope this video that you just watched earned your subscription
here to Smarter Every Day, but more importantly, I hope you will stick around and subscribe and ring the bell so that
you can see that next video. Because Linus and Luke both teaching me really interesting things about one of the most important computers
ever built by human hands is a fascinating trip. We went to the US Space and Rocket Center. All that’s in the next video
so please consider subscribing if you’re into that sort of thing. If not, that’s no big deal. Anyway, I hope you enjoy this. I’m Destin, you’re
getting smarter every day. Have a good one. Thanks Linus, bye. Let me show you a trick. This is a trick I learned about computers. – Yeah. – So I have a pen and paper and sometimes I write things down.
– There we go. No, I remember the IP. (Sebastian laughing) This is a better way.

100 thoughts on “160TB Server with Linus! (From Linus Tech Tips) – Smarter Every Day 222

  1. If you'd like to weigh in on this video publicly I will now attempt to trick you into following me on Twitter. Here's the original tweet:

  2. Its been years that ive been watching your videos. Let me tell you. Its much much much better than any prank videos and yolo kids doing crazy videos out there.

  3. Need to get a 10 Gigabit network going Especially with all the file ingestion that needs to take place, a good excuse to get some fiber installed for your home network 🙂

  4. Youtube have bigger hub houses where they have all the memories but what linus do and what it do for youtube why the harddisks are so less only terabites.

  5. On another similar one, I said exactly what you suggest.

    I'm a mix of the two worlds as I know a lot of the IBM guys who were on the project as I did their it system (now not way back).

    I'll assume you used the space up by now and I'm going to have to shoot a video to the "next problem". …

  6. Hope you have fast network at home. I can't imagine to transfer data from server on slow wifi and so on 😉 Also, which file system you have installed for this storage and type of data?

  7. Very nice! And what about backup? If the drives with the initial data are not spinning regularly they might not last long…
    I guess there will be a part II for the backup server…

  8. Every time Linus installs a server for another youtuber he puts a little icon on the side of his car just like how fighter pilots put pictures of planes they have shot down.

  9. You should ask YT directly how to do it.
    They probably store 160TB of new data every five minutes, fully backed up, in multiple data centers and available forever.

  10. There is nothing better than authenticity, especially in the YouTube world. I truly appreciate the authenticity you bring, Destin.

  11. why keep all that crazy sized raw footage? I'd just compress and put on cloud. Do you really go back to all that raw footage? … someone has a HOARDER gene 😀

  12. Don't listen to Linus on building servers. He's the complete opposite of what you do if you need to build something reliable.

  13. Unraid is so slow though, since it's just JBOD with parity. FreeNAS with ZFS would be a far better solution for the filesizes Destin is dealing with.

  14. Didn't expect a collab with the creator of the Linux kernel and founder of Linus Repair Group, LLC based in Manhattan NYC

  15. oh my goodness i didnt know he was just down the street from me. man i wish i would have had the chance to meet him while he was here in huntsville

  16. I watch Linus everyday,(WTH is he doing here?)this guy is everywhere! hey Linus! anything new about gaming in Linux?

  17. literally everyone know who Linus is . you dont need to put extra description in the title. LOL probably just putting its name on there got you extra 30-40% views on this video. just saying

Leave a Reply

Your email address will not be published. Required fields are marked *