Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

Companies

[00:00] Ram is the only platform built to make your finance team leaner, faster, and better, saving
[00:06] businesses 5% in and on average so you can save focus on growth.
[00:10] Ram just opened for business in the United Kingdom.
[00:12] So if you're running a business in the UK, you can now use Ram's AI powered finance platform
[00:17] to manage cards, expenses, bills, and pervals and accounting all in one place.
[00:22] I run my business on Ram and so should you.
[00:25] Learn more at ramp.com slash invests.
[00:28] Logo is the AI platform purpose-built for financial institutions serving tens of thousands of
[00:33] bankers and investors at hundreds of leading firms worldwide.
[00:37] Rogos AI agents autonomously execute large chunk of your firm's workflows, they can
[00:41] screen deals, drafts, sims, run-bire outreach, and due diligence on data rooms, all
[00:46] with full security and compliance handle.
[00:49] Every action feeds Rogo intelligence, the firm's context layer, turning years of deal
[00:53] experience into a secure, governed, institutional memory that makes each new transaction stronger
[00:58] than the last. To learn more, visit rogo.com slash invest.
[01:03] Opening eye, cursor, entry and versel all have something in common. They all use work OS. To
[01:09] achieve enterprise adoption at scale, you have to deliver on core capabilities like SSO,
[01:15] skim, R back and audit logs. Instead of spending months building these mission critical capabilities
[01:20] yourself, you can just use WorkOS APIs to gain all of them on day zero.
[01:25] Visit WorkOS.com to get started.
[01:30] Hello and welcome everyone, I'm Patrick O'Chonicey and this is Invest like the Best.
[01:33] This show is an open and indivects exploration of markets, ideas, stories, and strategies that
[01:38] will help you better invest both your time and your money. If you enjoy these conversations and want
[01:42] to go deeper, check out Colossus, our quarterly publication with in-depth profiles of the people
[01:47] shaping business and investing. You can find Colossus along with all of our podcasts at Colossus.com.
[01:54] Patric Oshanasi is the CEO of PASTIVE SOME. All opinions expressed by Patrick and podcast guests
[02:00] are solely their own opinions and do not reflect the opinion of PASTIVE SOME. This podcast is for
[02:05] informational purposes only and should not be relied upon as a basis for investment decisions.
[02:11] Clients of PASTIVE SOME may maintain positions in the securities discussed in this podcast.
[02:15] To learn more, visit PSUM.VC.
[02:22] My guests today are Gavin Uberti and Rob Lockett, the founders of Edge.
[02:26] A few years ago, when they set out to build a better AI chip than the largest companies in the world,
[02:30] almost everyone I called told me it could not be done. They've since done it,
[02:34] taping out a working chip on their first attempt and becoming the first hardware company founded after
[02:39] chat GPT to do so. There you have more than a billion dollars of customer demand for their first
[02:43] product and effort is $800 million to build it.
[02:46] Edge to build chips and systems designed to run AI models faster and lower cost.
[02:51] They started the company in 2023 and the product is a complete rack for inference.
[02:55] The chip, along with the boards, the power delivery, the interconnects, and all the manufacturing
[02:59] to produce it all. We talk about the technical bets behind their architecture,
[03:03] how they hired industry legends and paired them with elite 22 year olds,
[03:07] and why they believe inference will become one of the largest markets in the world.
[03:11] I think you will find a story of what they have built hard to forget.
[03:14] Please enjoy my conversation with Gavin and Rob.
[03:18] Our gentleman, it's been three years or so, Gavin, since you and I lasted this,
[03:23] which is nuts. And at the time I was just wildly intrigued by your story and what you're
[03:27] going to build, I didn't know a lot about chips at the time. I was considering investing in the
[03:30] company and so I was calling everyone I couldn't see about that could give me an opinion or something.
[03:35] And at the time, basically the consensus was these kinds of companies are not
[03:41] built by young people, that the semi-sworld, the best companies are founded by 40, 50-year-old,
[03:47] people that have had a whole careers where they've experienced, have learned all the problems,
[03:51] have shipped multiple chips. Two, 21-year-olds are like not going to do this. It's just not going
[03:55] to work. It was indicative of a theme which was, nobody believes in us. That's obviously
[04:00] changed a lot now. You know, just walk the halls and talk to the people that have chosen to come work here.
[04:03] But in the early days, it felt like this was something that you had to face down. What was that
[04:09] like facing that down were a set of incumbents and industry worth of people and investors and
[04:14] everyone else sort of didn't believe in you. And what did that anneal in you to build the company,
[04:19] the way that you are? Like what was the impact of that? I think there's a sort of level of
[04:23] naivety required to think that you could build a chip better than every other AI chip ever built
[04:29] and build a company to do it way faster than ever has been done and we have the naivety.
[04:34] There's many times where we would say like, why isn't this possible?
[04:38] And you know, really push on it.
[04:40] And it turns out that like, everybody's answers are extremely siloed to a set of constraints
[04:44] that aren't true anymore.
[04:45] And the reality is the entire semi-conductors and data center industry is built on buffer.
[04:51] And what I mean by that is every part of the staff from the EDA tools, to the power modules,
[04:56] to the circuit boards, to the chip design, and standard cells, everything is built to be
[05:01] general purpose for everything, not just in the data center, but IOT on the edge and so forth.
[05:06] And when you have a specific use case, you're really trying to design for.
[05:10] You can change the constraints a lot.
[05:12] And I'll give you just a very simple example that we're not the only one who does, which
[05:15] is one of the things you care a lot about is the clock speed of your chip.
[05:18] It's proportional to the throughput of your system.
[05:21] When you are doing fine now for different timing, what clock speed you're actually going to be
[05:24] able to run on when you tape out your chip?
[05:26] There's this concept called corners, which is what temperatures are you going to be able to
[05:30] run at this clock speed.
[05:31] The default configurations for a lot of these EVA tools
[05:34] assume that you're going to be running your chips
[05:36] in freezing temperatures.
[05:38] Now, I don't know about you, but I've never seen an AI data
[05:40] center with ice in it.
[05:41] So we can feel pretty confident that our chips
[05:43] don't need to run at full speed at zero degree Celsius.
[05:45] In fact, they're never really going to be running below 80 degree
[05:48] Celsius anyway.
[05:49] And just by knowing that that's a constraint that doesn't matter,
[05:52] we can make a ton of changes throughout the entire system.
[05:54] That's a very simple one.
[05:55] But there is many more that you get 20% here, 50% there,
[05:59] to X here and these compound to assist them
[06:01] that can be radically better for inference.
[06:03] But thank you, Foundry.
[06:04] Two kinds of people.
[06:05] There are some folks who went purely on heuristics.
[06:07] A thing, the young founders, a claim they can go beat
[06:10] the days come in a world of performance, it cannot happen.
[06:13] And there is no thing you could go say to me,
[06:15] that would make me change my mind.
[06:17] But there's also people out there who,
[06:19] of course, skeptical, for no one to go ahead
[06:21] and say, I'll spend the time, I'll do the work.
[06:24] And it is actually possible.
[06:26] Like for example, one of our earliest earliest supporters with Mark Ross, and Mark was a very prestigious
[06:33] semiconductor expert.
[06:34] He used to be CTO at Cypress Semi, that's all for $9 billion.
[06:38] And when we met him, we were just a couple of guys in a dorm room, and we came to him
[06:42] to say, hey, you want to go build hardware for inference, we can be much faster than an
[06:47] video.
[06:48] And Mark's like, no, you can't, it will not work.
[06:50] But, if you want to go convince me, if you're right, a white paper, you go ahead and build
[06:55] a functional simulation and that's only me.
[06:57] And so after a lot of very flowing nights, we're back to Mark and said, hey, here's a simulation.
[07:02] What do you think?
[07:03] And he was like, huh, this works.
[07:05] But you go to a company like this, you'll need a large amount of capital.
[07:08] And the only three million dollars even they get up, you started.
[07:11] And that way hadn't raised five or even a lot more after that.
[07:15] And then he was again surprised, but got more involved.
[07:19] And then he became an advisor.
[07:21] Then a half-time advisor eventually, or full-time CTO,
[07:24] they saw more and more of the development of progress.
[07:26] And I think in general, this gap is in as a filter to really heavily,
[07:30] folks who want to go ahead and be right regardless,
[07:33] or should we want to be very true seeking and say,
[07:36] sure, I'm skeptical, but I will go ahead and work with the numbers myself.
[07:39] And if I can go figure out why this is possible,
[07:42] well, let's go build it.
[07:44] The specifics that you've made bets on,
[07:47] the way that you built this system are immensely interesting to me,
[07:51] And because so many people are trying to do this now, build new chips that will do a better job of serving
[07:56] inference at massive scale, the world is interested in the research approaches, the different
[08:02] architecture approaches that people are taking to building a new AI chip. And I'd love me to just
[08:08] start by describing what this thing is, what it does. But maybe more interestingly and more importantly,
[08:13] the process that you went through to decide what bets to take, what technologies to invent,
[08:19] and comparing contrast those with what you've seen the rest of the marketplace tried to do.
[08:25] Yeah, I think that you guys start with the product.
[08:28] We're not just building a chip.
[08:29] We're building a full infrared solution and that means a rack.
[08:33] That means the chip, that means the power delivery into the chip.
[08:36] That means the board, which is since that means the internet, I was to chip talk to each other.
[08:40] That means the production, for this mass volume, a racks.
[08:43] Really the production is the product.
[08:45] We think about how we get a revenge.
[08:47] There are two key parts of running inference.
[08:49] There's pre-fill and there's deco.
[08:51] Now, we have two key tech batches, both of these things.
[08:54] Pre-fill is reading in a huge volume of text.
[08:57] And deco is then using that data and generate output tokens.
[09:00] When you go out and run pre-fill, your K-jad is not to go predict tokens.
[09:03] You already know the text.
[09:05] Your job is to go and get the models memory.
[09:07] We call this KV cache into the right state.
[09:10] Then you can go ahead and run decode with that same KV cache.
[09:14] So we will often go to do is we call PD-design aggregation.
[09:17] Pre-sold decode deseg.
[09:18] You will have a more cluster of servers,
[09:20] running these pre-fills.
[09:22] You'll then transfer those model memories
[09:24] to those KD cache over to the decode cluster
[09:26] and then go ahead and use that cluster to go to the next tokens.
[09:30] To sort of like loading the gun and then firing it,
[09:32] like I think about it in super simple terms.
[09:33] Yeah, you got it.
[09:34] It's into the models, remember the right things,
[09:37] and then using those things to go to do tasks.
[09:39] Can you really think about this market a bit lazily?
[09:41] This is a pre-filtre chip or a deco chip.
[09:43] If you're a deco chip or an HPM chip or a UNSFM chip,
[09:45] how are you 3D DRM chip?
[09:47] Are you using optics, using copper?
[09:48] When we started this, we just wanted to understand
[09:51] why extremely smart people were working on these different directions.
[09:54] We seriously looked at architectures,
[09:55] like having a bunch of DDR memory and a shared memory pool
[09:59] and looking at it.
[10:00] is packaging to basically break out of the shoreline.
[10:02] We looked at things like,
[10:03] there's a way to put memory dies on top of compute dies.
[10:06] In doing so, we realized that there's no free lunch.
[10:09] Everything has a trade-off, right?
[10:10] 3DD around you have a thermal issue,
[10:12] you have a supply chain issue, you have to figure out
[10:13] hybrid branding, you have to figure out the flop,
[10:15] scenario, decode chip.
[10:16] So we went through everything, both on the pre-fold and the decode sign.
[10:19] In doing so, we realized there's a few design spaces
[10:22] that nobody had seriously tried to explore
[10:24] because there were never done in AI chips.
[10:26] And we asked ourselves,
[10:28] What are the actual metrics that are going to matter the most?
[10:30] On the pre-filtre side, the thing that matters is Flops and Flops density.
[10:34] People talk about Flops often as a headline number,
[10:37] but in reality, you should care about the Flops you're getting when you're running real workloads.
[10:41] There's this concept called MFU or Model Flops utilization,
[10:45] which is, for every peak-flop advertised,
[10:47] how many cents on the dollar are you actually getting?
[10:49] On GPUs, you often get somewhere between 20 and 50% depending on the workload.
[10:55] And actually, you can probably not run at 100% because you have a thermal issue.
[11:00] Whereas as you increase the flop utilization, you have more transistors going on and off,
[11:04] you draw more power, and the chip will self-regulate and actually lower its clock speed to make sure it doesn't overheat.
[11:10] So, as we looked at inference, we said, if we want way more flops,
[11:14] because we want to run at way higher throughputs, we fundamentally need to solve the thermal problem
[11:18] before we even think about adding flops to the chip.
[11:20] If I just add more flops to GPU today or another AI chip, I'm not actually going to get more performance,
[11:24] because it's just going to thermal throttle.
[11:26] So fundamentally, the essence of that is this concept
[11:29] of the Nard scaling, which is voltage
[11:31] is quadratically proportional to power.
[11:33] So if I two X my voltage, my power goes up by four X.
[11:36] If I come my voltage in half, my power goes down by quarter.
[11:39] So we asked ourselves, how could we run voltages lower
[11:42] than GPS?
[11:43] And we talked to a lot of people about this.
[11:44] Because we flew out to Silicon Valley after dropping out.
[11:47] And we asked, do you know, dozens of people
[11:49] and semiconductors at all these different chip companies?
[11:51] How they did it?
[11:52] And the answer we got was, like, you can't.
[11:54] You can't run a voltages lower than GPS.
[11:55] And this was very dissatisfying because there was many different
[11:58] industries of chips that run at voltages lower than GPS.
[12:01] Bitcoin miners run at under a quarter of the voltage of GPS.
[12:04] So this is obviously physically possible.
[12:06] The question is, are there issues with GPU architectures that make it
[12:09] unable to run at these voltages?
[12:11] And when we looked at the problem for a long time, we were able to create a new
[12:14] mechanism of running at much lower voltages, a new type of power delivery that we
[12:18] call low voltage inference.
[12:20] And we think all AI chips in the future are going to be low voltage chips.
[12:24] They're going to have to cram way more flops in the same silicon area and without thermal
[12:28] throttling run at way lower voltages.
[12:30] That's pretty simple.
[12:31] For Dico, it is all a memory game.
[12:33] More memory bandwidth.
[12:34] You can load the model faster, load the KD cache faster, and serve more tokens per second
[12:39] per user.
[12:40] We think people ask the wrong question here.
[12:42] People often ask, how much memory bandwidth is on your chip?
[12:45] You can be asking how much memory bandwidth is on your full scale up cluster.
[12:49] What we are able to do is add way more bandwidth.
[12:54] a much lower latency from chip the chip.
[12:57] Two are in Rp.X.
[12:58] The allows us to be able to go serve.
[13:00] The allows us at this much higher speed because you can go use the SRM and the HBM from the full scale up cluster as a single pool.
[13:06] And SRS, a technical bet.
[13:09] We'll be called cluster scale memory.
[13:11] And on GPUs today, the cluster memory bay was often very badly utilized.
[13:16] Because the time to go hop from one GPU to another is extremely long.
[13:20] For example, on black wolf sets, it can be about 4,000 nanoseconds to a point to point.
[13:25] And that means that if you go ahead and go to an ADXTP setup, you will get way away less
[13:30] than an ADX improvement in your Token's per second per user.
[13:34] And what we did is build our own totally custom interconnect stack.
[13:37] Let me do a talk.
[13:39] Everything about the second layer is a met, built a full custom.
[13:42] Now we can go out and do far for better latercies and advance this way too.
[13:46] We can go ahead and cut this by more than a factor of 5x.
[13:49] And that allows us to then use the memory of other chips much more effectively.
[13:53] As you scale the world size, you're a time per token, you're done proportionally.
[13:58] Yeah, and it's not that surprising given all these architectures for built before China TTT.
[14:02] So if we're trying to build a chip for the modern workloads, it's going to look very different.
[14:06] The way we organize our flops, the way we do our voltage domain, the way we do our power planes,
[14:11] super different. The way we do the packaging is going to look super different, the way we do
[14:14] the board design is going to look different. And then the deco inside, the way we connect everything
[14:18] is going to look very different. So we're now bringing forward our first generation of this low
[14:24] voltage in-front technology, which is running at under half the voltage of any other AI chip.
[14:28] If you zoom all the way out, why is this so important? Like why is the delivery of much higher
[14:34] throughput, much lower cost per token, better tokens per watt? Like all of these metrics that the
[14:39] University of the Star talking about more and more.
[14:41] Everyone knows the supply side of the equation is a big problem right now.
[14:45] Why is this in the bigger picture looking at a decade?
[14:47] The bottleneck in the technology world?
[14:50] Well, I think it comes down to productivity.
[14:52] Where we are at this extremely interesting moment in history civilization,
[14:57] where there is real artificial intelligence.
[14:59] Not like sci-fi stuff, but like these models can solve problems that most humans can't.
[15:04] And it's going to create new scientific discoveries.
[15:07] It's going to create instant access,
[15:08] the medical care, instant access to education.
[15:10] And now it's just about how many people can use it
[15:12] at the same time, how many products can serve this
[15:14] at the same time, and also the speed of doing different tasks.
[15:17] So when you think about wall clock time,
[15:20] if we can take an agent that can run at a certain
[15:23] multiple quality and can take a year to solve a certain task
[15:26] using inference time compute, if you have way faster decodes
[15:28] speed, you can compress that into a month.
[15:31] So the amount of scientific innovation and the amount
[15:33] of actual proliferation of technology will happen much faster.
[15:36] And then the second part is concurrency, where today it's just not possible for a billion people to use these models concurrently.
[15:42] Ultimately, some people are going to get downgraded, some people's models going to be slower, some people just won't be able to access the hardware.
[15:47] A few years from now there's going to be giant models serving billions of users.
[15:51] We're very much in early innings of AI today where the paid plans, there's only a few million users in the world using paid plans of AI models.
[15:57] So we're at one one thousandth of the global population actually using this stuff.
[16:01] So if you want to serve a giant scale, a lot of things change, and one of them is the number of
[16:06] chips that communicate together, where people usually think about this in the context of training.
[16:10] You have these giant training clusters, you have classes with over 100,000 GPUs that are all
[16:14] networked together. And the inference side today, people usually think about it as an H chip cluster,
[16:18] or maybe just, you know, NVL72, the scale of domain. But very quickly, this is going to become
[16:24] thousands of chips and tens of thousands of chips. And the way to get the most performance there,
[16:28] where the time between sending data from one ship to another,
[16:32] that primitive matters way more than is getting cut right now.
[16:35] So when we think about optimizing memory bandwidth for the system,
[16:39] you have to think about how fast these ship can communicate together
[16:42] because they can only communicate really quickly with themselves
[16:44] and very slowly with other ships.
[16:46] They're not going to actually be able to serve giant models
[16:48] at 10,000, 20,000 tokens per second.
[16:50] So we need multiple orders of magnitude
[16:53] of infrastructure built out throughout the entire stack.
[16:55] From the way for the watt,
[16:56] transistor to the token to actually bring this stuff to the world.
[16:59] And I think that you look at most other goods, like the iPhone 4 example, they've gotten
[17:04] this economy as a scale for as a result.
[17:07] More money does not really buy it better iPhone.
[17:10] That if you're a billionaire or if you're just the average American, you buy the same phone.
[17:14] And tokens aren't like that yet.
[17:16] We're still in the very early days where the general purpose system relatively small one
[17:21] is kind of hand-crafting these tokens, like the image screws, back and later the Renaissance.
[17:26] And I wanted to live in the world where you have the same economies of scale for token making that you do for making say iPhones or cars or anything else.
[17:34] I think that is one of the huge unlocks that allows a huge group of people to go to use the best quality models.
[17:41] Condemnese scale have all made capitalism very, I don't know, fair.
[17:45] I think that allows you to go ahead and have the same product in many, many different hands.
[17:49] And you're able to go then to serve way more user than a single scale up cluster.
[17:53] of lies you'd get closer to that point for coconut serving too.
[17:55] Yeah, and also just certain products aren't usable if they're slow.
[17:59] So if you want to serve coding models and you want people to actually use them,
[18:03] like there's a certain number of tokens per second you need to hit.
[18:05] So the question is, while maintaining that per token speed,
[18:09] how many users can I serve at the same time?
[18:12] And you can basically decide, I'm going to shut off a bunch of the world from using this stuff,
[18:15] or everyone's going to get a worse experience.
[18:17] So fundamentally, you need to find ways to push out the curve,
[18:20] and that's why there's such a pressure for new hardware.
[18:23] Vanta Automates Compliance, so your team can spend less time on security reviews and more
[18:28] time getting customers.
[18:29] Vanta cuts audit prep by 82% and gives you instant up-to-date proof of trust without
[18:35] all the manual work, so you can close deals faster and with less friction.
[18:39] Customers report a 526% return on investment, and more than 16,000 companies use Vanta, including
[18:46] ramp, RV, and snowflake, get started with $1,000 off at vant.com slash invest.
[18:53] Ridgeline is the first end-to-end system of record with embedded AI for investment management firms
[18:58] running portfolio counting, reconciliation, reporting, trading, and compliance on one unified platform.
[19:04] Firms are moving off like a C technology and on to Ridgeline because of how far ahead Ridgeline's AI features
[19:10] are compared to anything else in investment management software. I've been hearing from a lot of investment
[19:15] managers about AI, and they fall roughly into two camps, with someone sure where to even
[19:19] start, and others convinced they can build their own order management system over a weekend.
[19:24] The reality is that running an investment firm will always require governance controls,
[19:28] and a single source of truth for your data, and no amount of AI enthusiasm changes
[19:32] that requirement.
[19:34] Ridgeline is built, unexactly, that foundation, which is why I believe that the firms that
[19:38] come out ahead in the AI era will be the ones running on Ridgeline's Unified Platform.
[19:43] If you're serious about your firm's AI strategy,
[19:45] Ridgeline should be part of that conversation,
[19:47] and you can request a demo at Ridgeline.ai.
[19:51] I like to take some time to step back and hear
[19:54] both of your stories for how you came to this idea in this company,
[19:58] and then kind of walk through what it's been...
[20:00] like to build it.
[20:01] Because I think in so doing,
[20:02] we'll understand the system that you've built
[20:05] for the company itself that will then be able
[20:07] to power subsequent generations of products like this one
[20:10] for this crazy inference feature that we're staring down.
[20:13] Rob, maybe starting with you,
[20:14] just take a however far back you want,
[20:16] but what I'm curious about
[20:18] and your personal story was,
[20:20] actually the very first thing I ever heard
[20:21] from either one of you was your personal story.
[20:24] Many years ago now, which really kind of blew me away.
[20:26] I'm most interested in your motivation ultimately
[20:29] for being here doing this thing.
[20:31] It starts back actually in high school for me.
[20:33] I've been very unlucky and lucky
[20:34] at different points in life.
[20:36] This is one of the tougher times.
[20:37] At the end of my sophomore year of high school,
[20:39] I got injured at a martial arts tournament
[20:41] next day, couldn't walk for some reason.
[20:43] I thought it was something wrong with my SI join
[20:45] or something.
[20:46] I went through physical therapy at a different types of scans.
[20:48] I couldn't figure it out.
[20:50] And eventually they found this big bump on my back.
[20:53] And in MRI and told me it was a tumor.
[20:55] Stage four bone cancer was told I had under 30% chance
[20:57] survival. It was like a two year crazy chemotherapy surgery learning to walk again experience.
[21:04] I mean, you go through something like that. It changes the overton window if human experience
[21:08] and makes you appreciate like what actually matters. And you also ask yourself, what are you going to
[21:12] do if you have the chance to live? If you actually want to get through something like that, you need to be
[21:17] hoping for something. And I always knew I wanted to do something very impactful if I had the chance to get through it.
[21:22] And it took me a couple of years to figure out what that was going to be. And at the same time, as I got
[21:26] got to college and met a bunch of other people,
[21:29] building cool tech.
[21:30] I got extremely excited by AI models,
[21:32] especially once GPT3 came out.
[21:34] And I was like, wow, this is the first model
[21:35] that can kind of speak English.
[21:37] And these things aren't really smart.
[21:38] What happened was when GPT4 came out,
[21:41] there was GPT4V, which was the first model
[21:43] with image uploading.
[21:44] So I went through my camera role
[21:46] and I found a picture of my back with this bump on it
[21:48] before I was diagnosed.
[21:49] And I said, hey, CHPT, pretend you're an expert doctor,
[21:52] a patient comes in, and they says they have this bump
[21:55] on their back, what could it be?
[21:57] And it immediately says like,
[21:58] this could be a tumor, you should get an MRI,
[22:00] and immediately go to the doctor,
[22:02] and I just kind of sat there still.
[22:04] It was like, it took me six months.
[22:06] And yesterday, this feature wasn't there.
[22:08] Today it's here, I go to show my parents,
[22:11] and I got this notification being like,
[22:13] you're all out of image credits today.
[22:15] Like you need to see if you can get a pro-flat,
[22:16] and I was like, holy crap.
[22:18] Like this is gonna change everything.
[22:20] And we clearly don't have the infrastructure to serve it.
[22:22] And there's very few things you can work on
[22:24] that can actually bring this technology at scale to the world faster.
[22:28] I mean, there's plenty of people that are super smart working on models.
[22:31] And the fab seemed maybe unreachable to work on, but it seemed like the hardware was all designed before
[22:36] chat GPT. Every GPU, every GPU, every AI chip that was serving these models were just fundamentally built before
[22:44] this and our retrofit to serve these modern models, there's going to be an entire new wave of architectures that came out.
[22:48] And what a more exciting thing to work on than bringing this to everybody. Very different angle at the same time.
[22:53] I was running a startup incubator called prod, which is incubated a bunch of different companies.
[22:57] Some of the earliest ones being cursor on any sphere which merged and Merch Corps and
[23:01] etched went through it and a handful of others. At the time, as these models were getting smarter,
[23:05] it was the 2022, I was realizing all of these companies are spending all the money they raised on compute.
[23:11] And I had this realization as I was working on some of my own stuff that like, oh my god, all the
[23:15] products I want to build are going to cost tens of millions of dollars a year in inference.
[23:18] Hey, this is not going to be tenable.
[23:20] Like the cost structure of every software company in the Cogs is not going to be like zero anymore
[23:25] for an incremental user.
[23:26] It's going to be like quite high and it's going to be a function of inference.
[23:29] And then the topics of every business is going to also be in front of people use more and more
[23:33] coding agents.
[23:34] So fundamentally, it seems like inference is going to be really important.
[23:37] And it feels like we are on a decade march for inference to become the biggest market in
[23:41] the world.
[23:42] So when you think about that, 10 years from now, if there's going to be these giant projects
[23:46] or everything in that data center fundamentally hasn't been designed today.
[23:49] We should go pick something and work on it.
[23:51] That's kind of how it got started.
[23:52] Yeah, I'm really excited for you to go back about as far, probably early in high school, maybe even earlier,
[23:57] and tell your favorite hash marks on the timeline that ultimately led to your ambition to drop out of Harvard and start this company.
[24:04] But first of all, that was that a company called X-NOR, where I did a criminal development.
[24:08] I was 17 and a 17 year old can't sign a legally-boundary contracts.
[24:13] So I then go ahead and do a traditional CIA.
[24:15] Then we had to sat me down and said, Gavin, don't wish I had this information.
[24:20] And Expo was one of the only companies that saw it,
[24:22] hey, maybe this is a good trade, and afterward building kernels.
[24:26] And a number of other companies sent, it's not about Bob out for $400 million,
[24:29] did the same thing at Octo, they got bought by Nvidia for $100 million.
[24:34] But when you do this for the kernel's work, what you realize is that the math is relatively easy,
[24:39] but to get high speed decode, I think the math is a data movement.
[24:42] Almost all the work that you do is optimizing how do you move data around a single chip for a cross multiple chips?
[24:49] Yes, why we went ahead and built this cluster scale memory tech.
[24:52] We bring that interconnect time way, way, way lower.
[24:55] You can do it way more movement.
[24:56] And as a result, get a much faster time to generate each subsequent token.
[25:01] And build these crazy things we're all talking about
[25:03] for doing a year's worth of work in a month or more than that in the future.
[25:06] Can you talk about the competitive drive that's evident in some of the high-school competitions
[25:12] that you participated in and won.
[25:14] We did a couple.
[25:16] For example, I was very active in it after zero about it.
[25:19] I was lucky to have a very talented partner, Safra.
[25:21] For a long time, we had a professional school team,
[25:24] where we were about 20 guys, all working together as an
[25:27] authentic school at first-tech challenge.
[25:29] And the goal is to build a robot that scores the most points in a much
[25:32] other things too.
[25:34] In first, they put a lot of emphasis around collaborating with other teams,
[25:38] around trying to do really good documentation,
[25:40] around trying to go ahead and get others inspired to go do the same thing.
[25:43] So for an eye-to-side, we're going to go ahead and do it this way, we're going to win.
[25:47] And we did nothing else, besides it bolder about the score to the most points.
[25:50] As a two-person team, a two-person player than a twenty-person team, that we were much
[25:54] much smaller than almost every other team in the competition.
[25:56] We figured that if we were going to go a specialize, if we were going to go out and do this,
[26:00] when the damn game was really well, we would need to go ahead and advance based on the quality
[26:04] of a documentation or of our outreach.
[26:06] We were just going to go win.
[26:07] And so we did, we embraced off both the two percent team.
[26:10] But both they were about to decide if we were going to go ahead and redesign it every three months.
[26:14] And we did.
[26:15] We actually had the world record for the highest board during this competition at one point.
[26:19] We were rated by OPR third in the world for software development.
[26:24] And it was a damn good machine.
[26:25] What from that episode can I translate as an analogy on how you built that's the company?
[26:31] We think how you want to go ahead and do a full Raskel product like this for the couple key ideas.
[26:37] One of them is like velocity velocity velocity velocity.
[26:40] That you win by shipping.
[26:42] You're not going to go out win by having the best outreach or the best communications.
[26:47] You win by building the best product.
[26:49] And similarly, we think you can do it with a lot of fewer folks.
[26:53] That if you're willing to go ahead and just focus on product product product and
[26:57] Paralyzer letlessly, you don't need 20,000 people like the big companies have.
[27:02] You can do the best product in the world with far fewer people.
[27:05] You know, they're just saying of the best part is no part.
[27:07] I think for us, it's also the best vendor is no vendor.
[27:10] As much as possible, we want to vertically integrate the entire product.
[27:13] Both because we get more performance, but we can move way faster.
[27:16] So everything from the chips to the boards to the cold plates and the interconnects to even the
[27:20] production, we want to do all of it as in-house as possible.
[27:23] I think we're the only startup right now that's building its own rack, as well as its own chips.
[27:27] And we did it all at the same time. A couple years ago, as a last time we were public,
[27:31] at that point we just started building our acting.
[27:33] and we brought over by an lawyer who built all of Nvidia's HGX and TGX systems, which is like
[27:38] 80% of the revenue. And we said we're going to build the rack at the same time.
[27:42] We actually went through multiple iterations of the rack before the chips even came back.
[27:46] Before the chips came back, we made thermal chips to had the exact same hot spots as we expected
[27:50] our chips to have, so we could build the code plates, we could overpressurize them and blow them up.
[27:54] We haven't had a single leak since our chips came back with the code plates, because we already
[27:57] validated them. We have a factory and Taiwan only, a few dozen people out there. We built a
[28:02] a clone of a bunch of the test stations in our office.
[28:04] We have a two mega watt data center on this floor, and we did 24, 7 development cycles,
[28:09] people are doing dash shifts and night shifts to actually get the hardware up and running.
[28:12] It's quickly as possible.
[28:13] But it's that extreme vertical integration and extreme parallelization of the schedule that lets you
[28:17] get products to market way faster.
[28:19] If you think about the building of the early team and what it required as two young guys,
[28:24] building just company, there's lots of very talented young entrepreneurs out there.
[28:28] Maybe for the first time of this scope or magnitude in the long time,
[28:32] all of whom probably can benefit from the lessons that you've learned,
[28:35] getting very sophisticated talent and people that come join you even after careers at the other great companies.
[28:40] If you were teaching this as a class, like here's how to get a lead talent when you're a young and
[28:44] inexperienced and naive, what would be this office? We were pretty by modal talent philosophy.
[28:49] It starts with what we call the legends, which is when we're trying to solve an incredibly hard
[28:54] technical problem and generally do something that hasn't been done before.
[28:58] We need to find the very best person in the world.
[29:00] And often the number one guy in the world versus the number 10 guy versus the number
[29:03] 100 guy huge difference in whether it's actually possible to solve the problem.
[29:07] We created the system we call project based recruiting where we map out all of the
[29:11] hardest technical problems across all industries that anyone has ever had to solve.
[29:15] We look at temporality.
[29:16] So who are the people who did the zero to one?
[29:18] Who is in charge, quote unquote, who actually did the work?
[29:21] We talked as many people as possible.
[29:22] And then we just track it. And you'd be surprised by the amount of people who say yes after the first conversation is pretty low,
[29:28] but the amount of people who say yes after the 20th conversation is surprisingly high. You really gotta keep that up.
[29:33] When you hear no from somebody who really is the best from the world, then that really means, hey,
[29:38] she doesn't come back when you had a few more miles to get up. I think it's one of the most convincing things to see is hey, we make bold claims.
[29:46] And when you hit those again and again and again, that is really belief and inspiring.
[29:50] When we decided we wanted to build a rack and not just a chip.
[29:54] We were looking at this and we were saying,
[29:56] how many products have actually shipped that scale?
[30:00] for a RAC scale system that actually have the power density that we're trying to solve.
[30:04] And we just kind of said, if we were going to wave a magic wand, what would the best possible person in the world look like?
[30:09] And we're like, well, if we could find somebody who started that in video and built the entire RAC team through all their different generations,
[30:17] learned all this different stuff, but still scrappy, still understands the start of culture, but a scene scale, like that would be the best possible person.
[30:24] So we mapped all of the different teams that related to all of the different RACSKL products of video.
[30:28] And we found three people that we thought could fit the bill.
[30:32] And we talked to all of them and two of them have just retired.
[30:35] And one of them was planning to do one more generation for our video and then retire.
[30:40] The same is Brian.
[30:41] And over time we convinced them to join.
[30:43] Brian started the HGX and DJX team at a video, which was a majority of Nvidia's revenue,
[30:49] tens of billions of dollars a quarter.
[30:50] And the other two guys ended up investing by the way.
[30:53] When you have somebody like that, they just know what good looks like,
[30:55] because they've seen it.
[30:56] And there's so many times where we'd talk to Brian,
[30:58] and you just point to us and be like,
[30:59] that's a billion dollar, like a billion dollar lesson I learned,
[31:02] a billion dollar lesson I learned.
[31:03] And like, yeah, that just saves the cycles.
[31:05] And you pair someone like Brian with somebody like Sanford.
[31:09] Do you have any name for them?
[31:10] So Brian's a legend.
[31:10] What's what Sanford?
[31:12] Yeah, we say chips on shoulders,
[31:13] put chips and data centers.
[31:14] Yeah.
[31:15] Sanford and Gavin in high school were rolled robotics champions,
[31:19] and Sanford was finishing a senior year of college.
[31:23] And we called him up a couple years ago,
[31:25] and we said, hey, can you come check out what we're doing?
[31:27] We need some help on the platform side.
[31:29] He comes for a week and we say,
[31:31] can you build a cold plate this week?
[31:33] And if you asked any thermal engineer,
[31:36] anything like that, they would think you're just
[31:38] totally naive.
[31:39] I mean, these things take months to do.
[31:41] And to be clear, they do.
[31:42] But you can make real progress in a week.
[31:44] If you put your mind to it and you think it's possible.
[31:46] And he built the contraption in a week
[31:48] that like actually de-risk like a pretty key power question we had and you put those two together
[31:53] and they've done incredible things. What is that possible about the other because you need
[31:57] the extremely driven people that just keep asking why and don't know where the bodies are buried
[32:03] to like take tons of aggressive risks and then you need the people who've seen scale and still have
[32:07] the startup scrapping mentality to help them along the way. It's really the legends plus some naive
[32:12] a TA, you know, offers principles type talent. It's not just that you have both in the company
[32:16] and said they're working together.
[32:17] That's right.
[32:18] If I think about that funnel,
[32:19] anything else more interesting to say
[32:21] about how much better you've gotten at recruiting
[32:23] and why those metrics keep getting better?
[32:25] One of the shocking things is
[32:26] I'm sure it means a different theory in Bet.
[32:29] Kind of self-selects.
[32:31] That like, sure the kind of person
[32:32] who is some opportunistic,
[32:34] shouldn't go join wherever the hot company is,
[32:36] or I think go ahead and do the diligence,
[32:38] you will not come work here.
[32:40] And just one of the things I've worried about
[32:41] as we announce more and more of the product in its specs,
[32:44] we may have loosened it,
[32:44] So we're not very careful.
[32:46] You kind of have to be sick in the head to join our company.
[32:48] But you think about it on paper.
[32:50] It's like you, a person who is probably
[32:53] a very accomplished engineer,
[32:54] they can be good amount of money,
[32:55] it's liquid, it's predictable somewhere else.
[32:57] You're going to convince your family to move to San Jose
[33:00] and live in this apartment on this housing program
[33:03] for the semiconductor company run by two,
[33:05] 24-year-olds now, that's pre-product,
[33:08] that is going against the biggest companies in the world,
[33:11] and the most supply constraint environment ever created,
[33:14] with a design that they're saying is not going to be like 10% better,
[33:17] but it's going to be 10x better.
[33:18] Something must be wrong with you to do that.
[33:21] People are just wired differently here that they really want to not prove people wrong who don't believe,
[33:27] but prove people right who do believe.
[33:29] They just take it personally.
[33:30] And that's really fun to find those people.
[33:32] And frankly, just the nature of the company makes it very easy to whittle out of people who aren't like that.
[33:37] One of the very first things you and I talked about Rob was,
[33:40] I started asking about servo, which is the name of the first product here.
[33:43] And you said we can talk about that in great detail.
[33:46] But the thing you should know is that what we're really focused on is building a machine
[33:50] that can at scale produce these things and generations of them as efficiently at the highest
[33:56] possible quality levels. We want to build the company or the machine that is the company is the
[34:00] thing that will produce this thing and then subsequent things. So I'd like to talk about a few
[34:04] principles or cornerstones of the company. We've alluded to some of them. You said velocity,
[34:09] said, vertical integration is to become more popular topics. Parallelization is something
[34:15] maybe that we should talk about. But I'm especially interested in your guys' willingness to take
[34:19] huge risk to go faster. Maybe tell your favorite story about why this is the philosophy,
[34:26] what it's allowed you to do that maybe other companies haven't done as a number of stories here.
[34:30] But one of my favorites is there was a time where we were getting close to taking out the chip.
[34:35] We realized, wait a minute, one of our vendors is a way way behind schedule and we have two very bad options.
[34:42] One option, just keep the current vendor and push our down-lines out by on the order of a year.
[34:48] Another option, let's say, if we switch vendors, start over and also go push timelines out by a year.
[34:53] Neither of these was a good option. So we had to look for option number three.
[34:57] And what that was was we say we're out there all in Bangalore.
[35:00] But in the end, it's actually going into doing the work.
[35:02] We went out into the ship.
[35:04] It does not have our top engineers across the world,
[35:06] two Bangalore for six months.
[35:09] That was there as well.
[35:10] I left in Bangalore for four and a half months personally.
[35:13] At every morning, we'd go ahead and walk across
[35:15] at Crazy Busy Bangalore streets into the office.
[35:17] We'd be the first ones in.
[35:19] We'd go out and put a wide variety of tools.
[35:21] Both things like, hey,
[35:22] odd thing I huge amount of the code that was going in,
[35:25] building a bunch of tools of all to make this go even faster.
[35:28] Making sure we're making the right design decisions.
[35:30] on the spot right there, note,
[35:31] to allow our back at force to head and decide immediately.
[35:35] And then at one end, we'd go walk back through the
[35:37] now empty bank groceries and that do it all again the next day.
[35:41] We still had a bunch of the team in the US.
[35:43] We ran these 12 hour on each side,
[35:45] handoffs, where we had 24 hour developments like over at 8 a.m.
[35:48] and 8 p.m. every day.
[35:49] We'd all get on the Zoom.
[35:51] We'd share all the day that we'd say,
[35:52] when I wake up, like this must be done.
[35:55] Like we must get this chip out.
[35:57] And it was extremely intense.
[35:58] At the same time, we saw other chips at the same stages
[36:01] as us with that same vendor that ended up taking years.
[36:04] That's still aren't out today.
[36:05] That's still haven't even taped out today.
[36:07] And it's that level of extreme urgency that's required
[36:10] to bring products to market.
[36:11] What is the key to doing this while this has become a trope
[36:14] because it will be lawnmowsly that like his special skill
[36:17] and others that seek to emulate him would try to do this too.
[36:20] He's figure out what the binding constraint is
[36:23] and just flood the zone personally on that thing,
[36:25] which is kind of like going to Bangalore or something.
[36:27] It seems like this is a central tenant of the business and of any business that's going to do this kind of vertical
[36:33] integration. What's the key to doing that well? Like again, what have you learned about that specific act?
[36:38] For me, I think there are two key tricks to this. The first one is that you can't build a chip alone.
[36:44] It's a gotta be a team problem and you're most important job to go get great people to go with you.
[36:50] And if you go ahead and be inspired and excited to go ahead and do crazy things like this,
[36:54] That is a huge act to say, hey guys, up your lives for six months or in one case, 12 months.
[37:01] That way it sent one guy out while ahead.
[37:03] It sucks, but we're lucky to have team members who are enough for the right reasons.
[37:07] By the way, the second big thing too is being able to make decisions very fast.
[37:12] That one of the worst items is when there's a factory where there is a vendor who is waiting for you to go ahead and make some call and is just stalled.
[37:21] And this happens all the time, even for very small things.
[37:25] To send folks to get a big amount of responsibility to them and say,
[37:30] make a reasonable call.
[37:32] Okay, if you're wrong every now and then,
[37:34] but I would have much, much rather be right,
[37:36] most of the time they give an answer immediately,
[37:38] than wait every time for the perfect response.
[37:41] Speed wins.
[37:42] What about spending money to go faster?
[37:45] There's this learned by doing things
[37:46] which has become so interesting
[37:48] and as the world has gone away from software and towards
[37:50] more hardware again in the world of technology that we outsource so much of the
[37:55] learn by doing by shipping stuff overseas and effectively just being the idea guys
[37:59] here in the US.
[38:00] Tune back that obviously is reversing and you've adopted this way of learning by doing like
[38:04] you want to be in that iteration learning loop.
[38:07] And part of that is willingness to spend and take risk with dollars.
[38:11] Yeah.
[38:12] Can you talk about that a little bit?
[38:13] I think it was a great quote of like the biggest risk is not taking risk.
[38:16] Very similar here, which is like every day there's over a billion dollars of revenue in this category.
[38:20] and a lot of its inference. So every day we don't ship, we're just leaving tons of opportunity on the table.
[38:26] So you're willing this to spend money should be extremely high if you can get a very clear ROI out of it.
[38:31] So we have this concept that we call pre-fetching, which is when you're waiting for one thing to get done,
[38:37] when you know you're going to do other things once you have it, is there ways that you can
[38:40] paralyze the entire schedule? So for example, like we know our trip is going to come back in a certain date,
[38:45] We wanted to be that everything possible that can be done without the chip is done before the chip lands.
[38:51] And this cost a lot of money. This means that we want to build our entire software stack beforehand.
[38:56] Like we shipped racks to customer data centers without our chips and them with all the networking, all the CPUs, all the storage, all set up.
[39:03] So we could bring all that data centers off or up before the chips came back and meant that we took over 700 FPGAs and put the entire full,
[39:10] medical chip on an FPGA cluster and ran a dozen different models with our full inference
[39:15] stack on them before the chips came back. It means that we built a thermal chip to mock
[39:19] the thermal profile of our chip and built cold plates based on that before the chip came back.
[39:24] It means we had the entire production line ready. It means we did many revs of the circuit board.
[39:27] It means the entire product was ready to go before the chips came back. And this is what it
[39:32] gives you. There's another very famous AI chip company that took 10 months to go from getting
[39:39] their silicon back to having them running inference in Iraq.
[39:42] And this was like publicly announced to their investors and it was a really big deal.
[39:46] We were able to do it in 40 days.
[39:48] And it's because by the time the chip came back everything was boring.
[39:52] The software was already written.
[39:53] The rack was already there.
[39:54] The production line was already set up.
[39:56] We were just go, go, go, go, go.
[39:58] Get everything together.
[39:59] You don't always-
[40:00] at everything, you make some tweaks on the fly and then off you go.
[40:03] And that particular case too.
[40:04] That was a big part of it.
[40:05] It also like the shift, I think, made a big difference.
[40:07] Still totally.
[40:08] We went out and literally had a day shift in a night shift.
[40:11] There were team members who would come in a 10 AM and leave out around midnight.
[40:15] We'll just come in at midnight as leave a 10 AM.
[40:18] You run it around the clock to get to that 40 days.
[40:21] I mean, over half the company lives next to the office.
[40:23] So it makes it easier to do that type of thing.
[40:25] You pay them to do that, right?
[40:27] Pay them extra and you still do that.
[40:28] The invisible hand does wonders.
[40:30] I mean, hey, it works for me too.
[40:32] We're both there.
[40:34] I looked like one big step back
[40:35] and talk a bit about just the broader ecosystem here.
[40:38] The amount of shortages on a supply side,
[40:42] the exposure of risks in the global system
[40:46] and the supply chain around this stuff
[40:48] has become like every day,
[40:50] Wall Street Journal from page news,
[40:52] like the stocks that people are watching
[40:53] and investing in and excited about
[40:55] about the memory stocks.
[40:56] These were born in commodity, like nothing burgers five years ago, and now that the center
[41:01] of global attention, if you just assess because you've been building in it, the global
[41:06] connected supply chain that's required to make stuff like this possible.
[41:12] Just riff on it.
[41:13] What scares you?
[41:14] What's working well?
[41:15] What needs to change?
[41:16] What do you hope you change by virtue of how you build this thing?
[41:18] What's your assessment of this story right now?
[41:20] I like that one of the most undervalued pieces of a supply chain story is,
[41:26] Almost none of these things are, you buy them and they don't talk to them to begin.
[41:31] You have to go collaborate. That is the most important part of the process of
[41:34] creating a system with TSMC or memory vendors. You need that partnership.
[41:40] I think that for TSMC in particular, people don't understand why it is so valuable.
[41:44] People look at the tech and the tech is the best in the world. But for me, the
[41:48] the real value is all in these service.
[41:51] TSMC customer service is way, way better, and I have to hand at any other company in any other
[41:57] industry.
[41:58] If the kind of thing where if you say, hey, we're going to prove your yield by making this
[42:02] change.
[42:03] You can go make them a recommendation and they will go one next to the experiment.
[42:07] On their own dime in our case, see if they could actually get the higher yield.
[42:11] And when we found that we were right in the history of our work, they moved over the rest
[42:15] of the line.
[42:16] that kind of thing doesn't happen,
[42:17] the most industries.
[42:19] If I go to like the steelworks plant,
[42:21] say hey, I want you to change the competition of the steel.
[42:23] And I'll say screw you.
[42:24] Not TSMC.
[42:26] It is why they are the number one
[42:28] and why they're going to win.
[42:29] One of the things that matters a ton
[42:31] is power availability and time to power.
[42:34] And the problem is the more power you want,
[42:36] the more shortage of the rich,
[42:37] it's actually very similar to chip clusters,
[42:39] which is like,
[42:40] why is Colossus charging $12 an hour for black walls?
[42:43] It's because they're the only place
[42:44] You can buy 20,000 of them at once, right?
[42:46] Why is the 500 megawatt data center so hard to find?
[42:49] It's the exact same reason.
[42:51] One of the things we need to think about is
[42:52] how do we get way more juice out of each megawatt?
[42:55] People are looking throughout the entire stack,
[42:57] whether it's just improving the PUE,
[42:59] but also entirely new hardware to get the most tokens
[43:02] for megawatt to solve this problem.
[43:04] But fundamentally, building new buildings is hard.
[43:07] It's much easier to go from 100 megawatt to a gigawatt,
[43:09] that from a gigawatt to 10 and 10 to 100.
[43:11] And we are pushing the limits of what's possible
[43:13] these timelines. So there is a lot of people trying to scale in their data centers as much as trying to scale them out.
[43:19] One of the interesting things about a system like this is what it replaces.
[43:23] Yeah, if I think about a rack like this versus, I don't know, the set of black wells or something,
[43:27] a Rubin's or whatever's coming next. How should I conceptualize that? It's not just what's
[43:32] a sort of physical space to re-versal talk about this. And there recently was called that literally
[43:35] this is a problem that there's literally no space to put the systems. How should I conceptualize what
[43:40] this represents or replaces in terms of other units of compute.
[43:44] Here's how customers think about deploying models generally, which is when I'm building a data
[43:49] center or I'm building a cluster, it's not like in the abstract of like, oh, I like these
[43:53] chips and like, this is the power footprint and so forth. It's like, I have a real production work
[43:57] load I'm trying to serve and for my product to be useful, there's a certain speed I need to
[44:02] serve for that. And for certain products, it's really fast and certain products is really slow.
[44:05] whatever the speed is, this is my speed. The question is, in a given amount of power, how many users
[44:11] can I serve while guaranteeing that speed? So another way to put it is ISO, what's called interactivity,
[44:17] what is my throughput? We are just finishing kind of the early innings of the AI infrastructure
[44:22] boom, where people really just cared about speed. You know, GPUs were not able to reach a lot of the
[44:26] speeds of other types of chips like all these SRAM chips, thousands of tokens per second, and that enabled
[44:31] tons of new use cases that got people very excited. There's an entirely new wave of AI chips
[44:35] us being one of them, they're all going to be able to hit these speeds. The question then is,
[44:39] if you're hitting these speeds, what is the number of users you can serve at the same time?
[44:43] And by proxy, if I have 100 megawatt data center, how many software agents can I run at the same time?
[44:48] So when people are doing that evaluation, our hardware is going to generally be able to get you
[44:53] an order of magnitude more concurrency at a given level of interactivity. So that directly translates into
[44:59] tokens for what? tokens for dollar? All the things people care about when they're actually serving.
[45:03] these giant make sure of export models. That's hell. There's these now famous
[45:07] interactivity curves. Not many people publish them, but you can see a blackwell curve. You can
[45:12] see an AMD curve which is a little bit worse than blackwells and it's still an $800 billion
[45:18] company. So if you think about what then the impacts are of shifting that curve, not just a little bit
[45:23] further out, but much further out. Yeah. What are the things that most excite you about what this will enable?
[45:29] I want to go out and solve some of the hardest problems, and I want to go solve these in
[45:33] less and less time. There were things growing up, but I was not sure I'd be able to live
[45:37] this city. For example, the unit is a good gesture. One of the things we talked about in college,
[45:42] and I was not sure I'd see that prove it in my life. And this was done by an AI model, and it was
[45:46] not over a long period of time, but if you're able to run the same model 10 times faster,
[45:51] you can go shrink the time to go have these breakthroughs. And there's a huge number of other
[45:55] problems and math like this as well. That I worry it will take thousands of years to
[45:59] go through a thing like this. Do you either have a much smarter model or a model of the same
[46:04] intelligence running much faster? You can shrink that and I can see it. It's so cool seeing
[46:09] these breakthroughs get made. I am so excited to see much more of this happen.
[46:14] I think too often people think about tasks and applications and stuff in these very short time horizons.
[46:20] doing a chat and it's like 50% faster is nice, but it's not like game changing.
[46:26] As these agents go longer and longer time horizon and the models get more and more capable,
[46:30] you're going to see gigantic bodies of work that would take months of compute.
[46:35] And we think about this in Wall Clock Time. Like if you talk to a pre-training researcher at
[46:39] a lab, they'll tell you that Wall Clock Time often is one of the most important things that matters.
[46:43] And what Wall Clock Time means is the time from starting your run to finishing it to actually get
[46:48] data back. If you can shrink this time from a six-month run to a two-month experiment,
[46:53] you're going to be able to do many more iterations and people will make changes on the model
[46:57] our connectors to actually improve the wall clock time. Very similar here in terms of how we think
[47:01] about the use cases, which is the exciting part about super low latency decode, is wall clock time
[47:07] on long horizon tasks becomes much shorter. So a year long compute build would now take a month,
[47:14] and a month long compute build will now take three days, not three days, compute build will now take
[47:18] seven hours and so forth and so forth. So that's the thing that I think is really hard to
[47:23] internalize because the models are just getting capable enough to do this stuff. I thought it was really
[47:27] cool months ago when cursor published that they had a bunch of coding agents build an entire browser
[47:32] from scratch in a week, totally nuts. And that will soon happen in under an hour, and there's going to
[47:37] be many of those types of things that are going to happen with these massive parallel agents all working
[47:42] not a given task. What are the ultimate limitations of these systems? Is it just like a physics
[47:47] question, like how many times faster cheaper can we get theoretically? There's a lot of
[47:54] room at the bottom of this. If you think about chip and chip late at six on an Nvidia product,
[47:59] it looking at 4,000 nanoseconds to go from one chip to another. We'll be able to do much better than that.
[48:04] Well, it's the mathematical limit. His speed of light. You can do it in just a handful, like two,
[48:09] two, three, and a second.
[48:10] And there are 4,000, 4,000 today.
[48:13] There is a lot of room at the bottom,
[48:15] a same thing as like power efficiency.
[48:17] That's sure we're able to go and use a mount
[48:20] by bringing it up to down by so much.
[48:22] But you can go lower.
[48:23] You can go much, much lower.
[48:25] It's very challenging, but why I think about 20, 30 years
[48:28] in the future.
[48:30] Then I think it's inevitable.
[48:31] And also for a economy's scale, for a cluster scale up.
[48:35] For a long time, ageups was the biggest
[48:37] a scale up to a vein. If they didn't have the NL72, bringing it to a 72. But you can
[48:42] be way, way bigger. You look like a fab for example. You have a fully billion dollar, single,
[48:49] monolithic building, with only a handful of lines running through it. You could have the same kind
[48:54] of thing for some futuristic, mega-cluster, $40 billion, $100 billion as a giant, mega-took
[49:00] factory, serving one or a handful of models for a massive number of users. To get that same
[49:07] the economies of scale thing, same model, massive number of people.
[49:10] You mentioned kernels engineering and that been your first job.
[49:15] That has emerged as a thing that nobody had ever heard of in their lives to now something
[49:18] they hear about all the time, the importance of it, to eke more performance out of the
[49:22] bare raw metal.
[49:24] When will that just be something that AI does entirely as well?
[49:28] Are humans still the best kernel engineers?
[49:30] Are they doing it with the assistance of AI systems?
[49:33] like how far down will humans still be in the loop of designing these things?
[49:38] Like when will that go away?
[49:40] Today it's all very hybrid.
[49:41] And the best kernels are still written by human AIC collaborations.
[49:45] Also, any AAMiles built with both these fundamental primitives, like Matt Moles, like convolutions,
[49:50] like a chip-the-chip operations collectives.
[49:53] And making these overlap and making these really fast matters enormously.
[49:57] And as the kernel of the ladders job, it took a hidden cigarette.
[50:00] Or connect over a lap, how do I allocate memory, how do I verify that if there's some issue, like every transmitit, it doesn't stall the whole pipeline.
[50:08] These things are very challenging, but they can go and make your wallet form to be, say, 3% better optimization, and you can do so many.
[50:16] And we thought about our software stack. We wanted to go to the skate-art where the puck is going to be.
[50:20] And three years ago, number kind of two ways you could build software. One of the most effectively and graph compilers.
[50:26] These things are not very performant, but they work out of the box.
[50:30] They don't require a human to go come in and tweak all the kernels.
[50:33] But we went the opposite direction.
[50:35] We are kernels first programming, and that means that it's for a long time to not work out of the box.
[50:41] But if you work kernels experts, you can get incredibly, incredibly high performance.
[50:45] And the thing about this is that now, as the coding models get better and better,
[50:49] they're doing more and more of the kernel generation task.
[50:52] And when the models keep getting smarter, they'll eventually do all of it.
[50:55] all of it, it will become a superhuman. So we're going to build before the world is going.
[50:59] And even today, we think about our profiling tools or debugging stack. We think about it from
[51:05] how will the model use these tools, more than we think about how humans use these tools.
[51:09] We sometimes run experiments internally and we had codex actually get GPTOSS running from scratch
[51:16] just based off of our docs completely by itself and did it in the overnight. We think about
[51:21] gain selection a lot and what we mean by that is making sure we're investing our energy in the right
[51:27] bets because regardless of what you choose to work on it will take tremendous effort and one of the
[51:32] things that we started with was you know the decision explicitly not to build an arbitrary graph
[51:36] compiler not to support arbitrary pie torch not to support arbitrary kuda not to support arbitrary on
[51:41] x graphs but instead we envisioned a world where there was going to be under 100 models that actually
[51:47] They were all going to look very similar from the underlying mathematical perspective, and that we were going to build primitives using physics that we're going to accelerate these as much as humanly possible.
[51:58] And we were going to allow the most sophisticated customers to have direct access to the hardware and do whatever they want.
[52:03] And that has saved us a tremendous amount of time not having to build a compiler, and that has allowed us to actually get much more performance.
[52:08] And finally enough when we started, a lot of people dismissed this idea, and the only people that took a seriously were in high frequency trading.
[52:15] They'll hate compilers too.
[52:17] They'll write their own kernels.
[52:19] And we've had dozens of people from high frequency trading
[52:21] join the team because they saw this philosophy, too.
[52:24] What are the limits to vertical integration?
[52:26] How do you know where to draw the line?
[52:28] And I'm starting with this question to talk a bit about the broader
[52:31] market.
[52:32] The circumstances of the broader market are really interesting to me
[52:35] where the vast majority of chips of AI chips get bought by a very
[52:39] small set of customers.
[52:41] Many of those customers are themselves trying to design their own
[52:44] and AI chips, OpenAI and dance, Taltino.
[52:47] It seems like there's very funny circumstance for
[52:49] the most valuable thing in the world,
[52:50] all kind of flows through a couple, chip makers,
[52:53] a couple chip buyers.
[52:55] They all seem to be thinking about doing each other's job.
[52:57] And then you've got the circumstance for like,
[52:59] okay, then these things go in a data center,
[53:01] and you've got Neoclouds and inference providers
[53:03] and this other part of the stack,
[53:04] you've got model builders and providers.
[53:06] Like I can imagine a world where,
[53:08] because you have the best hardware,
[53:10] you design models and you build data centers.
[53:13] unique outside of your current vertical.
[53:15] So like, how do you think about where to draw the lines for the business?
[53:19] We have a saying that production is the product.
[53:21] Ultimately, what matters here is we know in front of this going to be the biggest market in the world.
[53:25] Whoever produces the most tokens is going to be the most valuable company in the world.
[53:30] So all the decisions we make is how do we get the most token capacity online as possible?
[53:35] And part of that is building a really good product that has way more throughput.
[53:39] that can run away better late in season so forth.
[53:42] So we can purr chip we make, get way more tokens online.
[53:45] Another part of it is not doing parts of the stack,
[53:48] unless we absolutely have to to get the giant scale.
[53:52] So there are parts that we decided to do
[53:54] because it was absolutely required to get the scale,
[53:56] like building the rack instead of just building the chips.
[53:59] Like doing a CM model instead of doing a JDM model.
[54:02] But there are parts of it that are kind of noise to us right now.
[54:04] Like we're not going and building our own data center today.
[54:06] That doesn't actually help us get more capacity online.
[54:09] In general, our customers are actually making power and moving their clusters around to get our chips online because they're such high throughput.
[54:15] If there was a world where other things are constrained, we would totally go and integrate with them.
[54:19] But the reality is we're just purely focused on getting as many token online as possible.
[54:22] I think this comes down to economies of scale again.
[54:25] I sort of part to the stack and there are huge economies of scale and others there aren't.
[54:29] For example, on designing models, huge economies of scale there, for chip fabrication, same story.
[54:36] But for example, if you think about building some
[54:38] small metal part of that rack, there's not that same effect.
[54:41] We think the natural boundaries are on the chip side, on the bottom,
[54:44] as a modelator at the top.
[54:46] And we'll fill the whole gap between.
[54:48] A few weeks ago, there was a guy who was running a next-generation AI chip
[54:52] for one of the frontier companies.
[54:54] And he's trying to recruit one of our architects
[54:56] and actually kind of didn't do a reverse card.
[54:59] Instead of recruiting the person trying to recruit our guy,
[55:01] then within a week we hired him.
[55:03] And I was going on a walk as we were kind of finalizing the offer.
[55:05] I was like, you're leading the SuperMported Project.
[55:07] Why are you deciding to join?
[55:09] And his answer is super interesting, which was,
[55:11] if fundamentally is not existential for my company,
[55:15] for this product to win.
[55:17] For Google with TPUs, the revenue comes from search.
[55:19] Google won, fail of TPUs.
[55:21] That's right.
[55:21] Meta won't fail of MTIA fails, Microsoft won't fail of Maya fails,
[55:25] and opening AI won't fail jalapeno fails.
[55:27] Ultimately, this is our product.
[55:30] It is completely unsurprising that the best chip in the world
[55:33] is built by a company that only builds that chip.
[55:35] But it's in video, right?
[55:37] And like, for us, it is completely existential for us to get
[55:40] as much token capacity online as possible.
[55:42] It recruits a set of talent and retreats to support from suppliers
[55:45] and from customers that view it with the level of intensity
[55:48] that we do.
[55:48] Look at their off-flop steps.
[55:50] You will compare any of these chips built by the labs
[55:53] or by the hyper-scalers.
[55:54] The flop density for say FD at times FD8 is lower
[55:58] than the black-old B300.
[55:59] And that makes sense.
[56:00] They don't have to take their risk.
[56:02] They just have to go and build a similar enough product
[56:05] and not pay the Nvidia decks.
[56:07] As I think about you guys building the solution,
[56:09] the process of doing so is solving a sequence
[56:11] of really hard challenges.
[56:13] What has been the single episode that was the hardest
[56:15] to overcome?
[56:16] We were designing the chip.
[56:17] We built this massive FPGA cluster to go and verify
[56:21] the full chip work desks.
[56:23] And FPGA's are digital entities.
[56:25] You can get a digital logic, but not analog logic.
[56:29] And it turns out that when the chip came back,
[56:31] we began to go see issues in our TensorFlow adaptation
[56:34] of incorrect results. We realized, wait a minute, there's a problem with a back-pressure
[56:39] and logic across a clock-admin crossing is failing. This is a good way to do a positive
[56:43] to prove wrong results and is very hard to solve. We realized there was one and only one
[56:50] way to solve it. As we had to do a line up two clock signals on our chip, two within 50
[56:55] pico seconds. That is literally 50 trillion a second. We had to go get this signals aligned
[57:02] to this super small granularity and do it on every chip 2 billion times a second.
[57:07] A lot of people said this was impossible. We have people quit. Yeah, that people literally
[57:12] were like this problem is unsolvable and that's the luck guys. Well, when you have a problem
[57:17] like that, step one is okay. Let's assume the problem is solvable. How would it be solved? Well,
[57:22] first we realized what we have to be able to do is find a way to go ahead and move our clock
[57:27] phase by a tico second, 10 tico seconds. And we had an idea, what if we had these two clocks?
[57:34] I had to just a little bit of part for each other. We figured out that, hey, if you go out and
[57:39] figure out the phase, if we then go out and use a drifting mechanism, to go ahead and wait for
[57:44] just the right amount of time to get that always lined up, we can do this extremely reliably,
[57:49] and then lock the phases exactly where they have to be, we can guarantee this never happens.
[57:54] people were I think blown away that this worked and it worked as well as it actually did.
[58:00] But we made it work.
[58:01] How long did that take?
[58:02] That was actually about two weeks.
[58:04] It was dark three weeks.
[58:05] It was a very scary two weeks.
[58:07] But it was a kind of thing where when that kind of thing happens, that is the most important time
[58:12] to go ahead and be investing effort.
[58:14] That is the hardest time to go do it when you feel like things are hopeless.
[58:17] But like the sooner you solve that problem, you can get back to building and scaling up production
[58:21] to mass volumes.
[58:22] I think a lot of our story is like as Gavin says, assuming it is possible.
[58:27] Like, assuming it is possible to have a chip with way more flop something.
[58:31] Assuming it is possible to have a system with way lower latency between chips.
[58:34] Assuming it is possible to create a shared memory pool that can run away higher bandwidth.
[58:38] Like, how would one do it?
[58:39] A lot of the time when we do experiments, we will do dozens of experiments and all of them will fail.
[58:44] But we only need one to work.
[58:46] There was more time.
[58:47] I mean, Gavin, I think was leading the charge and our chip bring up with,
[58:50] I think, 30 different board experiments.
[58:52] and three of them worked and all three of them were worth their weight and golf.
[58:56] This is one of the things that people going to be like Gavin, almost no nearest
[58:59] very much works.
[59:00] Ali got to get lucky once.
[59:01] So one idea for one of these stories that I'm asking about, you know, difficult moments
[59:04] in the company's history is around the ability to raise capital to fund the thing.
[59:09] I think when you started it, you need capital, but you did not know you need the quantum
[59:14] of capital that you've ultimately raised and are spending to build the solution.
[59:18] And you hadn't raised money before.
[59:20] These are all new things, right?
[59:22] There was moments where it was really, really difficult because I was there. I saw it.
[59:25] Moments were as extremely difficult to raise the money that you did, that without which the company
[59:30] would not exist, it would have died. And like many great stories, there are many near-death moments.
[59:34] But money specifically in this new world, this isn't software. You don't just need a little bit of money.
[59:39] Maybe you could tell the story about the true hardest part about raising money early on.
[59:44] Before you had something that you could show people and be so proud of and performance,
[59:48] and so you could show them other socks off.
[59:50] It was just you guys talking about an idea,
[59:52] but talking about the early difficulties
[59:54] raising money because it was pretty hardcore.
[59:57] We've had some intense moments.
[59:58] It reminds me, uh...
[59:59] of probably early 2024, before we raised our series day.
[60:03] And we were at this point where we had done enough
[60:07] of the architecture, done enough of the design
[60:09] that we knew that the chip architecture was sound.
[60:12] We had to go build it, there was a lot more to do.
[60:14] We were ready to go into what's called the physical design stage.
[60:16] We needed to sign an agreement with the physical design vendor
[60:19] which it will cost you at least $40,000,000.
[60:22] And then we had this realization as the models
[60:24] were getting bigger and bigger.
[60:26] And you're seeing these giant MLE models come out
[60:29] that we were going to need to build the entire cluster, not just the chip, but we were going to need to build
[60:34] boards.
[60:35] We're going to need to build interconnection, build cold plates.
[60:37] So we're going to need to figure out all of the networking and everything.
[60:39] And if this was going to cost a lot more than the $15 million we had in the bank.
[60:42] And like, man, that was scary.
[60:44] Yeah.
[60:45] That would like, you're sitting in that moment.
[60:47] They think, holy crap.
[60:49] We can't afford this.
[60:51] And I'd be looking at like, huh, hard as it to go back to Harvard.
[60:56] At the end of 23, we put together this memo.
[61:00] We spent 100 hours on this because we have no idea how people are going to believe us.
[61:05] We asked for the amount of money we were about to ask for.
[61:08] It was 30 pages.
[61:09] It was extremely technical and in depth of all the different things we needed to build.
[61:13] All the milestones we needed to hit and how the market was going to evolve and all the new
[61:16] use cases in the cost-protokin and all this modeling.
[61:19] We went and talked to investors.
[61:22] Every major investor in the Valley past, immediately.
[61:25] They were just like, okay, two kids that just finished Harvard,
[61:29] haven't taped out a chip, no test chip, inference,
[61:32] who knows if this is gonna be a big market,
[61:34] everything's gonna be training,
[61:35] the model's still hallucinating,
[61:37] this could all be a bubble.
[61:38] At the time, the biggest semiconductor fundraiser
[61:41] for a series, I was around like $40,000,000.
[61:44] We were looking at this,
[61:44] and we were just like tallying the bill.
[61:46] I'm like, we're thinking we're gonna spend $100,000
[61:48] in the next 12 months.
[61:50] We really wanna do this.
[61:51] If we want to actually get to scale
[61:53] and actually get the performance we're talking about,
[61:55] This is going to be extremely capital intensive.
[61:57] How the hell are we going to pull this off?
[61:59] I think that one of our key ways we've got started was process.
[62:02] We thought to ourselves, where does the cheapest possible way we do this?
[62:07] Decided well, it may not be something.
[62:10] And if I ate nothing but run, then we would go ahead and spend basically just the money for the mass
[62:15] that we'll have for our trader tape out and I'll be that right.
[62:17] And if so, we'd go on doing it on $30 million.
[62:21] I'm insanely low-num.
[62:23] And that we actually went out and got a deprifier.
[62:26] They wanted to go ahead and lend us the money we need to go cross this trip threshold.
[62:30] There.
[62:31] I think it was first to go catalyze a series of other hate.
[62:34] Maybe we can go do one more thing.
[62:36] One more thing.
[62:37] Yeah, so we're at this moment where we're like,
[62:40] if we really want to build this company,
[62:42] we're not going to have to pass it.
[62:43] We're not going to go do a test trip and spend years on it.
[62:46] And let the entire AI market boom while we could be building the product.
[62:49] the product. Like, we're going to do it. We're going to go all the way. We're going to need to
[62:52] find a way to get $100 million. I remember Gavin and I were sitting down in the office in
[62:57] Cure, Pratino, late at night. Just like looking at each other, we're like, could we cut 500k here?
[63:03] Could we cut 100k here? How long could we convince everyone not to take a salary? And we're like,
[63:09] holy fuck. The math is not going to close. We really need to solve this. And there was a period of a
[63:14] a few weeks where you kind of just go into survival mode.
[63:18] And you call every person that could possibly
[63:20] know and invest, and you're like,
[63:22] we need $100 million to do this.
[63:24] If we do this, we think this could be one
[63:25] of the most important companies of all time.
[63:27] Do you know somebody that wants to take an aggressive bet
[63:30] that wants to believe in us?
[63:32] Like, here's all the information.
[63:33] Like, we're an open book.
[63:34] Here's the team.
[63:35] It's great people.
[63:36] We've been working super hard.
[63:37] We've done these things in record time.
[63:38] But we have these, you know, a hundred things to go.
[63:40] Like, do you want to do this?
[63:42] And like, the snowball starts and you get a million here
[63:45] and two million here and you're like, okay,
[63:48] we're not gonna run out of money this month.
[63:50] You get a $5 million dollar check, $10 million dollar checking,
[63:52] like, okay, maybe I can buy those FVGA's.
[63:54] And the snowball happened where we were very lucky
[63:57] that we ended up putting it all together.
[63:59] We had a board meeting.
[64:00] I showed you the spreadsheet.
[64:02] And we look at it and it's like 133 million.
[64:05] And it's like, these are all like soft commits.
[64:07] And well, look at each other and we say,
[64:09] we're gonna take it.
[64:10] And that was the series A, and it's been much easier since then.
[64:14] And we've raised almost a half dozen rounds since many of them
[64:16] from those investors just doubling and tripling down.
[64:19] That's allowed us to get to market so quickly.
[64:21] Like this rack would not be possible.
[64:23] How do we not have been so aggressive?
[64:25] That's the thing, like suppliers too.
[64:27] I think there's a little bit of a combination here.
[64:29] Yeah.
[64:30] TSMC, with willing to work with us back before we raised any of that $100 million.
[64:35] Back when it was still really, really scary.
[64:37] It's an office.
[64:38] Actually, when it hadn't let us get some of their emulators,
[64:41] are extremely favorable terms,
[64:42] we pay over many years,
[64:44] basically a big loan.
[64:46] It takes a lot of belief from your partners to do this.
[64:50] But I didn't know that you come out with this very strong team,
[64:53] and all the folks who back you are not in it,
[64:55] just out of pure financial sense, they believe.
[64:58] Why didn't you assume see the way you think?
[65:00] This is a great story, even for you and Joy Den.
[65:03] They were the conference.
[65:04] A Semi event.
[65:05] And I was one of the only young CEOs
[65:08] I think that's kind of a novelty.
[65:10] Ask me to come in there and speak.
[65:12] I get to the sendy event.
[65:14] And I am the only speaker there under 40 and only person there under 30.
[65:19] Out of the time of 22, I go up and I speak.
[65:21] There was a speaker's dinner afterwards.
[65:23] And by pure luck, happy syneptis, very senior TSMCVP.
[65:27] Isn't very nice dinner.
[65:29] There's like the former CEO of Arm there.
[65:31] It's very Boozy everyone's in the suit.
[65:33] And I'm there with this VP and it turns out we both do math and college.
[65:37] We don't get a little piece of effort.
[65:39] We begin talking in great detail about how to do modern AI models work at the actual
[65:44] per tensor by tensor level.
[65:46] And the guy gets it.
[65:47] We begin talking about, hey, how do you run this very effectively?
[65:51] Why is global concept of critical technology to make this work?
[65:54] And the following day, I get an email from T.S.
[65:56] I'm saying Gavin, I want to work with that.
[66:00] Trying to make it happen.
[66:01] I've been a great partner ever since.
[66:03] It's amazing to think about some of the tropes.
[66:05] And obviously, like, she break the fourth wall here, like,
[66:08] I'm a big-action investor.
[66:09] I've been involved for a long time.
[66:10] I think that's a world of you guys.
[66:12] So I'm incredibly biased in this conversation.
[66:15] I'm trying to ask questions that are broader and interesting
[66:17] and could be objections to what you're doing
[66:19] and we'll keep doing that.
[66:21] But it's so interesting to me that when you read about investing,
[66:24] everyone sites this idea of like,
[66:26] contrarian and right as the quadrant that makes all the money,
[66:29] and it sounds really nice.
[66:31] But contrarian means like,
[66:34] everyone else thinks you're stupid.
[66:35] And so when you go and you get the media it knows
[66:38] from literally everybody,
[66:40] it is a fascinating quadrant to exist in before you
[66:43] become consensus.
[66:44] What was it like for you?
[66:45] I'm super curious.
[66:46] Well, it's interesting at the time,
[66:48] it was the largest by a lot.
[66:50] First check that I had written,
[66:51] if I used to say I'm not a math expert
[66:53] or a semiconductor expert or an AI expert
[66:55] really at the time.
[66:56] And so it was much more of a believed in the concept
[66:59] of this market potentially being huge.
[67:02] you having made very, very clear bets on how the future was going to look, having position
[67:06] the company in order to attack those things in a hardcore way.
[67:11] And then just the two of you and what I felt about you was the majority of the reason why we
[67:14] made the bet when we did in 2023 or whatever it was.
[67:17] But at the time it was the biggest.
[67:19] And I think the same thing you said about Naiveta applies to investing as does to maybe building
[67:25] a semiconductor startup, which is like, I didn't know what I didn't know.
[67:29] And when I called experts, they were basically like this is stupid.
[67:33] They laid out in very logical terms like why this wasn't going to work and why it was such
[67:37] a low probability bet.
[67:39] And I think one of the things I've learned from it, it's just like you kind of have to
[67:41] damn the base rate.
[67:42] Like if you invested on base rates, you should do something other than what we and I do.
[67:47] It's always the index fund.
[67:48] Yeah, there's always something index fund exactly.
[67:50] So it's actually never been scary for me.
[67:53] I think probably most of that is because there's a lot I don't know.
[67:56] And if I knew more about what you guys have done in the difficulty, I probably wouldn't have done
[68:00] it.
[68:01] I don't know what that says about like maturing as an investor like maybe I don't want
[68:04] to know a lot more and have some of that healthy naivete, I don't know.
[68:08] Funny.
[68:09] A lot of the traditional semiconductor funds missed the entire AI chip like all the AI chip companies
[68:15] and like all the coding experts missed all the coding companies.
[68:18] I think it's very hard to realize the constraints have changed and when you've looked at
[68:22] that tape outs for 20 years and you've seen so many of them not work on the first try,
[68:27] the second try, the third try.
[68:28] Like you couldn't even run a workload.
[68:30] You totally forget that EDA tools are way better and that FPGA is existed in a way that
[68:35] they didn't exist before.
[68:36] And all the types of validation you can do today just wasn't possible for us.
[68:40] A lot of our believers either, they were kind of on two sides of it.
[68:43] They were just believers in the market in the team or they were building chips today and
[68:48] extremely technical like the high 50 trading firms where they would literally audit every
[68:52] everything from the micro architecture and the RTL to like the board designs and like the
[68:57] schedule and the software stack and we would sit down with like 10 of their people who build
[69:01] their own chips and they're asking us such detailed questions that were wondering are they
[69:04] going to build the chip? It was really on either of the sides and if you were anywhere in
[69:07] the middle, you just wouldn't understand it.
[69:09] In the investing world, they often talk about a very interperception, something that you see
[69:13] or believe that others don't, right? And that perception creates the opportunity. I think I'm
[69:17] invested five or so times and thatched.
[69:20] And every time when you do it,
[69:22] thanks are getting bigger and bigger.
[69:24] It does get a little scarier and scarier.
[69:26] And because you guys have been so quiet
[69:28] in the marketplace, I think it's very easy to dismiss you.
[69:32] As the stakes get bigger and bigger,
[69:34] those dismissals are harder to hear.
[69:37] I do think betting on something that you see
[69:40] when what you hear from the outside world
[69:43] is very different.
[69:44] That perception gap equals opportunity.
[69:47] Exactly. The last thing I would say is the accumulated evidence of your guys' and your
[69:52] teams' ability to solve seemingly impossible problems is one of the most interesting things
[69:58] a company can have.
[70:00] Anyway, like companies do this or they don't.
[70:01] That's the thing is a big advantage of people who've been here for a long time.
[70:05] You get some new joiners who are scared of shitness.
[70:08] You see a thing like this and there are old timers that are here for all of two years.
[70:12] This is the guy I see a trench of.
[70:14] Another one.
[70:16] Yeah, there's definitely a find-away mentality.
[70:19] If you're here, you're here because you're assume it's possible.
[70:22] So we can't be saying it's impossible.
[70:24] Everything is solvable and we're just going to work out until we figure it out.
[70:28] A favorite story I have, this guy who's kind of a legend in Silicon Validation who joined our team.
[70:34] And we were doing the early stages of what's called WayForSort.
[70:36] When your chips are coming out of fabric, I got a lot of these WayForers.
[70:39] You have this thing called a probe card that attaches to the WayFor before you dice it with these
[70:43] probe pads, and you send these electrical signals to basically test, which chips are good and bad.
[70:47] So when I slice the way for into a bunch of chips, I can package it and only package the good ones.
[70:52] We go through our first WayFor. It's like two, three, because we're doing it with TSMC over the phone and
[70:57] We have the screen with the way for its all gray and each ship is gray.
[71:01] And then as you start running the patterns, the squares are supposed to turn green or red.
[71:05] And the alternate red.
[71:07] I'm like fuck.
[71:09] It's like, this is really bad.
[71:13] Everybody's like, guys take a breath.
[71:15] You eat the back of your eyes like the puzzle begins.
[71:17] You have to have the attitude of like, yes, you will go,
[71:21] and you will go stare into the abyss.
[71:23] You will go see scary things.
[71:25] And we'll solve them.
[71:27] When did you see the first green square?
[71:29] I don't know what did it take that.
[71:31] But in the moment, it's extremely scary.
[71:34] And there's a certain type of person who just is addicted to that feeling.
[71:37] I'm just feeling the fear and solving it and we have a lot of people here.
[71:41] If you think about applying all of this earned know how from this last several years
[71:46] and now thinking ahead to Gen 2, Gen 3 and beyond,
[71:49] what will you be doing most differently as a result of everything that you've learned?
[71:54] just like in conceptual standpoint, like the way that you will attack, designing and producing
[71:59] this next one, based on what you learned during the first time.
[72:02] It took us a while to get to the preventives that we think are really what matters for
[72:06] scaling inference.
[72:07] We try to bunch of things early on from compilers that would turn different models into
[72:12] FPGA's to burning weights and silicon to splitting your HPM to KV-Cash and weights and
[72:17] all of these different things.
[72:19] And there's a lot of cycles of learning until we got to the point that we realized that,
[72:22] like fundamentally, if you want to run a majority of the tokens in the world, you need to do three things.
[72:27] You need to build a chip with the most flops and give in power budget.
[72:30] You need to build a chip that has the lowest latency between other chips, so the biggest scale up to me impossible.
[72:35] And you need to produce as much of it as possible.
[72:37] And I think probably in the first half of our journey so far, we learn the first two in that informed the design a lot,
[72:43] and that informs a lot about the bets we're making in the future, with the low voltage in front of the cluster scale memory.
[72:48] But the production part, I think, in the past year
[72:50] has become extremely obvious.
[72:53] How much people want to deploy this stuff
[72:55] if you can have it available today?
[72:57] The best ability is availability.
[72:59] If I have a thousand chips today,
[73:01] someone's gonna use them.
[73:03] And we need to build a chip that's not just way better
[73:06] than what's been built before,
[73:07] but it needs to be available at many gigawatts scale.
[73:10] We need to be building a product that is
[73:12] produceable at gigawatts per month in the limit.
[73:14] As we think about that, a lot of the design decisions
[73:17] making with our next gen, which you've seen already, is just about simplicity, removing tons
[73:21] of parts, trying to assemble and disassemble a thing again and again and then learning
[73:25] how to make it as quick and the cycle time as possible in production, making sure it's
[73:29] going to be reliable, making sure it's going to be serviceable, and making sure it's going
[73:32] to be produceable at gigantic skills.
[73:34] What about other problems in the ecosystem that are outside of your control, such as capacity
[73:39] at the leading nanometer at TSMC or availability of HBM4 memory, or some of these other things
[73:46] things were like everyone is fighting for scarce unit of capacity or whatever.
[73:52] How do you face up against those realities when you're trying to produce as much as humanly
[73:56] possible?
[73:57] If people deploying the most compute in the world, do you think about supply a bit?
[74:02] Which is, there's only so many way for being produced on a given nanometer node, on a given
[74:06] fab.
[74:07] And there's only so much memory being produced.
[74:09] And that's why actually for our first gen product, we built it on a different supply chain
[74:14] than the ribbons.
[74:15] But we're on 4 nanometer, we're on a different HPM than we're in so forth.
[74:20] So it actually is not a zero something.
[74:22] It's a positive something where more is more.
[74:24] So often when we're talking to people deploying at scale,
[74:27] it's not a decision between a gigawatts of a GPU and a gigawatts of us.
[74:31] It's two gigawatts.
[74:33] And I think it's much as possible thinking about
[74:35] supply chain early in the design decisions,
[74:37] because if you have them as performer product,
[74:39] and you can't produce it, then you're just a podcast.
[74:41] That's the other big thing about Rd.
[74:42] of integration too, is, or certain things like for the chips of the memory, you have to go
[74:47] head and partner. For most of the other stuff, those are also very highly in the manipulance,
[74:52] and the more that you build yourself, the more stuff you can do on top of what the world
[74:56] can currently build. It is not all you're taking on the build, it's not all you're taking on the build,
[74:59] it's not all you're taking on the build, it's not all you're taking on the build, it's just like you're
[75:00] adding way, way more. I think that's how you win. One of the things that really we haven't talked
[75:04] at all about is the models themselves, which is kind of crazy, the things behind all of this demand,
[75:10] Anything interesting that you would say about the way that you see models progressing,
[75:15] based on what we've seen so far, I guess I'm more interested in how US thinkers about hardware
[75:20] think hardware might impact where the models themselves go in the future.
[75:24] One of the most important ideas that we believe in is that machines don't think like people think.
[75:30] You look airplanes, for example. Airplane is slightly birds fly.
[75:34] That when you think about how mechanical devices have to work, it's often very different.
[75:39] And a much the same way for people storing data in loading memory is very cheap for
[75:45] neurons and doing math is relatively expensive.
[75:49] And it is the exact opposite for chips.
[75:52] Generally, one data is very expensive and doing math is a very cheap.
[75:57] And as time goes on, then you'll end up finding that math gets cheaper at a rate that is
[76:02] faster than a memory gift cheaper.
[76:04] Just do this fundamental limit on any kind of DRAM device.
[76:08] You should think about how can I make my model use a huge, huge amount of compute.
[76:14] What if I had, for example, many copies are running at the same time?
[76:16] What if I activated a huge number of experts?
[76:19] What if I had gigantic experts that could run on multiple server after the same time?
[76:25] That is how you'll build models that are the next generation of intelligence.
[76:29] And in context too, there's been a lot of work on a very efficient inference.
[76:33] What if I don't load the full context in the memory?
[76:36] And most of the time, I think they make a lot of sense.
[76:38] The only thing that built the super-attelligence.
[76:41] Why can't it go look at a billion tokens of context?
[76:44] Why can't it spend a huge amount of compute to go ahead and read all of the super fast?
[76:48] I would love to go to the talk to a machine that was able to go attend to every book ever written
[76:54] and a short-term memory.
[76:55] And I'd like to have to go into a point where you can.
[76:57] A theme in models right now is this focus on something called dynamism,
[77:01] dynamism, which is the ability to control the level of computation and memory spent at a
[77:07] per-token or per-user level when doing attention, as well as the ability to dynamically
[77:13] in-your-chip on the fly, send data to other chips for different Emily models doing certain
[77:17] types of operations. And the reason is fundamentally, as we are scaling context-length,
[77:23] as we're scaling model size, as we're scaling the amount of computation per-user,
[77:26] we're looking for ways to be more efficient.
[77:29] So the first thing is like Gavin says,
[77:31] you know, make sure of experts, you know,
[77:32] architecture is where, you know,
[77:33] maybe we don't need every parameter being used
[77:36] for every token.
[77:37] But maybe there are things where even at a token level,
[77:39] we can say, well, this token needs this context
[77:42] on this other token, they can share that memory.
[77:43] So we don't have to have overhead of using an memory as much.
[77:46] Maybe this token is really important.
[77:48] So we should spend more compute.
[77:50] We should have longer context on that token.
[77:52] So hardware that really accelerates these types
[77:54] of very dynamic computations extremely important.
[77:57] And you can imagine current hardware that was designed
[78:00] before the sites of architectures have lots of overheads
[78:03] in doing them.
[78:04] So you basically end up in these really bad worlds
[78:06] where you have inefficient hardware at doing this dynamism.
[78:09] So therefore you can't run it very well.
[78:11] Or you have these very blocky architectures
[78:13] that are kind of applying blunt force
[78:15] to many different tokens that all need more or less
[78:18] to a computation.
[78:19] I have two questions about the future.
[78:21] We talked a lot about what you built so far
[78:23] how you built it. The first is about the new ways that people might start using these systems.
[78:28] The raw technology, logger, run times, things in this nature. When inference gets much cheaper,
[78:34] faster, more accessible, there's more toll supply and it's better. What are the things that you think
[78:39] people will use that capacity to do, that are the most interesting exciting to both of you?
[78:44] Yeah, that was the viral tweet by No. 1 Brown. We said the athletes and my levels are having a longer
[78:48] longer time for ISIS, they can need to ask the takes say six months and there's often not enough
[78:54] time to go into the I-O-A, for such a long period of time because by that point you'll have
[79:00] a new mile at, you'll want to go to the I-O-A instead and with tech like we built our cluster scale
[79:05] memory you can go ahead and run that sick month job much faster. But there's a second piece of this
[79:10] too, I'm talking to them about it, it's now an angel as well, where it's not just the time, it's also
[79:15] Also, the number of people or agents who are working on this.
[79:19] If you're trying to go and evaluate, can't a human build a rocket?
[79:22] You will find that the answer is no.
[79:25] No one first can build a rocket.
[79:27] Instead, you have to put a team together.
[79:29] And I believe the same thing will be true of age as two.
[79:31] If you want to go ask, can it a-z it go out and build some crazy futuristic piece of software?
[79:37] You'll probably need a very large team.
[79:40] Maybe that's 10.
[79:41] Maybe that's a million.
[79:42] It has to go and have this enormous amount of both claustrophile memory to go ahead and have
[79:47] that very short time per token and a huge amount of flops to be able to go run that
[79:52] whole fleet.
[79:53] I'm going to be a little futuristic.
[79:55] I firmly believe we are on a global march of inference becoming.
[80:00] majority of global GDP. You may think more than 10 years, but it's going to happen. And right now,
[80:04] we measure productivity as a society as GDP for capital. But really, it's going to look much
[80:10] more like agents for megawatt or it may be agents for gigawatt by then. And while being futuristic,
[80:15] I think this is the second to last year or a majority of the workforce is going to be human.
[80:21] I think in 2027 you're going to see there's going to be more agents doing knowledge work in humans.
[80:26] and it's going to be extremely interesting to see what happens.
[80:29] You could imagine a world where for countries a majority of their energy ends up going into data centers,
[80:34] doing inference, and the energy efficiency of those data centers basically governs
[80:39] how many agents in there for how big their work forces.
[80:42] So you're going to see, like as Gavin is saying, one agent or a team of five to 10 agents working
[80:47] on group projects for a couple of days. So you can do pretty cool stuff because they're smart,
[80:52] But it's not going to be civilization scale.
[80:54] What happens when you have countries that can have
[80:57] literally a billion concurrent agents,
[80:59] like a billion people in the workforce,
[81:01] working 24-7 concurrently, on the same stuff?
[81:05] It's just kind of unfathomable what's going to happen.
[81:07] It's going to be the biggest proliferation of technology
[81:10] humanities ever seen.
[81:11] I think it's well, like when you have these huge huge amounts
[81:14] of demand, you get this idea of economies of scale again.
[81:17] Yep. We're thinking about people.
[81:19] I have a brain.
[81:20] I'm not using the whole thing all at the same time.
[81:22] That's only part of us going to be active, and this is the way healthy brain works.
[81:25] And the way models have worked much the same way.
[81:27] On an end-of-way model, only a small fraction of the parameters is being used for any given token
[81:32] at any given moment.
[81:33] But do you have a large number of users?
[81:36] On a piece of hardware, you can go kind of take that brain, cut it up in a many different
[81:40] experts and many different servers, and run a huge amount of volume through it.
[81:44] So you'll have a bunch of different pieces of the traffic.
[81:46] You'll have many of them using each part of the brain at a given point in time and you'll
[81:50] have to make the cost per thought, cost per token away, way lower.
[81:53] So you're going to end up with these giant scale distributed brain as the form factor.
[81:58] This is a big data center with a bunch of chips that use amount of flops and a huge amount
[82:02] of scale up in our connect.
[82:03] You think we'll see a trillion dollar individual data center?
[82:07] Absolutely.
[82:08] It is a matter of time.
[82:09] It's like asking, will you see a billion dollar fab or a ten billion dollar fab or a hundred
[82:14] billion dollar fab?
[82:15] is inevitable that the economy's a scale don't stop at a 40 billion dollars of the
[82:20] netting number for FADs.
[82:22] Note, the cost way for keeps going down as you keep spending more money.
[82:26] And the same thing will be true of plants that go out and make steel or plants that go out
[82:30] and make tokens.
[82:31] A very smart alien lands on Earth and wants to know from each of you how you would frame up
[82:37] this opportunity that you guys are tackling.
[82:40] What do you say to them?
[82:41] it as thinking is really valid that every company in the world runs on thinking.
[82:46] And we are entering this really unique moment in time where you have a machine that
[82:51] can go think almost as good and that soon asks good and that soon better than the best
[82:56] humans can.
[82:57] The little thing of these machines is going to go to be a huge opportunity.
[82:59] But more important than that, the way in which you go ahead and run is kind of thinking is
[83:04] going to be very, very different as demand goes higher and higher and higher and higher.
[83:08] a unique moment of right now to build a new set of solutions, a new roadmap.
[83:13] For how do you run a future quadrillion parameter models for a billion people all the same
[83:18] time on a gigantic scale up cluster?
[83:21] We are in a new era of intelligence where the cost of producing intelligence is dramatically
[83:28] so much cheaper than the value of the intelligence that we are in a many year probably many
[83:34] decade supply shortage of these tokens. And basically any chip or any system that can produce tokens
[83:42] is likely to be extremely valuable. And you should find some part of the supply chain of the token.
[83:48] I can be everything from model training down to what we're doing in the silicon, in other ways,
[83:52] to spend time on and push the frontier. And that the companies that are the largest are frankly
[83:57] going to be the companies that produce most of the global supply of tokens and own majority of the
[84:03] supply chain of that token. And importantly, it's people who build systems that as they get more and more
[84:09] just put together, get cheaper. The way you want this to scale is not that if I want to go
[84:14] serve 10 times 4 tokens, I buy 10 times more servers. And must be some solution where
[84:19] if I only have 10 times more tokens, then I get some economies of scale benefit with my
[84:24] set of Clusterscale memory tech. That allows me to then not charge as much as 10 times more for those
[84:30] Dr. Token, what are ridiculously exciting future that you guys are building to enable?
[84:35] When I do this with Gatom last time I asked him my traditional closing questions so this
[84:38] time I'll ask you, what is the kindest thing that anyone's ever done for you?
[84:41] During my cancer treatment there was a big decision I had to make.
[84:45] The doctors came to me and said, it's time for you to decide, do you want to get surgery or
[84:49] do you want to get radiation? Here's the trade-off. If you get surgery, you're more likely to live,
[84:54] but you have to assume you'll never be able to walk again. If you get radiation,
[84:58] You'll be able to walk again, but it's not the same probability that you'll live.
[85:02] You may die. What do you want to do? And I was 16 and my parents said you have to make this decision for yourself.
[85:09] I thought a lot about it for a long time and decided I'm going to do the surgery.
[85:12] I get the surgery. One of the things you do when you get a tumor resection is they do something
[85:17] called a necrosis analysis. Or they look at all the different cells and say is a cell data or a
[85:22] live because if you have a bunch of cancer cells that alive, you have a problem and they
[85:26] looked at it and they said, you know, you usually want 98, 99% necrosis for us to say,
[85:31] you're in the clear. You're below that. You should go get radiation. And there was only a few
[85:36] machines in the world that actually could do the type of radiation I needed. I wanted them
[85:41] to be in Boston. I was in a wheelchair and I needed to move the Boston for multiple months.
[85:45] And both of my parents have decided to move out and drop everything they were doing and
[85:49] and live with me.
[85:50] And I'm eternally grateful.
[85:51] Who's beautiful?
[85:52] Nice, guys.
[86:19] You know how small advantages compound over time that's true in investing and just as true
[86:38] in how you run your company.
[86:39] Your spending system is your capital allocation strategy.
[86:42] Ram makes it smarter by default that our data better decisions better economics over time.
[86:46] see how at ramp.com slash invest.
[86:49] As your business grows, Vantis scales with you.
[86:51] Automating compliance and giving you a single source of truth,
[86:54] for security and risk.
[86:55] Learn more at vant.com slash invest.
[86:58] The best AI and software companies from OpenAI
[87:00] to cursor to forplexity,
[87:02] use WorkOS to become enterprise ready overnight, not in months.
[87:05] Visit workos.com to skip the unglomerous infrastructure work
[87:08] and focus on your product.
[87:10] Ridgeline is redefining asset management technology
[87:12] as a true partner, not just the software vendor.
[87:15] They've helped firms five X and scale enabling faster growth, smarter operations, and a competitive
[87:19] edge.
[87:20] Visit regelineapps.com to see what they can unlock for your firm.
[87:24] Every investment firm is unique and generic AI doesn't understand your process.
[87:28] Rogo does its an AI platform built specifically for Wall Street connected to your data,
[87:32] understanding your process and producing real outputs.
[87:35] Check them out at rogo.au slashinvest.