localfirst.fm

A podcast about local-first software development

Listen

Conference

#25 – Tanner Linsley: TanStack DB

The guest of this episode is Tanner Linsley, creator of the TanStack ecosystem including projects such as React Query and TanStack Router. This episode will talk about the newest project, TanStack DB and explore the problems it’s trying to solve and how it works.

Mentioned in podcast:

Links:

Thank you to Jazz for supporting the podcast.

Transcript

#25 – Tanner Linsley: TanStack DB
00:00My perspective on localf-first is that, I think ideally every developer would
00:06love to be able to interact with their APIs as if they were localf-first.
00:11and then I say that not necessarily meaning like offline, but just , I
00:15synchronously did this thing and I can trust that it's going to
00:17be persisted where it needs to be persisted and then move on.
00:22and just kind of feel like everything gets instantaneous.
00:24So, I have no doubts that that is what everybody wants.
00:28It can be difficult to get there.
00:30and if you try and get there too quickly, I know you can get into some
00:34trouble and there's really smart people who are slowly trying to carve out
00:38this space and make that a reality.
00:40Welcome to the localfirst.fm podcast.
00:42I'm your host, Johannes Shickling, and I'm a web developer, a startup founder, and
00:46love the craft of software engineering.
00:49For the past few years, I've been on a journey to build a modern, high quality
00:52music app using web technologies, and in doing so, I've been falling down the
00:56rabbit hole of local first software.
00:59This podcast is your invitation to join me on that journey.
01:03In this episode, I'm speaking to Tanner Linsley, creator of the TanStack
01:07Ecosystem, including projects such as React Query and TanStack Router.
01:11In this episode, we talk about the newest Project TanStack DB and explore
01:16the problems it's trying to solve and how it works before getting started.
01:20Also, a big thank you to Jazz for supporting this podcast, and
01:24now my interview with Tanner.
01:26Hey Tanner, it is so awesome to have you on the podcast.
01:29How are you doing?
01:30I'm doing great.
01:31Thanks for having me.
01:32I'm very excited to have you.
01:34so most of the audience I'm certain are familiar with your work, are
01:38probably using a lot of your tools, but would you mind introducing yourself?
01:43Sure.
01:44my name's Tanner Linsley and I've been building open source software
01:49for, I don't know, almost 10 years.
01:52been involved in some startups, one of them for about a decade where
01:56we were building a, a SaaS product for SEO Marketing Analytics, which
02:02was pretty intense, a lot of fun.
02:05And, yeah, now I mostly work full-time on TanStack, which is kind of the
02:10umbrella of all the tools that that I've built or have helped build in
02:16the last, eight to 10 years, so.
02:19That's, that's what I do.
02:21That is awesome.
02:22I think it's safe to say that everyone who's listening has at the very least
02:26used a product that is using one of your tools, but I'm also pretty certain that
02:30almost everyone is also directly using one of your tools and libraries in some
02:36package days, dependencies, me included.
02:39I'm a huge fan of your work and you and I actually had the chance to catch
02:44up in person again at last year's next JSConf in, very sunny San Francisco.
02:49And this is where I got the chance to show you what I've been working on
02:53over the last few years with Live Store and generally chat about localf-first.
02:58And I think this was probably not your first touch point
03:01with, localf-first as such.
03:03I think that goes way back further.
03:05But maybe you can share a little bit of like your relationship to localf-first.
03:11As I understand it, you are not yet fully in the center, you're aware
03:17of it, but you also are developing your own perspective on it.
03:20So maybe you wanna share that.
03:22When we met at Nex Conf and what you showed me was impressive.
03:26It was crazy.
03:28and the stuff you had been showing me before and after
03:30that is just mind boggling.
03:32I think the furthest back I can go for like a localf-first, or at least what's
03:36aspiring to be localf-first experience was like back with Meteor and PouchDB.
03:42I remember playing with those extensively just thinking like, wow, this is so cool.
03:46'cause, and back then it was like, there was a, a big push to have
03:50everything be offline, you know, first, and then, sync when it can.
03:55obviously, the syntax and the tools, the tooling and everything around local-first
04:01like evolved drastically since then.
04:03it doesn't even look the same.
04:05but that was kind of my first experience with the feeling of localf-first
04:10that you get where you kind of just, you can interact with your data.
04:14Pretty much as if it's synchronous and then just kind of forget about it.
04:18And that, I think PouchDB was probably the first one that I felt that with and
04:22I was like, wow, this is really magical.
04:25you know, it's definitely far from perfect and there's a reason that, you know, not
04:29everybody's using PouchDB anymore, or even were using PouchDB, yeah, I think, in
04:35the more modern, tool chains, everything you've been working on has been kind
04:41of the most eye-opening along with some of the other stuff from like, Convex.
04:46Convex is a TanStack partner, and they.
04:49Kind of like that server-first, localf-first experience as well,
04:55and they have really direct ties with TanStack Query, which is why
05:00I've experienced a lot of that.
05:02so yeah, my perspective on localf-first is that, I think ideally every developer
05:09would love to be able to interact with their APIs as if they were localf-first.
05:14and then I say that not necessarily meaning like offline, but just in a
05:18very synchronous, I synchronously did this thing and I can trust that it's
05:22going to be persisted where it needs to be persisted and then move on.
05:27and just kind of feel like everything gets instantaneous.
05:29So, I have no doubts that that is what everybody wants.
05:34But obviously, as I'm sure we're going to discuss here over the next few minutes,
05:38is that it can be difficult to get there.
05:41and if you try and get there too quickly, I know you can get into some
05:44trouble and there's really smart people like you and many others who are, or
05:48who are slowly trying to carve out this space and make that a reality.
05:53So I am very anxiously and excitedly watching and also trying, we're trying
06:00to take part in that journey where it makes sense for TanStack to take part.
06:05That is awesome.
06:06Well, first of all, thank you so much for the kind words.
06:10I'm a huge fan of just like overall the way, also how you approach your
06:14projects, that you get to work on those projects in a sustainable way.
06:18And I think this is really why so many people trust your projects.
06:23So just wanted to mention that.
06:25And there's like many different ways how to reach and go after that
06:30goal that you've motivated, that you get instantaneous data, you get the
06:34simplicity coming from synchronous data.
06:37and I've certainly taken probably a more radical approach on that over
06:42the last couple of years where really like, I had the luxury of working on a
06:46greenfield project, so I was not burdened with everything that was there before.
06:52So I could make much more.
06:55radical and free design decisions.
06:57So in the case of LiveStore is why I've embraced event sourcing.
07:01But I think you are a bit more on the other side of the spectrum with
07:05a technology like React Query, which, at acknowledged the reality that in
07:11most projects there is already an API data source somewhere, and that
07:16makes it so much easier to adopt.
07:18And that shows in the download numbers and usage of, TanStack
07:23Query, formerly React Query.
07:25I think this is what most people really find themselves using on a daily basis,
07:31but as they aspire to make their app faster, to load more quickly, reload more
07:36quickly, then they also try maybe even make their app work in a offline scenario.
07:43Then they try to get there step by step.
07:46So maybe you can share some anecdotes of people doing that with
07:50TanStack Query where they reach for optimistic mutations, et cetera.
07:55Yeah, that happens all the time.
07:57I think that's one of the reasons why it took off, is because it's very
07:59pragmatic, for the slop that most people are working with every day.
08:04Whether that's your own slop, hopefully you've been in your job
08:08long enough to have to deal with your own slop, or it's slop from before,
08:13or you're just moving so fast that.
08:15That's just what it is.
08:16But, but the reality is that, you know, things are just messy, when
08:21you're shipping real products to real people as fast as possible.
08:24and I built React Query 'cause I needed a way to make the experience better for, for
08:31developers and users without needing to get in the middle of the entire process.
08:38like Meteor was trying to do, or like GraphQL was trying to do
08:42or Apollo, you know, I needed as little buy-in as possible with
08:46the maximum amount of, effect.
08:48And, that's what it came to be and I think it's very natural.
08:53when you start playing with Query, you start to realize that it is
08:58in a way, a sync engine, right?
09:00You have defined these declarative.
09:03subscriptions to your data, it's very manual, right?
09:08Very, course brain manual timings of like, how often do I want
09:12to go and get this thing right?
09:14So it's in the middle of like a one-off, API call and a sync engine.
09:19but it is living and breathing and, the life cycle of that life
09:26in this, Query cycle still takes some effort from users, you know?
09:30So when you change something, you need to let Query know, that you've
09:35changed something at the very least.
09:37And we call that this invalidation, right?
09:39And this is kind of like the bare minimum for What I would call
09:44probably the worst sync engine you could want is just like, I've changed
09:49something, invalidate everything and just re-sync everything, right?
09:53It's, it's very manual.
09:54You are the one deciding when to re-sync.
09:57And it turns out that a lot of developers just weren't
09:59doing that in the first place.
10:01so it's crazy to think that just teaching, giving people a good tool, like Query
10:07and then telling them, Hey, when you change something, you need to just
10:10mark everything as dirty and re-sync.
10:14It.
10:14Just, that has improved everybody's apps a ton.
10:20And user apps, you know, it's like, oh, data is up to date.
10:24You know, it's not super efficient, it's not automatic or whatever, but it is.
10:28so I think that's, the bare minimum, and that's the reality where a lot
10:31of developers are today is they're probably still coming, they may even
10:37be coming from .net or, you know, some monolithic backend shop into front
10:44end, and they've never done any front end syncing, data syncing at all.
10:49And they might just think, well, I'm just gonna call my API and display the data.
10:52Right?
10:53So I think that that's the reality for most developers.
10:57Most, successful companies are not running the latest and greatest, and many of
11:02them probably don't even know, you know, the first thing about designing a sync
11:06engine, I barely know anything about them.
11:09So I, I wouldn't expect many, you know, many of your run of the mill
11:13developers to know a lot about it either.
11:15So, but I think that's the first step.
11:18And I, I think personally that once you taste that, it's addicting.
11:22You want more of that and you naturally start to say, okay,
11:26how can I make this faster?
11:27Right?
11:27And that's on the read's end, which is kind of like the bare minimum.
11:30It's like, I wanna read my data and make sure it's up to date.
11:34And then you start thinking about the writes and you're like, oh, well you know
11:36what would be cool is if when I click this button it just showed that it's done,
11:42even though we know it takes, you know, a hot second for it to go and be done.
11:47And then to resync all the data, it'd be cool if it was
11:50just looked like it was done.
11:52And so we call that an optimistic update, right?
11:55Like optimistic, data manipulation and Query has that built in.
11:59Again, it's very manual.
12:01you have to manipulate the cache that's in your browser, in the Query client
12:07to represent that new state manually.
12:10and you fire off the mutation or, the change that's going
12:13to result in that state.
12:14And then you just kind of cross your fingers.
12:16That when you invalidate your data, when it's done and that new
12:19data comes in, it matches what you optimistically put in there.
12:23Again, this is like from my perspective, if you can implement optimistic
12:29mutations with something like Query, you're already so much further ahead
12:33than what the standard has been for a very, very, very long time.
12:38but like I said, it's very addicting.
12:41So then after that, you know, you start to think, okay, this
12:45mutation thing that I'm doing, these optimistic updates, those are tedious.
12:50You know, it's like, well, I have to, I have to make those perform, like make
12:54those optimizations myself and manipulate the cache and it feels dirty, right?
12:59And so that's when it starts to get a little more intense with like,
13:03okay, how could we make it so that when I say I clicked this button.
13:10I should like this post somewhere that I don't have to go in and manually
13:16kind of do the optimistic update that my mutation can just know what it's
13:23going to affect, make the update, send it to wherever it needs to go, and
13:29then it will sync when it needs to.
13:31And then we can just carry on, not just as like a user interface to click
13:35something and be able to carry on immediately, but also as a developer
13:39to be able to say, this is what's changing, and then just move on.
13:43You know, not have to worry about optimistic updates and cache
13:47and validations and whatnot.
13:48So this is where I think, at least for me, where things start to get really,
13:54really fun and really, really scary.
13:56Mostly because it's just, it's a new space, right?
14:01And, you know, to, give you my perspective on it, I haven't gone
14:05too far down that rabbit hole to be honest, because it involves a
14:10lot of knowledge about a system.
14:12usually involves knowing about schema or, creating schema in some way.
14:18there's different implications for where that data is being synced, whether it's
14:23local like offline or on a server or through web sockets or REST or whatever.
14:28so there's just a lot that comes with that and that has been intimidating
14:33for me because it's difficult for me to generalize where like, React
14:37Query was easy to generalize.
14:39but it doesn't mean that we're not thinking about it.
14:42You know?
14:43this was a, perfect tour from the beginnings of the motivations that led
14:48to React Query and now TanStack Query, the benefit it provides and like how
14:53far it gets you, but also like to the point where it brings you where things
14:56are already working pretty well, but maybe you face some new challenges
15:01and you would like things to work even more automatically in the right way.
15:06You now need to better understand that.
15:09And this is also like nicely reflects, like a lot of my thinking over the
15:13last decade where I've been wrestling with the same ideas and the same
15:18challenges and a few points always like bubbled up to me again as like those
15:24can't be ignored in a generalized way.
15:27And I think this is really the, key point to underscore here, like looking
15:31for a generalized solution and like at least I don't want to be the bearer of
15:35bad news, but I don't think for data there is a generalizable solution because
15:40data is like the ultimate as it depends.
15:44And I think this is where a technology like React as like
15:48the first of its kind in a way.
15:50showed us a, like close to generalizable solution that applied so perfectly
15:57to so many different situations and use cases, and unfortunately I don't
16:02think that is realistic for data.
16:04Just because for data you have just different trade-offs.
16:08Let's say you're building a banking app, this is where you really want to be
16:12certain has that wire gone out or not.
16:15But if you're building, let's say, a note taking app, it is actually more
16:19important that you can just carry on writing down your thoughts, then
16:24that you perfectly know that it's also already arrived on the server.
16:29And so it's just different trade-offs and those different trade-offs
16:33need or could be productized and systematized into something.
16:38And this again, is where I think React Query has been just so phenomenal in
16:44striking those trade-offs it meets people, where most of them are, where
16:49they don't need the perfect thing.
16:51Something good is already better than nothing.
16:54And then just like calling, fetch themselves and then,
16:58dealing with that somehow.
17:00But for people who wanna now go further for a more specialized path,
17:05I think this is where now there's like a, tree of potential new approaches
17:10and potential new technologies.
17:11And some of them you've been investigating.
17:14And I think, one particular like hairy topic here to investigate further is
17:19like, how do you get for the writes that you're doing for the optimistic
17:24updates or the updates more generally.
17:26What happens when you're updating data, like most directly, you're
17:31updating data in like a local variable somewhere, somehow running in your,
17:37let's say, React app, but then you're also maybe posting to an API endpoint,
17:43and now it's already, the universes are already potentially splitting.
17:48So you've updated your variable, maybe your API request.
17:52Goes through.
17:53And as that API request is being propagated further, fail
17:58somewhere there and now you need to unwind the entire thing.
18:01But let's say maybe you do two API requests, one goes through
18:04the other, doesn't go through.
18:06Now you're like in the split brain situation.
18:09And so now you need to, should you wind it back, should you partially wind it back?
18:14And now you need to handle that on your API server as well as on your client.
18:19And on the client, maybe you just say like, you know what, if we hit this
18:22case, it's so rare we just show an error message and say the user like
18:26reload, but maybe it's not so rare.
18:28And then you need to think about it in a more principled way.
18:32And this is where another kind of manifestation of that is cash
18:36inconsistency and There is a really great blog post, that goes
18:41along the lines of like, your app shouldn't become a database.
18:45we'll link it in the description.
18:47But, I think that is like a very nice and provocative post that shows the
18:52reality of most ambitious apps that as you pull more on that thread, your
18:57app becomes a database with all the hard things about a database, and it
19:03needs to be transactionally consistent.
19:05So if you click that button, maybe it should reflect in
19:08instead over here and over there.
19:11And if it doesn't, your app feels broken or things might actually like
19:15you get the dreaded, undefined is not a function because like some, like
19:20array index is not there, et cetera.
19:23And the more you pull on that, the more you want kind of like a
19:26React-like foundation that you can just trust that is maybe a bit more
19:31specialized to your use case, but you can actually trust and it just works.
19:36And this is where I hope things are going.
19:38Like, basically a tree of different possibilities for, mature and well
19:43designed data systems that, match the corresponding set of trade-offs.
19:49And it's now our role as educators to educate developers which trade-offs
19:55work well for their apps and what are good, solutions and systems for those.
20:01And I think that something like React Query has been an excellent catch
20:06all that already lifts people up.
20:08So I'm curious whether, that makes sense to you.
20:11I think that makes total sense and, you know, the way that I have always looked
20:16at like trying to solve these kinds of problems is: i'm always looking for like
20:21the least common denominator, right?
20:24And you can never find something that's perfect, but you can
20:27find something that's 90%.
20:29And then once you have tackled that problem, you can kind of move
20:33down the spectrum to, the use cases that might be more important or
20:38more intense, but you know, are not the least common denominator.
20:41and I think that's where React Query sits for me.
20:44You know, it's not the tool for all of the really, really specific
20:48difficult use cases, but 90% of the time it's the right choice.
20:53and then we can also, if you can build something where you can build
20:58on top of it to get to those other use cases, then that's even better.
21:01in this scenario, I personally am opening up more to the idea where
21:06there, you know, there could be this, general like solution, right?
21:12I don't know exactly how that looks.
21:13Like you said, I mean React kind of just happened and it was like,
21:16oh wow, this is a great solution for the problem components.
21:19I don't know if I've seen the component solution for like data
21:25yet, you know, 'cause there's so many different trade-offs.
21:29Like you said, I'm so interested in, see in synchronous data management
21:34through things like signals.
21:36Like I talk with Ryan Carniato all the time about, he's working on all
21:40this new Signals stuff and I just love it and I'm like, wow, this in terms
21:45of like reactivity, this is amazing.
21:48You know, this is something to keep your eye on.
21:50but a lot of the Signals stuff that we see also isn't crossing
21:57the network gap all the time.
21:59and so that gets weird into like, well, how do you then roll things
22:03back or how do you do transactions?
22:06And so there's this split world, like you said, between lots of
22:11effort going into both of these.
22:13and I don't necessarily think that's a bad thing.
22:16so I'll fill you in a little bit on something that we have been working on.
22:20a little while ago we teased a library called TanStack Optimistic.
22:25And the idea around TanStack Optimistic came from Kyle Matthews, and he's actually
22:30the one who's been spearheading it.
22:32and which is a crazy full circle because if you go back six or seven years, Kyle
22:37and I were actually competing at one point building static site generation software.
22:43I was building React Static, and he was working on Gatsby.
22:46Next Js was getting into static site gen, and it was cool.
22:50he went and raised money and turned Gatsby into this awesome thing.
22:53And I was like, well, I have a company already, so I'm gonna fold.
22:59I said, I'll have to do this later.
23:00And what do you know?
23:02now he's working on a new, on at a company doing ElectricSQL
23:07and really cool stuff there.
23:08And now I'm the one building the framework.
23:10So the, the roles feel reversed, but we're working together on some things
23:14right now because Kyle at ElectricSQL, they've become very interested in,
23:19this exact kind of general toolkit that we have kind of been alluding to.
23:25and one of the first problems that they ran into with ElectricSQL and, and
23:29React Query and all this was that, they needed those optimistic updates to work.
23:35but in a way that was kind of compatible.
23:37Not just ElectricSQL, but just kind of in general, like what's this general
23:42kind of optimistic transactional layer?
23:45And that's what TanStack optimistic started out as.
23:48and I mean, it worked, it was functional.
23:49it still works today, right now, but it's, it's already evolving.
23:53So that was just, you know, a couple months ago that we were
23:56talking about TanStack optimistic.
23:58So problem number one, TanStack optimistic is a bad name.
24:02because we already have optimistic updates in React Query, TanStack Query.
24:05So it's like, well, does this have something to do with Query?
24:09so it was confusing, the second problem was that, we wanted to figure out, you
24:14know, does this overlap with Query or not?
24:18and could it, should it utilize Query or is it something completely different?
24:23Right.
24:24And.
24:25So it turns out that, part of this layer we believe up to this point
24:30is that, like you said, if you start pulling on this thread long
24:34enough, it becomes a database, right?
24:37And I don't really think that's a bad thing because a
24:42database is a very loose term.
24:43If you look at React Query, there's a Query cache inside of
24:48query, and that's a database, you know, it's a key value store.
24:52it's not relational, it's not, you know, sophisticated in any way.
24:56I should clarify.
24:57like the application should not become a database, typically, it
25:01accidentally becomes a database.
25:03And that's the point.
25:04The app shouldn't be the database, but an underlying technology that
25:07you use that goes above and beyond to be a database to do all the hard
25:12work that it takes to be a database.
25:14That should be the database.
25:15And I think that's exactly what you're saying.
25:17Totally agree.
25:18Yes.
25:19And that's, I think that's the most interesting thing, is that the database
25:23is gonna be there no matter what, whether it's a, a replication, a cache,
25:29a lens, whatever you wanna call it, all these different terms, right.
25:33But that's, the direction we're headed.
25:34So we, we've actually renamed TanStack, optimistic to TanStack DB.
25:40And we're only working on a small part of it right now, which is the optimistic
25:46parts and the collections part.
25:49So let me get into that a little bit.
25:51We don't need to go too far into this because I'll be honest, Kyle
25:54knows 100% more about this than I do.
25:58I know 1%.
26:00we're just kind of consulting together just to make sure that like we can
26:04build on top of each other's things.
26:06what I understand at this point is that, you know, optimistic updates with
26:11something like Query only get you so far.
26:13And at some point you need that schema, like I was saying, you
26:17need some kind of structure.
26:19and you need some way to declaratively kind of build up these
26:22relationships between your data.
26:25of your data and that you're pulling from somewhere and your, mutations and, the
26:31actions you're taking against your data.
26:34And just having consistent actions against your data solves a lot of the problems
26:40around, well, if it was just signals and we're just kind of firing reactivity
26:44events off all over the place, how do you transaction those things together?
26:48How do you roll things back?
26:49I mean, there's ways to do that.
26:51but it, we found that if we formalize things into mutations still, which
26:55is very common across a lot of different database, implementations
26:58that, that you can transact on those much easier and roll things back.
27:02I mean, it's the same way with Convex, you know, in LiveStore too.
27:06so we want a layer that is purely for the front end, it's purely
27:12for the client where we can define these collections and define these
27:19mutations that are going to happen and the relationships between those.
27:23and then those can be fulfilled by syn So for instance, you can make a collection
27:31to say, you know, here's all my users.
27:34here's my filters on those users.
27:37here's pagination, or here's just all the things that kind of create the
27:40lens into this data source that I have.
27:44And it's formalized contract, formalized schema on how you get that.
27:49And then you have mutations that say, you know, I'm mutating this thing.
27:54and this thing is a formalized structure within my application, and it has schema.
27:58And then.
27:59Behind that layer, you have an adapter to say, okay, now that we
28:04have this formalized structure, we can wire this up to whatever we want.
28:08We could read using a rest API, and kind of we could say, well, we want to pull
28:15it, or we want to use invalidation, or we want to use SSE or something like that.
28:21Or you could set it up to work with web sockets, or you could set it
28:24up to work with SQLite or whatever.
28:27it just needs to be fulfilled.
28:28so in a similar way to React Query, saying, a promise is all you need, right?
28:35for TanStack db, the loose vision right now is that you need something
28:42that can satisfy the interface or not even the interface, but
28:47the lifecycle of a sync engine.
28:50That might be super drop in, like ElectricSQL or whatever, or it
28:56might be more manual, which is what I'm really excited about to say.
29:00I already have a rest, API, I'm already using React Query.
29:03We can't upgrade into web sockets or like do a bunch of crazy infrastructure
29:08stuff right now on the backend.
29:11But if we could kind of upgrade the lifecycle into this
29:15contract of sync engine, right?
29:19What does it mean to be a sync engine for the front end?
29:22Then you could technically upgrade your front end data reading and writing
29:28experience to what feels like a sync engine, but behind the scenes, It
29:33could just be going out and kind of using the same APIs that existed in
29:38your company a year ago or whatever.
29:40So I think that's the vision and to me that sounds like a least common
29:45denominator approach to it to say, data could get here somehow, and we're gonna
29:51send data out somehow and we're gonna build formalized contracts around that.
29:56and then hopefully, we see adoption around adapters and patterns and utilities.
30:03and possibly even around things like TanStack Query where maybe you
30:08don't have web sockets or service and events or sync engine happening.
30:14So maybe, maybe TanStack Query is the fulfiller of this contract, I think
30:21that starts to look very interesting.
30:22to be very blunt, that's kind of the, the envelope that's the bleeding edge of my
30:27opinions and my knowledge on the topic.
30:29everything honestly, from here forward is probably gonna be more, nuanced opinion
30:35That is awesome.
30:36And I think that really nicely closed the loop.
30:38Something that stuck with me that you said, the beginning of our conversation is
30:43that if you squint, React Query is already kind of like a mini sync engine is, the,
30:49the benefit is it works with everything.
30:52The downside is it only goes so far and doesn't handle all of those edge
30:57cases in the most resilient way.
30:59Right?
30:59But with like Pareto principle style, like you do 20% of the
31:03effort, you get 80%, of the outcome.
31:06And now you're connecting this to, a variation of it where what React Query
31:12already does is like, it's a Query part.
31:15So what do you Query?
31:16You Query from like something where data comes from and now you
31:20basically say, actually that is.
31:23Already very useful.
31:25But where we are getting the data from, there's sort of like this
31:28in-between stage where typically you resolve a Query to a database.
31:34And so we only have that, we don't really acknowledge that there is a
31:37database in the middle, but yet there is a database in the middle, right?
31:41And once we have that, and once we acknowledge it and build all
31:44the things around it, we can actually Query much more, much
31:48better, much faster, much safer.
31:51we get like super fast filtering.
31:53We these updates, optimistic updates become simpler, et cetera.
31:57Sort of like embracing that
31:59when you, when you sprinkle schema into things, a lot of dev tools get better.
32:05So, you know, the dev tools for Query are awesome, don't get me wrong.
32:09They, I think they're, I think they're great.
32:11But when you have schema.
32:13And formalized interfaces on top of your data dev tooling
32:19can get way more sophisticated.
32:22in fact, I'm remembering all of the awesome stuff that I've seen
32:26from LiveStore, you know, out, out of dev tooling that's like,
32:30it would be impossible to do with a tool with just Query, right?
32:34but, but you see some of that come into play when you know,
32:38okay, GraphQL gets involved.
32:39It's like, well, now we have schema.
32:40Now we can do all this cool stuff.
32:42it's kind of the sa it's kind of the same thing here.
32:44Whenever you add schema into something and create stricter contracts, even if those
32:49contracts are with yourself, like you can build much better tooling around it.
32:53I couldn't agree more schemas, like nine out of 10 cases.
32:58It's a great deal.
32:59You put in a little bit of work and you get so much out of it constantly
33:04in all of those different dimensions, types, validation, and like you can
33:09also like, and I think as a industry we've only scratched the surface what
33:13we can actually do with schemas, like at least as a JavaScript industry in
33:17other program languages, schemas are like highly leveraged to, for example,
33:23work with data in a more efficient way.
33:26So this is where like Protobuf, JRPC, et cetera.
33:29A lot of that is used to minimize the amount of data that we send
33:33across the wires or have more stable contracts as we are evolving things.
33:37And I think we're just scratching the surface of like what will be the
33:41javaScript equivalent where we have, so right now, maybe we have something that
33:46also gives us the benefit of something like Protobuf in let's say 2030.
33:51I don't wanna be overly optimistic here.
33:54Sounds accurate.
33:54But, but yeah, I, I think this is super compelling what you're pitching here.
34:00And like, I don't think this is just a, pitch, but it's actually happening.
34:04And, Kyle, who we also had previously on the guest.
34:08Who's now at Electric, is an excellent person to lead the charge on this, so
34:12I'm really excited to, see that evolve.
34:15I have many more questions that I would like to ask you or Kyle.
34:18Let's do it.
34:18But I'll, I'll, uh, I won't go too much into that there.
34:22Just as you mentioned that you're, only so far in the weeds so far.
34:26but one thing that I'm also curious about in regards to TanStack DB is do
34:33you already have a hunch of like, what is that contract, that makes when you bring
34:38a plugable, sync engine that you provide there as a data source, what does that
34:44need to quack like, that it's a duck,
34:48right?
34:49yes, we do have a flexible and changing idea of what those, contracts look like.
34:56in fact, in inside of, it's still at slash.
35:00Still at TanStack slash optimistic on GitHub.
35:03but we do already have, like some of the core hooks and logic set up in there
35:08already to around collections, that the documentation obviously isn't there
35:14yet, but if you dig into the code in the examples, you can start to see, kind
35:19of the silhouette of what syncing is.
35:22So, you know, we have some configuration.
35:25there's like a sync option that takes a sync configuration.
35:30So it has been formalized, for now in terms of what we believe
35:35it needs to be, you know, to get a proof of concept out the door.
35:39and you know, one of the good examples to go look at in there is there is
35:42an Electric, there's an Electric sync configuration that now dog foods, that
35:48that interface in that contract to kind of give you an idea of how that could feel.
35:53Now things are gonna change, things are gonna get better, upgrade, you
35:56know, we'll, we'll try and add more adapters and break our expectations.
36:00So, the idea is that the, those sync contracts are going to evolve
36:04drastically, over the next little while.
36:08That sounds awesome.
36:09And I'm particularly excited about all of those adapters since once you start
36:13adjusting your own perspective, then almost anything can be a data source
36:18or a sync engine for that matter.
36:21It might not be inherently reactive, but then you throw polling on
36:24it and until that data source is itself reactive, you can sort of,
36:29paper over that through polling.
36:31And just like a, let's say the GitHub, API could be actually
36:35a sync engine implementation.
36:37Absolutely.
36:37And now you can build a local app that works with like GitHub data just as
36:43it's local, like code, like is local.
36:45That's the idea.
36:46And, you know, going beyond that, I think, with Query right now, it's easy
36:50to get trapped into thinking like, oh, I can just Query my, my one database.
36:55Or you start thinking, well, I can Query multiple.
36:58or even start Querying anything that is a promise.
37:01So, so once you start thinking about the, the sync protocol or the sync contract
37:06as just something to be fulfilled, it kind of opens up the same gate.
37:11It's like a parable to when people are like, oh, I can use React Query to
37:15use, like device APIs because they're just based on a promise, you know?
37:20So I imagine the same thing will happen, for these sync contracts as we kind of
37:25just learn that, you know, oh, anything that I can read and write to, I can turn
37:29into a sync, like a sync configuration.
37:32And I think it gets really exciting when you start looking at the ability to have
37:36multiple sync configurations working together and potentially merging into
37:41the same schema in your application.
37:44and they might handle different responsibilities.
37:46'cause the reality is that.
37:48I mean, a lot of people would love offline.
37:51You know, you can do offline for a lot of things, but I don't know of
37:55any application that's providing a lot of value these days that doesn't
38:01eventually need to connect to the server.
38:05And like you said, eventually need very strict transactions on I did
38:10this thing and did it happen, and we cannot proceed until it did.
38:13Right?
38:14So we're trying to make sure that we don't code ourselves into a corner
38:18here and we wanna design it so that, you can do all of those things
38:22kinda, you know, synchronous first.
38:24you can have sync contracts that are very optimistic and just fire
38:29and forget and we'll sync it later.
38:31And when you need to.
38:33You can kind of go deeper and say, okay, I'm finding off this mutation,
38:38but I'd like to opt into some more strict confirmations around it to
38:44make sure that we don't proceed.
38:45You can lock things easier.
38:49So I don't know, I like this future where it's, it's kind of like Query
38:54for Query in a way where you know, we're just adding organization, we're
39:00adding more intelligent lifecycle where we're adding a heartbeat to
39:07what people experience today is react Query is like, oh, I'm syncing data.
39:11We're just going to breathe a little bit more life into that engine,
39:16without trying to control everything about the experience, about the
39:22backend, and about the language that you use to do things and.
39:27you know, if we can create that adapter economy there, I think that, we could
39:31see some really cool stuff happen, we're working really closely with ElectricSQL
39:35and also with Convex, you know, convex uses Query directly, which is amazing.
39:41And we also have really good relationships with TRPC that uses Query.
39:46So we have a lot of people involved in this, space to different varying degrees.
39:53And I'm confident that that's, really why we think now is a good time
39:57is because we have enough signal.
40:01to design something that could be helpful to a majority of people.
40:05We're trying to take a slow though, to be honest, because this is
40:08not something you just rush into.
40:11Oh, I completely agree.
40:12Maybe using this very topic as an opportunity to pull back
40:16the, the curtains a little bit.
40:18what I see as just like the common threads through all of the, the projects, the
40:23all the TanStack projects, is that they have a very, kinda like the, the TanStack
40:28way of like how the API looks and feels.
40:30It's like, it's very intuitive.
40:32It's nicely modeled around the common scenarios, but then also allows for like
40:38reaching for that extra configuration, that extra functionality once you need
40:42it, but it doesn't overwhelm you upfront with like, needing to know of all of that.
40:47And now you're going into a new field and in an existing field even deeper.
40:53And so now you need to apply like that's.
40:56At the end, there is a new TanStack DB products and library emerging,
41:02and I'm certain that will feel just as intuitive and as nice
41:06as the other TanStack libraries.
41:09But that doesn't just happen, by itself.
41:12But there's something, a lot of intuition, a lot of experience
41:15that's going into that.
41:17And I'm not sure whether you formalized that, but we, we got
41:19some questions from the audience.
41:21whether you have any sort of like process guidelines, your own
41:25guidance, your inner guidance, intuition, how would you frame that?
41:29that could also serve as advice to others who aspire to build tools,
41:34like, like the ones you're working on?
41:37Well, I like to think about it.
41:39I like to approach it in a way of like, there's different tiers
41:44of things that you can control.
41:47So for me personally, if I'm working on a project.
41:50Building it myself.
41:51Like I can control a lot and I'm just very picky and, I try to
41:57be my best customer in a way.
41:59Like, I try to approach using my own software as if I don't know everything
42:04about it and call me schizophrenic.
42:07But like, you need to be able to kind of assume this other identity
42:10to where you're like, okay, I don't know anything about this project.
42:13I'm trying to use this API, this is confusing.
42:17What does this even mean?
42:19So you need to wear multiple hats at the same time and be able to switch
42:22between them very quickly to say, oh, yeah, okay, now I'm gonna go back
42:25to, architect Tanner and, and, change this to, make it easier to understand.
42:31So.
42:32A lot of it is just, I think just being very picky and just using your
42:35own tools, like a lot, using your own tools a lot and showing other people
42:40your tools before you release them and say, well what do you think about this?
42:44And trying to teach your tool to people and they'll say, Hey, that's
42:47really cool, but this is super weird.
42:49Like, I don't agree with that.
42:51So you just need to be getting feedback all the time and say,
42:54always be shipping, right?
42:55You can ship to a friend, you can ship to your company, you can ship somewhere.
43:00It doesn't have to be NPM before, you know you can get feedback.
43:05so that's kind of rule number one.
43:06after that, I believe that at this point for TanStack, things
43:12are also changing a little bit.
43:13Like I said, I'm not the one who's the driving force between TanStack DB Kyle is.
43:18and if you look at other libraries, the same thing has kind of been happening.
43:21So.
43:22TanStack form was, you know, the bulk of the entire project was spearheaded and
43:27completed and done by Corbin Crutchley.
43:30and it doesn't mean I didn't, I had a lot to do with the initial, like,
43:35parts of TanStack form, you know, for the first little while, but I
43:39can't be everywhere all at once.
43:42So I've mostly been focusing on router and start for the last two years.
43:47And so the next part of it is, you know, if you can't control everything
43:52on your own, you need to be able to find people that you trust.
43:56And that's really what it comes down to is I get petitions and, people every day
44:02on Twitter and on GitHub who are like, Hey, I have this idea for a library.
44:08We should do it.
44:09Or I have this idea for a feature or whatever.
44:12And.
44:13that's excellent feedback when we take that feedback into consideration
44:16for all, all everything that we see.
44:19but the choice of letting somebody come in and start making decisions for
44:23you is a really, really big decision.
44:25And it just needs to be one that you trust.
44:28And so if you see one of the, every single one of the core
44:31maintainers and core contributors to TanStack, did not happen overnight.
44:36except for Corbin Crutchley.
44:37I met Corbin and I knew it in 10 minutes that, he was the right person for the job.
44:41but it really comes down to trust.
44:43And, and at the end of the day, I trust, you know, Dominic and
44:47Kevin and Corbin, and Manuel, and Sean and Chris and Kyle, right?
44:54I trust all these individuals and many more that I didn't
44:58mention, because they understand.
45:01The brand.
45:02Now they understand our priorities and they understand what needs to happen.
45:07You know, we, prioritize type safety above all else.
45:11You know, we prioritize the 90% use case above all else, but we make it so that you
45:17can gracefully opt into advance features.
45:20it's not formalized in a document, but it is formalized in the experience that
45:26you get using one of these libraries.
45:29And if you know, you know, and if we're going to ship a new product,
45:33everybody collectively understands that it needs to meet the same
45:37standards as all the other products.
45:38So, and the people that I, invite in or, want to come and help me
45:43build stuff, those are people that I trust to execute on that vision.
45:46So.
45:47That is awesome.
45:47I fully agree with that.
45:49Thank you so much for, for sharing that.
45:51you've mentioned TanStack Router and TanStack Start both of those projects.
45:55I'm also using myself a lot.
45:58I'm curious if you now extrapolate a little bit further.
46:01TanStack Start is, just getting to a point where it's will hit I guess 1.0
46:07at some point as well and if you're now extrapolate forward TanStack start
46:12filling in all of those, features and, boxes that you have in mind.
46:16and if you then take TanStack DB and you combine them, there's
46:21possibly like some really interesting opportunities that weren't really
46:26thinkable or possible before.
46:28Are there any kind of napkin sketches, shower thoughts you can
46:32share where you imagine like some new emerging, superpowers or kinda like
46:37a new kind of framework that will be possible for this combination?
46:41not in the sense of like some new monolithic enclosure around these
46:47things, but like the vision that I have for it is that, already today you
46:53can, build a TanStack Start project.
46:57And when you build a Start project, you're not, just using start.
47:01People don't understand this either.
47:03So, I mean, I'll take the moment to explain it so that like, when you use
47:07Next js right, you install next JS and the router is enclosed inside of next
47:15js, and the server loading strategies are all enclosed inside of there.
47:19The SSR tools and everything is just, monolithic.
47:22It's, abstractions around a whole thing, right?
47:26And there, there're dependencies inside of each other.
47:28Now, TanStack start is very different.
47:31It's weird because we say the framework is TanStack start, but it's not.
47:36It's just a convenient marketing, term to make it sound different
47:41than what TanStack router is.
47:43So TanStack router is what you'll still import with a start application
47:49that you've used it, you know this.
47:51So when you import, most of the imports that you're making and the utilities
47:55that you're calling with TanStack Start are actually coming from router.
47:58there's actually very few that are coming from start until you need
48:02those server side capabilities, right?
48:05So when you have your server entry, or you want to write an API route, or
48:11you have a server function or you need some middleware, or you need to access
48:16request headers or something like that.
48:19That's when you reach for Start.
48:21And so it's interesting to think about Start and Router as actual
48:24their parallel dependencies together.
48:27and yes, Start does rely on like a peer dependency in a sense of TanStack Router.
48:34So that's why we kind of say, oh, you're using TanStack Start
48:37because it implies usage of Router.
48:40But where they're together, I think that's the most interesting part because
48:44you get into the app and you start building and you realize that you're
48:48really just using these tools together.
48:51And maybe you're using TanStack Router and you're like, oh, this caching is
48:55really good in the router and I like this.
48:57And then you get to a moment where you're like, oh, I really wish I had formalized
49:03mutations around this thing, or that I could share data between routes.
49:07So then you bring in TanStack Query and you know, you don't just turn
49:12it on in the router, you bring in TanStack Query and they work together.
49:16So.
49:17They have interfaces that, integrate with each other and
49:21they work alongside each other.
49:23And my hope for something like TanStack DB around the router and start is
49:31that it would be the same experience.
49:33You would bring it in and you would use it alongside these other utilities.
49:38but the solution should be that bringing in something like TanStack db, the end
49:44result should be that you should be able to, with a little bit more work,
49:48like you said, up at this top layer of producing, you know, providing some schema
49:54and providing some, some structure that you would be able to then get rid of a
49:58lot of the code, that you have in your.
50:02data loading lifecycle points and your user interaction points, in your actual
50:10business logic of your application.
50:13the hope for something like TanStack DB would be similar to the experience of,
50:18you know, somebody moving into TanStack Query, being able to get rid of all of
50:23that manual data fetching code with fetch and useEffect and all this other stuff.
50:29Moving to something like Query.
50:31I want that same experience for TanStack DB if it's warranted, right?
50:35So it's like you're unintentionally building schema and database and trying,
50:41you know, it's like you're trying to formalize all these contracts, but you,
50:44don't have the right tool to do it.
50:46Moving to TanStack DB should be an event that removes the burden of a lot of code
50:53that you've written and removes a lot of the burden of linking queries together and
50:58linking mutations and queries together.
51:00that should be the goal.
51:01not only goal for the developer side of it, but also the user side of it,
51:05where when you move to Query from something that's very manual, all
51:10of a sudden your data is up to date.
51:11Now it's fresher.
51:13You can tell a website that's probably using React Query because you do something
51:20and it updates and it's always there.
51:21You focus the window and it's always there.
51:24same thing for moving something like TanStack db.
51:27You'll see better results around mutations.
51:30You know, you'll click something it won't just be optimistic,
51:33but it'll be synced everywhere.
51:35and when things happen on the server.
51:38And, you know, hopefully if you're using something like web sockets or some kind
51:42of servers sent event, you'll just kind of get that synced to you automatically
51:46without needing to invalidate or refocus the window or anything.
51:49So, if we can move both of those chess pieces forward, you know, both
51:54on the developer experience side of, Hey, I, get to delete code now and
52:00have better contracts and a better experience as a developer and just
52:04more confidence, in what I'm doing.
52:07And as a user, you get to move forward to say things are faster, they're
52:12more responsive, and the user also has more confidence that what they're
52:16doing is happening and what they're seeing is up to date and real, right?
52:21So I think that's, the end goal.
52:24and if we can't accomplish that, we won't launch it.
52:27that's the requirement,
52:28That definitely resonates.
52:30And I think that's sort of like a, a nice litmus test for a library that if
52:36you adopt a library in your application and the library does some things
52:41that you've previously done before, ideally the library should absorb
52:47everything that is not intent, that is very specific about your application.
52:52So everything that remains is like pure intent.
52:56That's about my application.
52:58Everything else is like, you've hopefully picked the right tool that like the
53:02choice of the tool is part of your intent, but then you're assuming down like one
53:06level, become even more meta about within the scope of this tool, this library.
53:12Now I'm specifying my further intent when you pick a really, really
53:15great tool, it's, you can notice it by, it removing even more code
53:21that you had previously written.
53:22It surprises you of like how concise you can express what you want to
53:27do in a way that you didn't even expect how simple this can be.
53:31And I think a really great library kinda like surprises you through its simplicity.
53:37and so I wanna round out this conversation on one last topic that
53:42I think you're a really excellent person to share your thoughts on this.
53:46most TanStack projects are more client side centric, like TanStack table,
53:52TanStack, form TanStack Query, et cetera.
53:55those are all primitives that help us to build rich client side applications.
54:00but if you're looking at the React ecosystem, on a larger scale, there's
54:05been a lot of momentum and movement and developments more on the server side
54:10spectrum of application development.
54:13And I think it can be a bit confusing for developers who are not as
54:18experienced on either side of the spectrum to what to kind of lean into.
54:22So it's not everything is always compatible and, there's maybe some
54:27applications that are being built that go heavily on server side
54:31technologies, even though that's not the best fit for them and vice versa.
54:35So I'm curious whether you can share some reflections on, that sort of
54:40broadening spectrum of react covering the server as well, and whether you can
54:45provide some guidance to developers, where to, focus their, attention on.
54:50That's a hard question.
54:52because there's a couple of different ways to look at that.
54:56There's a personal way to look at it.
54:57Like, what should you be learning yourself to, you know, be relevant?
55:03and I think that you should know as much of the web platform as you can, right?
55:09Whether that's client or server.
55:11You should, you know, if you're a web developer, you need to know the web.
55:16and that includes servers.
55:18So, on a personal level, you need to be learning as much as you possibly can.
55:24does that mean you need to be learning every language you can.
55:26No.
55:27but it means you need to know the concepts, to be able to at least
55:31communicate with, people or with a team, you know, accurately.
55:35So there's like this personal level thing that I, I think is really important.
55:40Like, I'm definitely not an expert at, you know other languages and like server side
55:47building, server side things, servers.
55:50but I've worked with teams that have built very, very sophisticated systems and I've
55:55helped architect some of those systems.
55:57and you know, I've been able to work through that, those thought
56:00processes and learn with them.
56:01So I think that's very valuable.
56:03Now, in, practice, I think practice is a little bit different.
56:07and this is where it's like a big, it depends, because you might be at,
56:12you know, Y Combinator startup where you just have to know everything and
56:16you are building full stack because you don't have a choice, right?
56:20and in that moment, your job requires you to do it all.
56:24So you better do it all.
56:25You better learn it, and you better in invest into everything that
56:28you need to ship your product.
56:31And I think that's really what it comes down to in practice is that, yes, you have
56:35to be thinking about your career, right?
56:37But not.
56:39being negligent to the task at hand.
56:42And if you work for a company where you're expected to be a, full stack
56:47developer, then you better make sure that you're investing in both the
56:50front end and the back end equally.
56:52if you work for a company and you're a front end developer and you know that,
56:57you know, pretty soon you're gonna, they're gonna start running React
57:00server components and server technology.
57:03Does it mean you need to become a database expert and learn SQL and learn
57:06how to run Redis and, you know, and Kubernetes or just Docker or whatever?
57:11No, it doesn't necessarily mean all of that, but it does mean that within
57:15the realm of, of like JavaScript running on the server, you should
57:19probably start getting familiar with JavaScript on the server.
57:22You should probably start learning, you know, what those patterns look like.
57:28And I think, you know, on, on the, really far end of the spectrum, which is more
57:33common than you think, because there, there's companies out there that have been
57:37around for a really long time and they're running technology that is, that's old
57:42and outdated, and they move slow and they make a lot of money and they have a lot
57:46of teams and a lot of employees, and maybe they're publicly traded, but that doesn't
57:51mean that, you know, that they're even allowed to run JavaScript on the server.
57:56I mean, there are companies out there that are, you know, billion dollar
58:00companies that are not allowed to run JavaScript on the server yet.
58:04So, in situations like that, it's like, well, do you need to focus on
58:09learning React server components?
58:11Sure, you can go check it out and, and learn how to do some things
58:15with SSR and play around with it.
58:17But if that's your job, your job is probably to be the best front
58:21end engineer that you possibly can.
58:24And if, that's the case, you just need to make sure that you're focusing
58:28on what's important to your company.
58:30So I don't want to sound like, you know, you need to sell out
58:33your career to your company.
58:34obviously I, you know, I like to stay on the bleeding edge of everything as well.
58:38I'm just a really pragmatic person, and at the end of the
58:41day, everybody wants to get paid.
58:44Everybody needs to get paid, and we should not ignore that that is a reality.
58:49And if your paycheck is important to you it's a, a critical piece of your life.
58:55Then you need to be focusing on, on what's gonna make sure that that paycheck
58:59keeps growing and getting bigger.
59:01And if that paycheck isn't going to get bigger or grow with your effort, Then you
59:06should be learning other technologies that will help you secure either a different
59:12job or a promotion or whatever that's going to let you use that new knowledge
59:17to grow yourself and grow your career.
59:20That, that's kind of where I put it.
59:22So it is, a lot of it depends, right?
59:25totally.
59:25I think that's fantastic advice.
59:27We can leave it at that.
59:28I know that, you only have so much time.
59:31I just wanna highlight one point.
59:33You've used the word pragmatic and this is really what, for me resembles that's the
59:38essence of like the entire TanStack family of tools, like it's above everything else.
59:44I think it's pragmatic.
59:46So in case you ever wanna formalize that, that is like my
59:49candidate I'm offering to you.
59:51I wanna thank you so much for taking the time, sharing, everything.
59:55That you've shared today.
59:57and for all the loves, sweat and tears you're putting into all of
1:00:01your projects for such a long time.
1:00:03I'm really excited to see where all of this is going.
1:00:06Particularly excited for TanStack DB and, thank you so much
1:00:11for, coming on the show today.
1:00:13Absolutely.
1:00:13It's been a pleasure.
1:00:15Thank you for listening to the localfirst.fm podcast.
1:00:17If you've enjoyed this episode and haven't done so already, please
1:00:21subscribe and leave a review.
1:00:22Please also share this episode with your friends and colleagues.
1:00:25Spreading the word about the podcast is a great way to support
1:00:28it and to help me keep it going.
1:00:30A special thanks again to Jazz for supporting this podcast.
1:00:34I'll see you next time.