The gang talks to Jeremy Miller about some of his open source projects.
Download / Listen: Herding Code 224: Jeremy Miller on Marten, Postgres and Alba
Show Notes:
- Marten
- (01:20) Jeremy’s shops really like using document databases because there’s a lot less friction to develop and ship your applications. They were having trouble with their document databases in production, though, so when they saw Postgres has JSON support they decided to build a client library on top of it so they’d have the developer productivity of a document database, but with the stability and tools that come with Postgres.
- (03:24) K Scott mentions Jeremy’s recent blog post comparing the simplicity of building applications using a document database as opposed to using an ORM. Jeremy explains how document databases allow you to have a more evolutionary design that’s able to handle changes.
- (04:35) K Scott asks about their recent use of Marten as an event store. Jeremy says they’re building up view projections that express the read side state of the application. The advantage of having them in Marten is that they’re just documents, and you don’t need additional infrastructure on top of your application document database.
- (06:07) K Scott remarks on the great documentation and API design. Jeremy says that after years of abuse over his StructureMap documentation, he’s decided to make documentation a much bigger emphasis for this project.
- (06:45) K Scott asks about how the documents are stored in the database. Jeremy says each document type has an underlying table with some metadata columns, and the main document is in a single JSONB column. JSONB is a Postgres data type that’s a binary representation of a JSON document that’s much more efficient for searching and querying.
- (08:12) Jon asks about differences between BSON, JSONB, etc.
- (09:12) Jon asks what the main reasons for using Marten over just storing documents in JSONB. Jeremy says the LINQ provider is a lot more user friendly and productive than parsing through JSON yourself. There’s also the unit of work which gives you transactional semantics, there’s database migration tooling that handles things like foreign key changes. Strong expression typed support, calculated indexes, and working with the in-database JavaScript support to map data in the database.
- (12:22) Jon asks if he’d looked at Entity Framework. Jeremy says that due to differences in data structures, it’s a pretty different model. There’s nothing that prevents you from using Marten for documents in a Postgres database alongside EF or Dapper for relational tables in the same database.
- (13:51) K Scott asked where Marten falls in transactional support, since document databases take pretty different views on that. Jeremy says that Marten is ACID all the way down – you can query for a document as soon as it’s committed, and you don’t need to deal with eventual consistency unless you opt into it for event store.
- (15:12) K Scott asks about the testing story. Jeremy talks about his shop’s integration testing strategy, and that document databases are a lot easier to integration test than relational databases. From a unit testing perspective, they do some minimal mocking, but he recommends integration testing rather than unit testing in most places.
- (17:42) Kevin asks about JSON patching support, since Postgres has support for that.
- (18:48) K Scott asks about their use of Sqitch. Jeremy explains how it’s a simple database migration system using SQL semantics.
- (20:35) Kevin has a question about data migrations for updating existing documents. Jeremy recommends using their patching API for that; if you can’t do that use JavaScript in the database.
- (22:02) Jon asks about the release notes mention of lowering memory usage. Jeremy says the main work was in minimizing serializing to JSON strings and using object pooling.
- (24:04) K Scott mentions the successful release and active Gitter channel. Jeremy says this is the smoothest experience he’s had with an open source release.
- (25:05) Kevin asks if there are other libraries out there that are making it easier to work with Postgres, and asks if there’s any hope of standardization across Postgres, Oracle, and SQL Server.
- (27:22) Scott K asks about the main selling points of Marten on top of Postgres.
- (28:22) Scott K asks if they’ve looked at cloud document databases as well, like DocumentDB. Jeremy explains that it’s pretty tied to Postgres.
- (29:40) Scott K asks about best practices on querying. Jeremy says they invested a lot of time in performance optimizations, and talks about calculated indexes, calculated fields for complex structures, and Marten’s compiled queries in C# code (which bypass compiling LINQ queries). The npgsql library they run on top of also supports batching, which can give you big boosts in chatty applications. There’s also support for including related fields in one database roundtrip.
- Alba
- (34:04) K Scott asks about Alba. Jeremy talks about the need their shop has for unit tests that run the full HTTP request cycle without hitting a webserver. This allows you to test the full end to end output for cases where you have filters, caching, middleware, etc. It was originally inspired by the Play framework in Scala.
- (38:33) Jon asks when it makes sense to move from the ASP.NET Core test host to use Alba. Jeremy says that it’s mostly a matter of preference, but Alba does a good amount of legwork for you for things like routing.
- (39:52) K Scott asks how Alba fits in with integration tests with Selenium. Jeremy says he’d really just want to use Selenium when there’s some significant UI functionality, but for the most part he’d avoid it as much as possible and use Alba.
- (41:00) Jon asks about Jeremy’s experiences in porting to .NET Core. Jeremy says that the newer projects were straightforward, but his older projects that had a good amount of reflection were really hard. He’s sorry to see project.json go, and he things the dotnet CLI is worth upgrading just about on its own since it makes automation almost trivial.
- Changes in the .NET Open Source world
- (43:47) K Scott circles back to ask about Jeremy’s comments about Marten being the most positive experience he’s had with .NET open source. Jeremy says the .NET open source ecosystem is a lot more positive and helpful. Marten fills a pretty good niche that many people found helpful. Jeremy says that he’s seen the user community become a lot more collaborative compared to previously treating open source maintainers as product support. Jeremy says he also gets pull requests for documentation, which is something he’d never expect a few years ago.
- StructureMap
- (46:26) Kevin asks about Jeremy’s thoughts on turning StructureMap over to someone else. Jeremy says it’s been 14 years, people use it in ways he wouldn’t expect, ASP.NET Core has a built-in IoC Container, and it’s hard to make StructureMap work with the ASP.NET Core IoC requirements. The project is still viable and continuing, it’s just going to be better for everyone under new ownership.
- Misc Questions
- (49:15) Frank asks on Twitter “Why so much hate for Scrum?” Jeremy talks about the ceremony and tracking in Scrum take all the joy out of Agile programming. Jeremy says Scrum is the Scrappy Doo of the Agile Programming world.
- (51:22) Jon asks about Jeremy’s recent post: The Different Meanings of “I take pull requests”. Jeremy reviews them from the most positive to the most negative.
- (54:05) K Scott asks about Jeremy’s post: Thoughts on Agile Database Development. Jeremy says that relational databases have long been the final frontier of agile development, and the database community has resisted agile development.
- (56:42) K Scott asks what Jeremy’s up to, and Jeremy talks about his next release in May: a human.