[Feature Request] ZenStack for API integration

[ymc9]

NOTE: This is quite ambitious item, and chance is low it can be contained in V2, but including it here just for people who may find it very useful and want to give it a push

The proposal here is based on previous discussions with people on Discord. Moving to GitHub for better visibility.

---

Goal

Prisma’s schema offers a pleasant DX to model and program against a database. However, the pattern - DSL schema + TS code generation, has more potential than working with databases. Since ZenStack has already replicated the entire Prisma schema language with many extensions, we can try to achieve more above that.

Web development has been gradually shifting from hosting everything in-house to leveraging external services. That means databases, as important as they still are, only represent a (shrinking) part of the entire application’s model. Databases often store “opaque references” to models of foreign systems, and developers have to deal with disparate APIs and orchestrate them with database operations.

Hopefully, we can make the schema a gateway toward unifying and simplifying the DX:

• Unified CRUD API style with natural joining and nesting
• Consistently strongly-typed everywhere
• Unified access control

ZenStack should provide a more powerful plugin mechanism, allowing plugin developers to contribute models and runtime behavior to interface with 3rd-party API systems.

Use Cases

1. Auth

When using auth services like Clerk, the main user repository is on the auth provider’s side. If you don’t choose to synchronize it to your own database, you can reference it as a foreign model:

plugin clerk {
    provider = 'zenstack-plugin-clerk'
    secretKey = env('CLERK_SECRET_KEY')
}

// ClerkUser model is imported from the "clerk" plugin and extended with a one-to-one "user" relationship
// Syntax here is subject to discussion
model ClerkUser imports clerk::User {

    // Imported models can only declare relation fields

    // a "virtual" relation to the `User` model
    // `onDelete` marks it to be deleted when `User` is deleted
    user User? @relation(onDelete: Cascade)

    @@allow('create', false)
    @@allow('all', user != null && auth() == user)
}

model User {
    id String @id

    // a one-to-one relationship with the `ClerkUser` foreign model,
    // internally mapped to a field storing clerk's user id
    clerkUser ClerkUser

    @@allow('create', true)
    @@allow('all', auth() == this)
}

model Post {
    id String @id
    author User @relation(...)
    authorId String
  
    @@allow('read', true)
    @@allow('all', auth() == author)
}

And the enhanced PrismaClient allows you to do things like:

// user will be typed as `User & { clerkUser: ClerkUser }`
// type `ClerkUser` is provided by the `zenstack-plugin-clerk` plugin
const user = await prisma.user.findMany({
    where: { clerkUser: { email } },
    include: { clerkUser: true }
});

// update user attribute on clerk side
await prisma.user.update({
    where: { id },
    data: {
        clerkUser: { update: { attributes: { level: newLevel } } }
    }
});

// fetch posts with user email
await prisma.post.findMany({
    where: { authorId: userId },
    include: { clerkUser: { select: { email: true } } }
});

2. Blob Data

Using blob services like S3 to store files and save a reference in database table is a very common pattern.

plugin blob {
    provider = 'zenstack-plugin-s3'
    region = env('S3-REGION')
    bucket = env('S3-BUCKET')
    ...
}

model UserImageBlob imports blob::Blob {
    user User? @relation(onDelete: Cascade)

    @@allow('read', true)
    @@allow('all', user != null && auth() == user)
}

model User {
    id String @id
    profileImage UserImageBlob?
}

await prisma.user.update({
    where: { id },
    data: { profileImage: { update: { data: buffer } } }
});

const user = await prisma.user.findUnique({
    where: { id }, include: { profileImage: true }
});

const imageData = await readStream(user.profileImage.data);

3. Subscription

Interfacing with payment systems like Stripe:

plugin stripe {
    provider = 'zenstack-plugin-stripe'
    apiKey = env('STRIPE-API-KEY')
}

model Subscription imports stripe::Subscription {
    team Team? @relation(onDelete: Cascade)

    @@allow('read', team.members?[user == auth()])
    @@allow('all', team.members?[user == auth() && role == 'ADMIN'])
}

model Team {
    id String @id
    members TeamMember[]
    subscription Subscription?
}

// pause the team's current subscription
await prisma.team.update({
    where: { id },
    data: { subscription: { update: { status: 'paused' } } }
});

Potential Approach

• Plugins can contribute “virtual” models.
• Plugins can provide CRUD handlers for the virtual models.
• CRUD handlers mainly interface with 3rd party APIs, work synchronously, and have built-in retries, but don’t guarantee transactional atomicity and consistency.
• [MAYBE] Plugins can contribute real models so that it creates database tables to their own data.

Open Questions

• Is it appropriate to use CRUD to model all interactions with 3rd-party systems? Or is it necessary for plugins to contribute extension methods as well?
• Calling remote APIs inside DB transactions can result in negative impacts due to long-running transactions. Maybe there should be an asynchonous “fire-and-forget” mode for mutation?
• Related to the previous one: what should a transaction actually mean if it mixes database operations and foreign service updates? Should db operations actually be completed and then proceed with service updates?

Related

Supabase FDW

[Azzerty23]

Hi @ymc9, and thank you for this RFC. If I understand correctly, these plugins (clerk, blob, stripe...) would be built-in to Zenstack (non-customizable)? Initially, I was thinking of a more extensible system where we can write the desired logic, with the plugin providing access to model fields and abstracting other fields if needed (abstract models). It would be a kind of wrapper for Prisma's client-extensions AND Zenstack's abstract models... In my use case, here's what I would have liked to be able to do:

// Google Cloud Storage
plugin GCS {
    // local_path/gcs.ts where I define custom methods (like the upload method and custom getter)
    provider = "./extensions/gcs"
    abstractModels = [File]
}

abstract model File {
    filename    String
    originalFilename    String
    bucket      String
    path        String
}

model User {
    id            String   @id @default(cuid())
    profileImage  GCS
    ...
}

model Product {
    id          String   @id @default(cuid())
    image       GCS
    ...
}

await prisma.user.update({  // --> not sure about that...
    where: { id },
    data: { profileImage: { file, bucket, path } } }
});

const user = await prisma.user.findUnique({
    where: { id }
});

const imageUrl = await user.profileImage;

In any case, I find the plugin system very useful, and the proposed syntax very clear.

[ymc9]

Hi @Azzerty23 , thanks for the comment and example. My apologies for not being clear in the initial draft.

The plugins are supposed to be implemented outside of ZenStack, and hopefully, many of them by the community. You can think of the "provider" of the plugin to be all independent and standalone NPM packages. In fact, most of today's plugins, like trpc, tanstack-query, etc., don't depend on any inner workings of ZenStack's core and only obey the plugin contract. I've made an update to the original proposal.

If we were to go in this direction, ZenStack's core would define a richer contact for the plugins, like how to contribute models (as shown in your example with the File model) and behavior (create/update an object on GCS).

I haven't thought much about if this should be implemented based on Prisma's client extensions. It's definitely worth a try, just not sure if we'll soon hit some limit there.

[bvkimball]

Not sure if is worth a look, but Supabase is creating something similar this from a database perspective here https://supabase.github.io/wrappers/stripe/

The wrapper expose data through the database as a plugin.... It would be really COOL to have a ZenSack plugin that added/synced with those data models. So i could query through my prisma client.

For me... i really only want to generate them in the client and ignore them in any "migrations", I think it would be great if we expose client queries to tables we don't OWN. This might not be important but maybe a consideration of what this plugins can do.

[ymc9]

Thanks for the pointer @bvkimball ! I didn't know Supabase has a set of Postgres FDW. They can be very good references. Yes, indeed the idea is quite close to FDW, just work at a higher level.

The foreign models won't be mapped to the database so shouldn't affect migration. The only trace they'll leave on the database is when you create a relation from a real model to a foreign model, on the real model side there should be a "virtual" foreign key that's persisted into the db.

[pedrosimao]

Very interesting. I just wonder if defining the plugins on the schema itself would later become a limitation in terms of adding custom logic to the plugin functions / helpers.

[ymc9]

Very interesting. I just wonder if defining the plugins on the schema itself would later become a limitation in terms of adding custom logic to the plugin functions / helpers.

Hi @pedrosimao , the plugins are just declared in the schema, and their implementation will be made in TS/JS.

[andrictham]

One question I do have:

Let’s say I plan to use Lemonsqueezy as a payments/billing service instead of Stripe, and the Zenstack library doesn’t have a first-party plugin for that, but only for Stripe.

How easy would it be to create our own custom plugins to wrap any external API we wish?

[ymc9]

One question I do have:

Let’s say I plan to use Lemonsqueezy as a payments/billing service instead of Stripe, and the Zenstack library doesn’t have a first-party plugin for that, but only for Stripe.

How easy would it be to create our own custom plugins to wrap any external API we wish?

Hi @andrictham , the idea is to make API integration part of the plugin mechanism, so that developers can introduce new integrations without changing the core. The detailed design hasn't been made yet.