Effect-TS
Add a drizzle-schema package
What is the problem this feature would solve?
I'm slowly getting into Effect and am coming from a codebase where I use drizzle-zod, which is a plugin for Drizzle ORM that allows you to generate Zod schemas from Drizzle ORM schemas.
For example, you would be able to do something like the following:
import { pgEnum, pgTable, serial, text, timestamp } from 'drizzle-orm/pg-core';
import { createInsertSchema, createSelectSchema } from 'drizzle-schema';
const noteTable = pgTable('note', {
id: serial('id').primaryKey(),
content: text('content').notNull(),
});
// Schema for inserting a user - can be used to validate API requests
const insertNoteSchema = createInsertSchema(noteTable);
// Schema for selecting a user - can be used to validate API responses
const selectNoteSchema = createSelectSchema(noteTable);
export const noteApi = pipe(
Api.make({ title: "Notes API" }),
Api.addEndpoint(
pipe(
Api.post("createNote", "/notes"),
Api.setRequestBody(insertNoteSchema),
Api.setResponseBody(selectNoteSchema),
Api.setResponseStatus(201),
Api.addResponse(ApiResponse.make(500, NoteApiError)),
),
),
);
What is the feature you are proposing to solve the problem?
It would be great to have an Effect equivalent (drizzle-schema) as described above.
What alternatives have you considered?
I found this gist in this Discord thread, but it's using an old version of Effect and I'm way too much on an Effect noob to even consider porting this to the latest version.
@amosbastian here's initial version without refine (the second argument) which passes insert test for mysql, pg and sqlite
https://github.com/arekmaz/drizzle-orm/blob/effect-schema/drizzle-schema/src/effect-schema.ts
edit: after some hours the insert tests are passing with refine implemented
here's the draft pr to drizzle-orm (I think it belongs there next to drizzle-zod)
https://github.com/drizzle-team/drizzle-orm/pull/2613
@amosbastian all tests are passing, insert select, with/without refine, for each db, if you want you can copy that file to your project and test it, the only thing left is some package cleanup, and probably some time will pass before they accept the merge request, if they do at all
Thanks for your hard work @arekmaz! I've finally got round to using it today, and it's not 100% clear to me how we can use this to do something like
const requestSchema = insertUserSchema.pick({ name: true, email: true });
as we would with drizzle-zod, but that's probably just me being an Effect beginner 😅
EDIT: it doesn't work at the moment but in the end you should be able to do this:
const insertUserSchema = createInsertSchema(usersTable);
const requestSchema = insertUserSchema.pick('name', 'email');
as the output schemas will be just Schema.Struct, probably I should mention that somewhere
it's documented here
there is also the Schema.pick operator but it doesn't work on structs for me at the moment
I'm a bit too busy to cleanup the package and request the merge at the moment, on weekend I'll probably get around to it
I created a new pull request https://github.com/drizzle-team/drizzle-orm/pull/2665 - I'm new to commit signing
let's see if they accept it
@datner it looks like the drizzle team did not even look at the pr, what do you think we should do with this?
maybe try to merge it into @effect/sql-drizzle? or create a standalone package? or just wait?
deferring to @mikearnaldi 🙏🏻 I am not sure what would be preferable. I default to wait but that does not guarantee a conclusion
I'd wait a bit longer and eventually evaluate if to merge in sql-drizzle
I was just looking for this today. Currently I have to define drizzle schemas, my own domain schemas and then make conversions between the two. It is a bit cumbersome, and I wish this was already available. To me it feels much more natural to look for this kind of things in the schema repository and docs.
@danielo515 given 4 parsers: foo, bar, baz, and qux. Knowing that drizzle-orm had drizzle-foo, drizzle-bar, and drizzle-baz in their repo. Where would you expect drizzle-qux to be? How would you expect it to be named? It's no ideal if we would be required to randomly host it on our end :
But if they are not interested we can't force them
You can ping the PR there if you're interested in pushing it in hehe
I updated the types slightly to account for the different modes of date, bigint, etc. Still possibly incomplete
export type GetSchemaForType<TColumn extends Drizzle.Column> =
TColumn["dataType"] extends infer TDataType
? TColumn["columnType"] extends infer TColumnType
? TDataType extends "custom"
? Schema.Schema<any>
: TDataType extends "json"
? Schema.Schema<Json, Json>
: TColumn extends { enumValues: [string, ...string[]] }
? Drizzle.Equal<
TColumn["enumValues"],
[string, ...string[]]
> extends true
? Schema.Schema<string>
: Schema.Schema<TColumn["enumValues"][number]>
: TDataType extends "bigint"
? Schema.Schema<bigint, bigint>
: TDataType extends "number"
? TColumnType extends `PgBigInt${number}`
? Schema.Schema<bigint, number>
: Schema.Schema<number, number>
: TDataType extends "string"
? TColumnType extends "PgNumeric"
? Schema.Schema<number, string>
: TColumnType extends "PgUUID"
? Schema.Schema<string>
: TColumnType extends "PgDateString"
? Schema.Schema<Date, string>
: TColumnType extends "PgTimestampString"
? Schema.Schema<Date, string>
: Schema.Schema<string, string>
: TDataType extends "boolean"
? Schema.Schema<boolean>
: TDataType extends "date"
? Schema.Schema<Date>
: Schema.Schema<any>
: never
: never;
Hey guys, I forked the work done by @arekmaz and did some major refactors, fixed a few bugs
- Jsonb columns were overloading the typescript server being a recursive type
- Providing a different name to your drizzle column that is not the key value prevented the key from being available in Schema.omit, Schema.pick etc)
Additionally, I wrote some more tests since there's been substantial changes.
I also included @ghardin1314's GetSchemaForType changes.
The code is available here and can be added to your project via pnpm add @handfish/drizzle-effect.
Let me know if this is well received, and maybe I'll give trying to open a pull to drizzle to get an official drizzle-effect another try.
The readme.md is still a TODO for my refactor.
@Handfish I think you should definitely try opening a PR on the official drizzle repo since the maintainers have publicly started acknowledging effect by tweeting about it, maybe they'll consider it now.
@ghardin1314 I noticed the refine property in createSelectSchema and such wasn't type safe. This seems to be working better for me
import * as Drizzle from 'drizzle-orm'
import * as DrizzleMysql from 'drizzle-orm/mysql-core'
import * as DrizzlePg from 'drizzle-orm/pg-core'
import * as DrizzleSqlite from 'drizzle-orm/sqlite-core'
import { Schema } from 'effect'
type Columns<TTable extends Drizzle.Table> = TTable['_']['columns']
type ColumnSchema<TColumn extends Drizzle.Column> =
TColumn['dataType'] extends 'custom'
? Schema.Schema<any>
: TColumn['dataType'] extends 'json'
? Schema.Schema<JsonValue>
: TColumn extends { enumValues: [string, ...string[]] }
? Drizzle.Equal<
TColumn['enumValues'],
[string, ...string[]]
> extends true
? Schema.Schema<string>
: Schema.Schema<TColumn['enumValues'][number]>
: TColumn['dataType'] extends 'bigint'
? Schema.Schema<bigint, bigint>
: TColumn['dataType'] extends 'number'
? TColumn['columnType'] extends `PgBigInt${number}`
? Schema.Schema<bigint, number>
: Schema.Schema<number, number>
: TColumn['columnType'] extends 'PgNumeric'
? Schema.Schema<number, string>
: TColumn['columnType'] extends 'PgUUID'
? Schema.Schema<string>
: TColumn['columnType'] extends 'PgDate'
? TColumn extends { mode: 'string' }
? Schema.Schema<string, string>
: Schema.Schema<Date, string>
: TColumn['columnType'] extends 'PgTimestamp'
? TColumn extends { mode: 'string' }
? Schema.Schema<string, string>
: Schema.Schema<Date, string>
: TColumn['dataType'] extends 'string'
? Schema.Schema<string, string>
: TColumn['dataType'] extends 'boolean'
? Schema.Schema<boolean>
: TColumn['dataType'] extends 'date'
? TColumn extends { mode: 'string' }
? Schema.Schema<string>
: Schema.Schema<Date>
: Schema.Schema<any>
// Simplified JSON types to prevent inference explosion
type JsonPrimitive = string | number | boolean | null
type JsonObject = { readonly [key: string]: unknown } // Match Schema.Record output
type JsonArray = readonly unknown[] // Match Schema.Array output
type JsonValue = JsonPrimitive | JsonObject | JsonArray
// Strict JSON types for full validation
type StrictJsonObject = { readonly [key: string]: StrictJsonValue }
type StrictJsonArray = readonly StrictJsonValue[]
type StrictJsonValue = JsonPrimitive | StrictJsonObject | StrictJsonArray
// Non-recursive JSON schema to avoid type inference explosion
export const JsonValue = Schema.Union(
Schema.String,
Schema.Number,
Schema.Boolean,
Schema.Null,
Schema.Record({ key: Schema.String, value: Schema.Unknown }),
Schema.Array(Schema.Unknown)
) satisfies Schema.Schema<JsonValue>
// For cases where you need full JSON validation, use this explicit version
export const StrictJsonValue = Schema.suspend(
(): Schema.Schema<StrictJsonValue> =>
Schema.Union(
Schema.String,
Schema.Number,
Schema.Boolean,
Schema.Null,
Schema.Record({ key: Schema.String, value: StrictJsonValue }),
Schema.Array(StrictJsonValue)
)
)
// Utility type to prevent unknown keys (similar to drizzle-zod)
type NoUnknownKeys<T, U> = T & {
[K in Exclude<keyof T, keyof U>]: never
}
// Simplified refinement types
type RefineFunction<TTable extends Drizzle.Table> = (
schemas: { [K in keyof Columns<TTable>]: Schema.Schema<any> }
) => Schema.Schema<any>
type RefineArg<TTable extends Drizzle.Table> =
| Schema.Schema<any>
| RefineFunction<TTable>
// Clean refinement type without ugly satisfies
type TableRefine<TTable extends Drizzle.Table> = {
[K in keyof Columns<TTable>]?: RefineArg<TTable>
}
// Build refine type that maps column schemas to refinement functions
// Properly typed approach that preserves specific column schema types
type BuildRefine<TColumns extends Record<string, Drizzle.Column>> = {
[K in keyof TColumns]?:
| Schema.Schema<any>
| ((schema: ColumnSchema<TColumns[K]>) => Schema.Schema<any>)
}
// Property signature builders - simplified
type InsertProperty<
TColumn extends Drizzle.Column,
TKey extends string,
> = TColumn['_']['notNull'] extends false
? Schema.PropertySignature<
'?:',
Schema.Schema.Type<ColumnSchema<TColumn>> | null | undefined,
TKey,
'?:',
Schema.Schema.Encoded<ColumnSchema<TColumn>> | null | undefined,
false,
never
>
: TColumn['_']['hasDefault'] extends true
? Schema.PropertySignature<
'?:',
Schema.Schema.Type<ColumnSchema<TColumn>> | undefined,
TKey,
'?:',
Schema.Schema.Encoded<ColumnSchema<TColumn>> | undefined,
true,
never
>
: ColumnSchema<TColumn>
type SelectProperty<TColumn extends Drizzle.Column> =
TColumn['_']['notNull'] extends false
? Schema.Schema<Schema.Schema.Type<ColumnSchema<TColumn>> | null>
: ColumnSchema<TColumn>
// Base schema builders
type InsertColumnSchemas<TTable extends Drizzle.Table> = {
[K in keyof Columns<TTable>]: InsertProperty<Columns<TTable>[K], K & string>
}
type SelectColumnSchemas<TTable extends Drizzle.Table> = {
[K in keyof Columns<TTable>]: SelectProperty<Columns<TTable>[K]>
}
// Refined schema builders - controlled complexity
type BuildInsertSchema<
TTable extends Drizzle.Table,
TRefine = {},
> = Schema.Struct<InsertColumnSchemas<TTable> & TRefine>
type BuildSelectSchema<
TTable extends Drizzle.Table,
TRefine = {},
> = Schema.Struct<SelectColumnSchemas<TTable> & TRefine>
// Clean API functions with type safety
export function createInsertSchema<TTable extends Drizzle.Table>(
table: TTable
): BuildInsertSchema<TTable, {}>
export function createInsertSchema<
TTable extends Drizzle.Table,
TRefine extends BuildRefine<Columns<TTable>>,
>(
table: TTable,
refine: NoUnknownKeys<TRefine, TTable['$inferInsert']>
): BuildInsertSchema<TTable, TRefine>
export function createInsertSchema<
TTable extends Drizzle.Table,
TRefine extends TableRefine<TTable> = {},
>(table: TTable, refine?: TRefine): BuildInsertSchema<TTable, TRefine> {
const columns = Drizzle.getTableColumns(table)
const columnEntries = Object.entries(columns)
let schemaEntries: Record<
string,
Schema.Schema.All | Schema.PropertySignature.All
> = Object.fromEntries(
columnEntries.map(([name, column]) => [name, mapColumnToSchema(column)])
)
// Apply refinements
if (refine) {
const refinedEntries = Object.entries(refine).map(
([name, refineColumn]) => [
name,
typeof refineColumn === 'function' &&
!Schema.isSchema(refineColumn) &&
!Schema.isPropertySignature(refineColumn)
? (refineColumn as any)(schemaEntries[name])
: refineColumn,
]
)
schemaEntries = Object.assign(
schemaEntries,
Object.fromEntries(refinedEntries)
)
}
// Apply insert-specific optionality rules
for (const [name, column] of columnEntries) {
if (!column.notNull) {
schemaEntries[name] = Schema.optional(
Schema.NullOr(schemaEntries[name] as Schema.Schema.All)
)
} else if (column.hasDefault) {
schemaEntries[name] = Schema.optional(
schemaEntries[name] as Schema.Schema.All
)
}
}
return Schema.Struct(schemaEntries) as any
}
export function createSelectSchema<TTable extends Drizzle.Table>(
table: TTable
): BuildSelectSchema<TTable, {}>
export function createSelectSchema<
TTable extends Drizzle.Table,
TRefine extends BuildRefine<Columns<TTable>>,
>(
table: TTable,
refine: NoUnknownKeys<TRefine, TTable['$inferSelect']>
): BuildSelectSchema<TTable, TRefine>
export function createSelectSchema<
TTable extends Drizzle.Table,
TRefine extends TableRefine<TTable> = {},
>(table: TTable, refine?: TRefine): BuildSelectSchema<TTable, TRefine> {
const columns = Drizzle.getTableColumns(table)
const columnEntries = Object.entries(columns)
let schemaEntries: Record<
string,
Schema.Schema.All | Schema.PropertySignature.All
> = Object.fromEntries(
columnEntries.map(([name, column]) => [name, mapColumnToSchema(column)])
)
// Apply refinements first with base schemas
if (refine) {
const refinedEntries = Object.entries(refine).map(
([name, refineColumn]) => [
name,
typeof refineColumn === 'function' &&
!Schema.isSchema(refineColumn) &&
!Schema.isPropertySignature(refineColumn)
? (refineColumn as any)(schemaEntries[name])
: refineColumn,
]
)
schemaEntries = Object.assign(
schemaEntries,
Object.fromEntries(refinedEntries)
)
}
// Apply select-specific nullability rules after refinements
for (const [name, column] of columnEntries) {
if (!column.notNull) {
schemaEntries[name] = Schema.NullOr(
schemaEntries[name] as Schema.Schema.All
)
}
}
return Schema.Struct(schemaEntries) as any
}
// Helper function to check if a column has a mode property
function hasMode(column: any): column is { mode: string } {
return (
typeof column === 'object' &&
column !== null &&
'mode' in column &&
typeof column.mode === 'string'
)
}
function mapColumnToSchema(column: Drizzle.Column): Schema.Schema<any, any> {
let type: Schema.Schema<any, any> | undefined
if (isWithEnum(column)) {
type = column.enumValues.length > 0
? Schema.Literal(...column.enumValues)
: Schema.String
}
if (!type) {
if (Drizzle.is(column, DrizzlePg.PgUUID)) {
type = Schema.UUID
} else if (column.dataType === 'custom') {
type = Schema.Any
} else if (column.dataType === 'json') {
type = JsonValue
} else if (column.dataType === 'array') {
type = Schema.Array(
mapColumnToSchema((column as DrizzlePg.PgArray<any, any>).baseColumn)
)
} else if (column.dataType === 'number') {
type = Schema.Number
} else if (column.dataType === 'bigint') {
type = Schema.BigIntFromSelf
} else if (column.dataType === 'boolean') {
type = Schema.Boolean
} else if (column.dataType === 'date') {
type =
hasMode(column) && column.mode === 'string'
? Schema.String
: Schema.DateFromSelf
} else if (column.dataType === 'string') {
// Additional check: if it's a PgTimestamp or PgDate masquerading as string
if (Drizzle.is(column, DrizzlePg.PgTimestamp)) {
type =
hasMode(column) && column.mode === 'string'
? Schema.String
: Schema.DateFromSelf
} else if (Drizzle.is(column, DrizzlePg.PgDate)) {
type =
hasMode(column) && column.mode === 'string'
? Schema.String
: Schema.DateFromSelf
} else {
let sType = Schema.String
if (
(Drizzle.is(column, DrizzlePg.PgChar) ||
Drizzle.is(column, DrizzlePg.PgVarchar) ||
Drizzle.is(column, DrizzleMysql.MySqlVarChar) ||
Drizzle.is(column, DrizzleMysql.MySqlVarBinary) ||
Drizzle.is(column, DrizzleMysql.MySqlChar) ||
Drizzle.is(column, DrizzleSqlite.SQLiteText)) &&
typeof column.length === 'number'
) {
sType = sType.pipe(Schema.maxLength(column.length))
}
type = sType
}
}
}
if (!type) {
type = Schema.Any // fallback
}
return type
}
function isWithEnum(
column: Drizzle.Column
): column is typeof column & { enumValues: [string, ...string[]] } {
return (
'enumValues' in column &&
Array.isArray(column.enumValues) &&
column.enumValues.length > 0
)
}
Drizzle team:
Hey folks, Is there an update on Drizzle-EffectTS native integration?
yes, we already have a PoC and going to start building fully featured support in 1-2 weeks
https://discord.com/channels/1043890932593987624/1235939318351007884/1422264874528014468
@AhmedBaset, Discord has a broken link system, and your link works only for the people already inside the server, not outsiders. What Discord server does your link point to?
I don't know if you're in Effect's Discord server, come join if not ;)
I shared there (Discord thread: Drizzle Team's Work with Effect in Drizzle ORM) a link to Effect integration source code, which is a directory with the Effect's part of implementation for the new Drizzle ORM version. And if I haven't missed anything, it's not connected to the schema in any way, just functionality to yield* the result of drizzle db method calls, instead of using promises.
