neosync icon indicating copy to clipboard operation
neosync copied to clipboard

Any option to define destination db's schema?

Open LodeSNA opened this issue 1 year ago • 3 comments
trafficstars

Is your feature request related to a problem?

Well, in some dev env, sometimes the db's schema is different, not just "public".

Describe the solution you'd like

Just like source db's public.aaa1 sync to destination db's abc.aaa1。

Describe alternatives you've considered

Table to table with mapping, just like source db's public.aaa1 sync to destination db's public.bbb1? Not sure if it's a good idea.

Thx !

LodeSNA avatar May 20 '24 06:05 LodeSNA

Hi there! Thanks for the feature request. This is something we have considered. It's not on our immediate roadmap, but is something we'd like to see added to Neosync eventually!

There are definitely a few different usecases here that I can think of:

  1. Schema Migration. (highlighted above). -- When you want to move all or some of your tables from one schema to another.
  2. Table Migration. -- When you want to move your data from one table to another

nickzelei avatar May 20 '24 15:05 nickzelei

@saytesnake Can you tell me more about your immediate need/usecase? Is it that in production/some env your tables are in X schema but in development you have them in Y schema?

nickzelei avatar May 20 '24 15:05 nickzelei

@saytesnake Can you tell me more about your immediate need/usecase? Is it that in production/some env your tables are in X schema but in development you have them in Y schema?

I have a usecase as follows, to synchronize tables from a specific database across multiple PostgreSQL instances into one database in a single PostgreSQL instance, using schema to distinguish each database. This is for simple offline analysis, statistics, etc., and to create views for other vendors to access and use.

I know some extension like postgres_fdw and dblink, but the performance is not ideal, especially when using views (although this might be related to the quality of the SQL queries).

Some tools, such as Airbyte, is too complicated for me, and maybe I don't need the big data tech stack (such as ETL, ODS, DW...). Therefore, we found neosync on Github, which is an excellent tool that is simple and easy to use.

graph TD;
	PG_1.public.aaa1-->neosync-->PG_AIO;
	PG_2.public.bbb1-->neosync-->PG_AIO;
	PG_3.public.ccc1-->neosync-->PG_AIO;
        PG_AIO-->PG_AIO.pg_1.aaa1;
        PG_AIO-->PG_AIO.pg_2.bbb1;
        PG_AIO-->PG_AIO.pg_3.ccc1;

LodeSNA avatar May 30 '24 06:05 LodeSNA