pgloader icon indicating copy to clipboard operation
pgloader copied to clipboard

PGloader - How to pass connection string in pgloader as environment variables

Open SamirMjs opened this issue 3 years ago • 3 comments

LOAD CSV FROM '/dbf/appworx/pgscript/temp_table_data.csv' (INV_ID) INTO postgresql://username:password@hostname:portname/dbname TARGET TABLE temp_inv_table

Running through shells script -

#!/bin/ksh

echo "Start" #enviroment variable . /xzy/DBset

echo "Fetching password value"

pgloader temp_table_load2.load

How do we pass value to temp_table_load2.load file using shell or if possible how we pass directly to load file I am getting same above commented error.

Thanks you for help in advance

SamirMjs avatar Jun 07 '22 13:06 SamirMjs

It's possible to use the bash source <config_file> command in your script and then use sed to replace placeholders like <password> with the values in the config you're sourcing. However I'd recommend taking advantage of the mustache templating that pgloader supports.

So you would create a config.ini and then change your pgloader command to pgloader --context config.ini temp_table_load2.load

I know the documentation is a little hard to decipher in places, but it's definitely worth a read.

TestSubject1498 avatar Jun 10 '22 15:06 TestSubject1498

Thank for valuable feedback. Its working now. Now i have another question how to set default value to column level like "Current_date', "Nextval" like. By using CSV loading

SamirMjs avatar Jun 13 '22 07:06 SamirMjs

I'm a little unclear what you mean. It sounds like maybe you want to add a CAST section to your pgloader script. Could you provide a code example with place holders? I believe I've seen references to this, though I haven't tested it extensively.

CAST
    column {{target_table}}.{{target_column)) to timestamp without time zone set default NOW()

you can also use the BEFORE LOAD and AFTER LOAD to execute SQL scripts and commands if you know how to do whatever you're suggesting in SQL.

TestSubject1498 avatar Jun 13 '22 18:06 TestSubject1498