pgloader icon indicating copy to clipboard operation
pgloader copied to clipboard

Direct URL can't be mapped to file system

Open bobir01 opened this issue 1 year ago • 2 comments

URL that does not end with simple file link query cant be read to system files data url is changed to public csv file due to risks of privacy, but the core problem is not about file content here ;(

  • [x] pgloader --version

    pgloader version "3.6.7~devel"
    compiled with SBCL 2.1.11.debian
    
  • [] did you test a fresh compile from the source tree?

    Compiling pgloader from sources is documented in the README, it's easy to do, and if patches are to be made to fix your bug, you're going to have to build from sources to get the fix anyway…

  • [x] did you search for other similar issues?

  • [x] how can I reproduce the bug?

    I am trying to insert csv file that has tailing tokens after its name, so pgloader could not write/read this file


LOAD CSV
    FROM 'https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z' WITH ENCODING utf-8
    INTO postgresql://user:*****@localhost:5432/db
    TARGET TABLE cdm_commerce.dm_basic_osa_hourly

    WITH
        skip header = 1,
        fields terminated by ','
        SET work_mem to '1GB', maintenance_work_mem to '2GB';

BORDET|Bobirjon Mardonov
  • [x] pgloader output you obtain
2024-08-26T13:09:57.012000Z LOG pgloader version "3.6.7~devel"
2024-08-26T13:09:57.012000Z LOG Data errors in '/tmp/pgloader/'
2024-08-26T13:09:57.012000Z LOG Parsing commands from file #P"/home/b_mardonov/csv.load"
KABOOM!
SIMPLE-FILE-ERROR: Can't find the TRUENAME of wild pathname /home/b_mardonov/https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z) (physicalized from /home/b_mardonov/https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z).
An unhandled error condition has been signalled:
   Can't find the TRUENAME of wild pathname /home/b_mardonov/https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z (physicalized from /home/b_mardonov/https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z).




What I am doing here?

Can't find the TRUENAME of wild pathname #1=/home/b_mardonov/https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z (physicalized from #1#).
  • [x] data that is being loaded, if relevant
[data.csv](https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z)
  • [ ] How the data is different from what you expected, if relevant

bobir01 avatar Aug 26 '24 13:08 bobir01

this seems very silly issue why there is no option to rename blob downloading ?

bobir01 avatar Aug 26 '24 13:08 bobir01