Always use the `copyToImport` logic when running SQL.

[SanjulaGanepola]

trafficstars

Changes

The setup I was running against had QCCSID 37. Also, *SECADM is required to run these tests so that the user profile's CCSID can be changed.

@worksofliam As we discussed this PR does not pass yet, due to the failing checkObject logic in getObjectList:

/home/SANJULA: /QOpenSys/usr/bin/qsh
system  "CHKOBJ OBJ(QSYS/"Aܧ#\$%Ä") OBJTYPE(*LIB) AUT(*NONE)"
{
    "code": 1,
    "signal": null,
    "stdout": "",
    "stderr": "CPD0012:  Characters in qualifier beginning 'Aܧ#\\$%Ä) ' not valid.\nCPF0001:  Error found on *N command."
}

Issue is also reproduceable manually in the object browser:

Checklist

• [ ] have tested my change
• [ ] have created one or more test cases
• [ ] updated relevant documentation
• [ ] Remove any/all console.logs I added
• [ ] have added myself to the contributors' list in CONTRIBUTING.md

[worksofliam]

@SanjulaGanepola This is still in draft. Are we waiting for more from you on this?

[SanjulaGanepola]

@worksofliam This initial PR was my attempt to create a test to reproduce the issues found in Issue #2059. However, in making this test, I see there is another bug with how the checkObject handles quotes (refer to the error message in the original issue message). You can see here that for the test, I got around this by using single quotes for the entire command that system will call and using pase. In the case for checkObject, can this be done as well and then switched to a pase environment? I tried this actually, but I get back no results as you can see below. Any ideas how to proceed?

[worksofliam]

@SanjulaGanepola Do you mind if I/we make commits to this PR in order to attempt to fix this issue?

To do:

• [ ] Fix for checkObject to support double quoted names.

[worksofliam]

@SanjulaGanepola If you do ls /QSYS.LIB/ABCD.LIB where "Aܧ#$%Ä" exists, can you share the output here?

[SanjulaGanepola]

Do you mind if I/we make commits to this PR in order to attempt to fix this issue?

Please feel free to commit to this PR.

If you do ls /QSYS.LIB/ABCD.LIB where "Aܧ#$%Ä" exists, can you share the output here?

[worksofliam]

I did kind of find part of the problem. The issue is that we were always executing ILE commands using QSH, which seems to break the encoding. If we swap out QSH with PASE, then the encoding is intact and works as expect. Now the issue is that object_statistics isn't working as expected. I am working with @forstie to see what he thinks.

-- set the ccsid
cl: CHGUSRPRF USRPRF(liam) CCSID(273);
select CHARACTER_CODE_SET_ID from table( QSYS2.QSYUSRINFO( USERNAME => upper('liam') ) );

-- create ugly object
cl: CRTLIB LIB("Aܧ#$%Ä") TEXT("§#$öße");
cl: CRTSRCPF FILE("Aܧ#$%Ä"/"àáãÄ£ø") RCDLEN(112) CCSID(273) TEXT("Üä$öß");
cl: ADDPFM FILE("Aܧ#$%Ä"/"àáãÄ£ø") MBR("§#$MAN") SRCTYPE(CBLLE) TEXT("§#$öße");

-- check it exists
cl: CHKOBJ OBJ(QSYS/\"Aܧ#\$%Ä\") OBJTYPE(*LIB) AUT(*NONE);

-- with sql
select 
  OBJNAME          as NAME,
  OBJTYPE          as TYPE,
  OBJATTRIBUTE     as ATTRIBUTE,
  OBJTEXT          as TEXT,
  0                as IS_SOURCE,
  IASP_NUMBER      as IASP_NUMBER,
  OBJSIZE          as SIZE,
  extract(epoch from (OBJCREATED))*1000       as CREATED,
  extract(epoch from (CHANGE_TIMESTAMP))*1000 as CHANGED,
  OBJOWNER         as OWNER,
  OBJDEFINER       as CREATED_BY
from table(QSYS2.OBJECT_STATISTICS(OBJECT_SCHEMA => '"Aܧ#$%Ä"', OBJTYPELIST => '*LIB'));

[forstie]

Both of these worked fine for me. That said, the operating system has sharp edges when odd characters are used in library names. Therefore, the user should be encouraged to avoid such names and we should focus on more important topics.

-- create ugly object cl: CRTLIB LIB("Aܧ#$%Ä") TEXT("§#$öße"); create table "Aܧ#$%Ä".foo(name char(10)); select * from table(QSYS2.OBJECT_STATISTICS(OBJECT_SCHEMA => '"Aܧ#$%Ä"', OBJTYPELIST => '*ALL')); cl: dltlib LIB("Aܧ#$%Ä");

cl: CHGUSRPRF USRPRF(scottf) CCSID(273); select CHARACTER_CODE_SET_ID from table( QSYS2.QSYUSRINFO( USERNAME => upper('scottf') ) ); -- create ugly object cl: CRTLIB LIB("Aܧ#$%Ä") TEXT("§#$öße"); create table "Aܧ#$%Ä".foo(name char(10)); select * from table(QSYS2.OBJECT_STATISTICS(OBJECT_SCHEMA => '"Aܧ#$%Ä"', OBJTYPELIST => '*ALL')); cl: dltlib LIB("Aܧ#$%Ä");

[worksofliam]

@forstie can you share the system QCCSID that you ran this on?

[forstie]

65535

[worksofliam]

I can now prove object_statistics is fine and that means getObjectList works ok too. But, the next issue is our getMemberList API.

This SQL proves no result:

Click me

With MEMBERS As (
        SELECT
          rtrim(cast(a.system_table_schema as char(10) for bit data)) as LIBRARY,
          b.avgrowsize as RECORD_LENGTH,
          a.iasp_number as ASP,
          rtrim(cast(a.system_table_name as char(10) for bit data)) AS SOURCE_FILE,
          rtrim(cast(b.system_table_member as char(10) for bit data)) as NAME,
          coalesce(rtrim(cast(b.source_type as varchar(10) for bit data)), '') as TYPE,
          coalesce(rtrim(varchar(b.partition_text)), '') as TEXT,
          b.NUMBER_ROWS as LINES,
          extract(epoch from (b.CREATE_TIMESTAMP))*1000 as CREATED,
          extract(epoch from (b.LAST_SOURCE_UPDATE_TIMESTAMP))*1000 as CHANGED
        FROM qsys2.systables AS a
          JOIN qsys2.syspartitionstat AS b
            ON b.table_schema = a.table_schema AND
              b.table_name = a.table_name
      )
      Select * From MEMBERS
      Where LIBRARY = '"Aܧ#$%Ä"'
        And SOURCE_FILE = '"àáãÄ£ø"'
        And NAME Like '"§#$MAN"'
        
      -- Order By NAME DESC

But if I use a LIKE clause to find something, we get a result, but not as I expected because we use for bit data

Click me

With MEMBERS As (
        SELECT
          rtrim(cast(a.system_table_schema as char(10) for bit data)) as LIBRARY,
          b.avgrowsize as RECORD_LENGTH,
          a.iasp_number as ASP,
          rtrim(cast(a.system_table_name as char(10) for bit data)) AS SOURCE_FILE,
          rtrim(cast(b.system_table_member as char(10) for bit data)) as NAME,
          coalesce(rtrim(cast(b.source_type as varchar(10) for bit data)), '') as TYPE,
          coalesce(rtrim(varchar(b.partition_text)), '') as TEXT,
          b.NUMBER_ROWS as LINES,
          extract(epoch from (b.CREATE_TIMESTAMP))*1000 as CREATED,
          extract(epoch from (b.LAST_SOURCE_UPDATE_TIMESTAMP))*1000 as CHANGED
        FROM qsys2.systables AS a
          JOIN qsys2.syspartitionstat AS b
            ON b.table_schema = a.table_schema AND
              b.table_name = a.table_name
      )
      Select * From MEMBERS
      where LIBRARY like '%Ü%'

Further investigation showed that we can't really use any SQL to fetch object or member information, because no matter the user profile CCSID or QCCSID, the system views are always 37/65535/ascii as that is what is expected of objects. I did kind of get the list to work with a fake bit data column, but the columns are still coming back incorrectly.

Click me

      With MEMBERS As (
        SELECT
          rtrim(cast(a.system_table_schema as char(10) for bit data)) as LIBRARY_BIT,
          rtrim(cast(a.system_table_schema as char(10))) as LIBRARY,
          b.avgrowsize as RECORD_LENGTH,
          a.iasp_number as ASP,
          rtrim(cast(a.system_table_name as char(10))) AS SOURCE_FILE,
          rtrim(cast(b.system_table_member as char(10))) as NAME,
          coalesce(rtrim(cast(b.source_type as varchar(10))), '') as TYPE,
          coalesce(rtrim(varchar(b.partition_text)), '') as TEXT,
          b.NUMBER_ROWS as LINES,
          extract(epoch from (b.CREATE_TIMESTAMP))*1000 as CREATED,
          extract(epoch from (b.LAST_SOURCE_UPDATE_TIMESTAMP))*1000 as CHANGED
        FROM qsys2.systables AS a
          JOIN qsys2.syspartitionstat AS b
            ON b.table_schema = a.table_schema AND
              b.table_name = a.table_name
      )
      Select * From MEMBERS
      where LIBRARY_BIT like '%Ü%'

I believe we should not support delimited object names due to this reason.

[SanjulaGanepola]

Updated this PR to always use the copyToImport logic when running SQL.

Before (Code4i):

Before (ACS):

After (Code4i):

[worksofliam]

@SanjulaGanepola Is it possible the test case can only pass on a system with a valid QCCSID? I am running this on a 37 and it's not looking too good:

code:
255
command:
'CRTLIB LIB("Aܧ#$%Ä") TEXT("§#$öße")'
signal:
null
stderr:
`CPD0020: Character '�' not valid following string '"A�       '.\nCPD0020: Character '�' not valid following string '"§#$ö�  '.\nCPF0006: Errors occurred in command.`
stdout:
''

edit: tested on another system also with 65535 and it worked. Confused

edit2: solved it. bad user profile

[SanjulaGanepola]

@worksofliam Any update on this PR?

[worksofliam]

New PR may solve this issue. I will close for now.