steampipe-postgres-fdw icon indicating copy to clipboard operation
steampipe-postgres-fdw copied to clipboard

Published 20 hours ago • verified publisher icon turbot

Partial results with an error should never be cached in the FDW

Open e-gineer opened this issue 3 years ago • 0 comments

We have an issue where timestamp columns create an error when set to nil. This happens for some rows only in the github_release table. But, I'm seeing that after enough runs, the "bad" partial result set may get cached - particularly when used in a join / aggregation.

Here is a long dump of the sequence I can use to get it to happen. Notably it doesn't seem to be cached from the plain table results, but it is cached when used in a join situation?

~/src/steampipe-plugin-github $ steampipe query
Welcome to Steampipe v0.7.2
For more information, type .help
> select * from github_release where repository_full_name = 'turbot/steampipe'
Error: failed to populate column 'published_at': rpc error: code = Internal desc = transform convertTimestamp failed with panic runtime error: invalid memory address or nil pointer dereference
> 
> select * from github_release where repository_full_name = 'turbot/steampipe'

Error: failed to populate column 'published_at': rpc error: code = Internal desc = transform convertTimestamp failed with panic runtime error: invalid memory address or nil pointer dereference

+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
| repository_full_name | assets                                                                                                                                                          
+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
| turbot/steampipe     | [{"browser_download_url":"https://github.com/turbot/steampipe/releases/download/v0.7.2/checksums.txt","content_type":"text/plain; charset=utf-8","created_at":"2
|                      | rl":"https://api.github.com/users/github-actions%5Bbot%5D/received_events","repos_url":"https://api.github.com/users/github-actions%5Bbot%5D/repos","site_admin"
|                      | %5D/events{/privacy}","followers_url":"https://api.github.com/users/github-actions%5Bbot%5D/followers","following_url":"https://api.github.com/users/github-acti
|                      | "},{"browser_download_url":"https://github.com/turbot/steampipe/releases/download/v0.7.2/steampipe_linux_amd64.tar.gz","content_type":"application/gzip","create
|                      | t%5D/orgs","received_events_url":"https://api.github.com/users/github-actions%5Bbot%5D/received_events","repos_url":"https://api.github.com/users/github-actions
+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> select * from github_release where repository_full_name = 'turbot/steampipe'
Error: failed to populate column 'published_at': rpc error: code = Internal desc = transform convertTimestamp failed with panic runtime error: invalid memory address or nil pointer dereference
> 
> select * from github_release where repository_full_name = 'turbot/steampipe'
Error: failed to populate column 'published_at': rpc error: code = Internal desc = transform convertTimestamp failed with panic runtime error: invalid memory address or nil pointer dereference
> 
> select * from github_release where repository_full_name = 'turbot/steampipe'
Error: failed to populate column 'published_at': rpc error: code = Internal desc = transform convertTimestamp failed with panic runtime error: invalid memory address or nil pointer dereference
> 
> 
> select * from github_release where repository_full_name = 'turbot/steampipe'

Error: failed to populate column 'published_at': rpc error: code = Internal desc = transform convertTimestamp failed with panic runtime error: invalid memory address or nil pointer dereference

+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
| repository_full_name | assets                                                                                                                                                          
+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
| turbot/steampipe     | [{"browser_download_url":"https://github.com/turbot/steampipe/releases/download/v0.7.1-debug.2/checksums.txt","content_type":"text/plain; charset=utf-8","create
|                      | events_url":"https://api.github.com/users/github-actions%5Bbot%5D/received_events","repos_url":"https://api.github.com/users/github-actions%5Bbot%5D/repos","sit
|                      | -actions%5Bbot%5D/events{/privacy}","followers_url":"https://api.github.com/users/github-actions%5Bbot%5D/followers","following_url":"https://api.github.com/use
|                      | ssets/41832524"},{"browser_download_url":"https://github.com/turbot/steampipe/releases/download/v0.7.1-debug.2/steampipe_linux_amd64.tar.gz","content_type":"app
|                      | /github-actions%5Bbot%5D/orgs","received_events_url":"https://api.github.com/users/github-actions%5Bbot%5D/received_events","repos_url":"https://api.github.com/
| turbot/steampipe     | [{"browser_download_url":"https://github.com/turbot/steampipe/releases/download/v0.3.5/checksums.txt","content_type":"text/plain; charset=utf-8","created_at":"2
|                      | rl":"https://api.github.com/users/github-actions%5Bbot%5D/received_events","repos_url":"https://api.github.com/users/github-actions%5Bbot%5D/repos","site_admin"
|                      | 5D/events{/privacy}","followers_url":"https://api.github.com/users/github-actions%5Bbot%5D/followers","following_url":"https://api.github.com/users/github-actio
|                      | },{"browser_download_url":"https://github.com/turbot/steampipe/releases/download/v0.3.5/steampipe_linux_amd64.tar.gz","content_type":"application/gzip","created
|                      | %5D/orgs","received_events_url":"https://api.github.com/users/github-actions%5Bbot%5D/received_events","repos_url":"https://api.github.com/users/github-actions%
+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> select * from github_release where repository_full_name = 'turbot/steampipe'
Error: failed to populate column 'published_at': rpc error: code = Internal desc = transform convertTimestamp failed with panic runtime error: invalid memory address or nil pointer dereference
> 
> 
> select * from github_release where repository_full_name = 'turbot/steampipe'

Error: failed to populate column 'published_at': rpc error: code = Internal desc = transform convertTimestamp failed with panic runtime error: invalid memory address or nil pointer dereference

+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
| repository_full_name | assets                                                                                                                                                          
+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
| turbot/steampipe     | [{"browser_download_url":"https://github.com/turbot/steampipe/releases/download/v0.7.0/checksums.txt","content_type":"text/plain; charset=utf-8","created_at":"2
|                      | rl":"https://api.github.com/users/github-actions%5Bbot%5D/received_events","repos_url":"https://api.github.com/users/github-actions%5Bbot%5D/repos","site_admin"
|                      | %5D/events{/privacy}","followers_url":"https://api.github.com/users/github-actions%5Bbot%5D/followers","following_url":"https://api.github.com/users/github-acti
|                      | "},{"browser_download_url":"https://github.com/turbot/steampipe/releases/download/v0.7.0/steampipe_linux_amd64.tar.gz","content_type":"application/gzip","create
|                      | t%5D/orgs","received_events_url":"https://api.github.com/users/github-actions%5Bbot%5D/received_events","repos_url":"https://api.github.com/users/github-actions
+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> 
> select
  r.name as release_name,
  r.published_at,
  a ->> 'name' as asset_name,
  a ->> 'download_count' as download_count
from
  github_release as r,
  jsonb_array_elements(assets) as a
where
  r.repository_full_name = 'turbot/steampipe'
  and a ->> 'content_type' in ('application/zip', 'application/gzip')
order by
  r.published_at desc,
  asset_name;
Error: failed to populate column 'published_at': rpc error: code = Internal desc = transform convertTimestamp failed with panic runtime error: invalid memory address or nil pointer dereference
> 
> select
  r.name as release_name,
  r.published_at,
  a ->> 'name' as asset_name,
  a ->> 'download_count' as download_count
from
  github_release as r,
  jsonb_array_elements(assets) as a
where
  r.repository_full_name = 'turbot/steampipe'
  and a ->> 'content_type' in ('application/zip', 'application/gzip')
order by
  r.published_at desc,
  asset_name;
Error: failed to populate column 'published_at': rpc error: code = Internal desc = transform convertTimestamp failed with panic runtime error: invalid memory address or nil pointer dereference
> 
> select
  r.name as release_name,
  r.published_at,
  a ->> 'name' as asset_name,
  a ->> 'download_count' as download_count
from
  github_release as r,
  jsonb_array_elements(assets) as a
where
  r.repository_full_name = 'turbot/steampipe'
  and a ->> 'content_type' in ('application/zip', 'application/gzip')
order by
  r.published_at desc,
  asset_name;
Error: failed to populate column 'published_at': rpc error: code = Internal desc = transform convertTimestamp failed with panic runtime error: invalid memory address or nil pointer dereference
> 
> 
> select
  r.name as release_name,
  r.published_at,
  a ->> 'name' as asset_name,
  a ->> 'download_count' as download_count
from
  github_release as r,
  jsonb_array_elements(assets) as a
where
  r.repository_full_name = 'turbot/steampipe'
  and a ->> 'content_type' in ('application/zip', 'application/gzip')
order by
  r.published_at desc,
  asset_name;
Error: failed to populate column 'published_at': rpc error: code = Internal desc = transform convertTimestamp failed with panic runtime error: invalid memory address or nil pointer dereference
> 
> 
> select
  r.name as release_name,
  r.published_at,
  a ->> 'name' as asset_name,
  a ->> 'download_count' as download_count
from
  github_release as r,
  jsonb_array_elements(assets) as a
where
  r.repository_full_name = 'turbot/steampipe'
  and a ->> 'content_type' in ('application/zip', 'application/gzip')
order by
  r.published_at desc,
  asset_name;
Error: failed to populate column 'published_at': rpc error: code = Internal desc = transform convertTimestamp failed with panic runtime error: invalid memory address or nil pointer dereference
> 
> 
> select
  r.name as release_name,
  r.published_at,
  a ->> 'name' as asset_name,
  a ->> 'download_count' as download_count
from
  github_release as r,
  jsonb_array_elements(assets) as a
where
  r.repository_full_name = 'turbot/steampipe'
  and a ->> 'content_type' in ('application/zip', 'application/gzip')
order by
  r.published_at desc,
  asset_name;
+----------------+---------------------+------------------------------+----------------+
| release_name   | published_at        | asset_name                   | download_count |
+----------------+---------------------+------------------------------+----------------+
| v0.7.2         | 2021-08-06 10:34:15 | steampipe_darwin_amd64.zip   | <hidden>            |
| v0.7.2         | 2021-08-06 10:34:15 | steampipe_linux_amd64.tar.gz | <hidden>            |
| v0.7.1-debug.2 | 2021-08-05 20:26:42 | steampipe_darwin_amd64.zip   | <hidden>              |
| v0.7.1-debug.2 | 2021-08-05 20:26:42 | steampipe_linux_amd64.tar.gz | <hidden>              |
| v0.7.1-debug.0 | 2021-08-04 14:34:54 | steampipe_darwin_amd64.zip   | <hidden>              |
| v0.7.1-debug.0 | 2021-08-04 14:34:54 | steampipe_linux_amd64.tar.gz | <hidden>              |
| v0.7.1-rc.0    | 2021-07-26 16:17:47 | steampipe_darwin_amd64.zip   | <hidden>              |
| v0.7.1-rc.0    | 2021-07-26 16:17:47 | steampipe_linux_amd64.tar.gz | <hidden>            |
+----------------+---------------------+------------------------------+----------------+
> 
> 
> select
  r.name as release_name,
  r.published_at,
  a ->> 'name' as asset_name,
  a ->> 'download_count' as download_count
from
  github_release as r,
  jsonb_array_elements(assets) as a
where
  r.repository_full_name = 'turbot/steampipe'
  and a ->> 'content_type' in ('application/zip', 'application/gzip')
order by
  r.published_at desc,
  asset_name;
+----------------+---------------------+------------------------------+----------------+
| release_name   | published_at        | asset_name                   | download_count |
+----------------+---------------------+------------------------------+----------------+
| v0.7.2         | 2021-08-06 10:34:15 | steampipe_darwin_amd64.zip   | <hidden>            |
| v0.7.2         | 2021-08-06 10:34:15 | steampipe_linux_amd64.tar.gz | <hidden>            |
| v0.7.1-debug.2 | 2021-08-05 20:26:42 | steampipe_darwin_amd64.zip   | <hidden>              |
| v0.7.1-debug.2 | 2021-08-05 20:26:42 | steampipe_linux_amd64.tar.gz | <hidden>              |
| v0.7.1-debug.0 | 2021-08-04 14:34:54 | steampipe_darwin_amd64.zip   | <hidden>              |
| v0.7.1-debug.0 | 2021-08-04 14:34:54 | steampipe_linux_amd64.tar.gz | <hidden>              |
| v0.7.1-rc.0    | 2021-07-26 16:17:47 | steampipe_darwin_amd64.zip   | <hidden>              |
| v0.7.1-rc.0    | 2021-07-26 16:17:47 | steampipe_linux_amd64.tar.gz | <hidden>            |
+----------------+---------------------+------------------------------+----------------+
> 
> 
> select
  r.name as release_name,
  r.published_at,
  a ->> 'name' as asset_name,
  a ->> 'download_count' as download_count
from
  github_release as r,
  jsonb_array_elements(assets) as a
where
  r.repository_full_name = 'turbot/steampipe'
  and a ->> 'content_type' in ('application/zip', 'application/gzip')
order by
  r.published_at desc,
  asset_name;
+----------------+---------------------+------------------------------+----------------+
| release_name   | published_at        | asset_name                   | download_count |
+----------------+---------------------+------------------------------+----------------+
| v0.7.2         | 2021-08-06 10:34:15 | steampipe_darwin_amd64.zip   | <hidden>            |
| v0.7.2         | 2021-08-06 10:34:15 | steampipe_linux_amd64.tar.gz | <hidden>            |
| v0.7.1-debug.2 | 2021-08-05 20:26:42 | steampipe_darwin_amd64.zip   | <hidden>              |
| v0.7.1-debug.2 | 2021-08-05 20:26:42 | steampipe_linux_amd64.tar.gz | <hidden>              |
| v0.7.1-debug.0 | 2021-08-04 14:34:54 | steampipe_darwin_amd64.zip   | <hidden>              |
| v0.7.1-debug.0 | 2021-08-04 14:34:54 | steampipe_linux_amd64.tar.gz | <hidden>              |
| v0.7.1-rc.0    | 2021-07-26 16:17:47 | steampipe_darwin_amd64.zip   | <hidden>              |
| v0.7.1-rc.0    | 2021-07-26 16:17:47 | steampipe_linux_amd64.tar.gz | <hidden>            |
+----------------+---------------------+------------------------------+----------------+
> 
>

e-gineer avatar Aug 17 '21 20:08 e-gineer