Missing columns when download data from dashboard (when columns have null values)

[fer-batista]

Describe the bug

When I download a table from the dashboard, only 35 columns are included. However, when downloading directly from the question, all columns are present in the file.

To Reproduce

1. Go to a dashboard with one table
2. Click on 'download results'
3. XLSX formated

Expected behavior

No response

Logs

No response

Information about your Metabase installation

{
  "browser-info": {
    "language": "pt-BR",
    "platform": "MacIntel",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
    "vendor": "Google Inc."
  },
  "metabase-info": {
    "databases": [
      "postgres",
      "mysql",
      "bigquery-cloud-sdk"
    ],
    "run-mode": "prod",
    "plan-alias": "pro-self-hosted-yearly",
    "version": {
      "date": "2024-12-21",
      "tag": "v1.52.4",
      "hash": "2f8c560"
    },
    "settings": {
      "report-timezone": "America/Sao_Paulo"
    },
    "hosting-env": "unknown",
    "application-database": "postgres",
    "application-database-details": {
      "database": {
        "name": "PostgreSQL",
        "version": "15.7"
      },
      "jdbc-driver": {
        "name": "PostgreSQL JDBC Driver",
        "version": "42.7.3"
      }
    }
  },
  "system-info": {
    "file.encoding": "UTF-8",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "21.0.5+11-LTS",
    "java.vendor": "Eclipse Adoptium",
    "java.vendor.url": "https://adoptium.net/",
    "java.version": "21.0.5",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "21.0.5+11-LTS",
    "os.name": "Linux",
    "os.version": "6.1.90+",
    "user.language": "en",
    "user.timezone": "GMT"
  }
}

Severity

P1

Additional context

No response

[npfitz]

Hey @fer-batista, I don't seem to be able to re-pro the issue. Can you confirm that you don't have any filters active on your dashboard? They will be applied when exporting data

[fer-batista]

@npfitz Hey! I only have filters on the dashboard. But for these extractions I am not applying them. I tried to create a new dashboard without filters and the issue persists. The file is downloaded with up to 35 columns. Have you tried with +35 columns?

[npfitz]

My apologies! I had read this as 35 rows, not columns.

I just tried testing with a very large dataset though, and I'm able to get +35 columns with both. Just to double check something else, can you confirm that you don't have any columns hidden in the dashcard? The viz settings applied on the dashboard will take effect when exporting from the dashboard view, and may be different from what's on the source question.

[fer-batista]

@npfitz Heey! The root cause of the problem lies in the next column (position 36), which contains a significant number of null values. This caused the export process to fail at that point. I resolved the issue by moving this column to a different position, allowing all columns to be successfully downloaded.

[dragonsahead]

@fer-batista so you can confirm that if a column has nulls then the process will skip those columns? is that correct?

[fer-batista]

@paoliniluis in my case most of the values ​​were null so the download skipped that column and the next one. It worked when I changed the column positions.

[noahmoss]

@fer-batista I'm taking a look, but I'm not able to reproduce on Postgres with 36 columns, where all columns have 20,000 rows and the 36th column has all (or a majority) NULLs.

Any of the following details would help debug:

• What DBMS are you querying?
• How many rows total are you exporting?
• How many columns total are in the table?
• What % of rows in the missing columns are NULL?
• Does this only reproduce for XLSX exports, or also CSV and JSON exports?
• Is there anything else you can share about the types of data in the table?

[cdeweyx]

Closing since we are unable to reproduce with the current information. @fer-batista if you can, please do give us a shout with more info and we'll re-open this to investigate further!

[eccel-d]

hey, hope it's ok to hop on this one since I believe I have figured out what causes the bug.

I bumped into something similar, but instead of a column with null values, it was a column that was set as hidden that was preventing every other column after it from being included on the exported files. I tested it on different "layers", while figuring it out:

On the native query/question that was the initial source for the following layers, there were no hidden columns and all 14 of them showed up on the dowloaded file, if I exported it from there; On the simple question based on the native query, the 10th column was set as hidden, which made any sort of exported file to display only the first 9 columns. After shifting the 10th column to the last position, making it the 14th column, all 13 other columns were included on the exported file. That same behaviour was observed when editing a question on the dashboard level.

[johnswanson]

@eccel-d I've tried to reproduce this as you described and I'm still struggling.

I made a postgres DB with the following table:

create extension if not exists "pgcrypto";
create table my_table (
  id          uuid      primary key default gen_random_uuid(),
  name        text,
  description text,
  amount      numeric(10,2),
  is_active   bool,
  created_at  timestamp default now(),
  updated_at  timestamp,
  code        char(8),
  qty         int,
  rating      real,
  category    text,
  meta        jsonb,
  tags        text[],
  parent_id   uuid
);
-- insert 1000 random rows
insert into my_table (
  name, description, amount, is_active, updated_at, code,
  qty, rating, category, meta, tags, parent_id
)
select
  substring(md5(random()::text),1,10),
  md5(random()::text),
  (random()*1000)::numeric(10,2),
  random()<0.5,
  now() - ((random()*86400)::int || ' seconds')::interval,
  substring(md5(random()::text),1,8),
  (random()*100)::int,
  random()*5,
  -- pick one of three
  (ARRAY['foo','bar','baz'])[(random()*3+1)::int],
  jsonb_build_object('x',random(),'y',substring(md5(random()::text),1,5)),
  array[
    substring(md5(random()::text),1,3),
    substring(md5(random()::text),1,3),
    substring(md5(random()::text),1,3)
  ],
  gen_random_uuid()
from generate_series(1,1000);

Then I generated a question, a native query from this table (just SELECT * FROM my_table;).

Then I made a question with the query builder based on that question. I hid column 10. Then I added it to a dashboard, and clicked "Download results":

I tried with/without the "formatted" checkbox, and in both cases I got all the expected columns.

Is there anything I'm missing in the repro here?

[eccel-d]

Hi @johnswanson The only things I can think of that could potentially make a difference are that the database I use is MySQL and my metabase version is a bit outdated (0.51.6). Otherwise, were you able to get the expected results for the downloaded file in both xls and csv formats?

[Tony-metabase]

@johnswanson on the cloud this was already hit 3 times, I have a feeling its related to a different version a question was created and the upgraded. the recent example if from a 55 version ... But i also cannot replicate just sharing some info will try to get you more information from these accounts

[johnswanson]

I'm un-assigning myself from this for now because I'm going OOO for the next ~2 weeks and haven't been able to replicate it. Don't want to discourage others from picking it up while I'm gone, if it's still unsolved when I come back then I can try to replicate again then.