kibana
kibana copied to clipboard
elastic
[Reporting] Full data table content must be contained in the pdf generated
Original comment by @tbragin:
Right now only the preview of the data table shows up in the PDF report, because we essentially take a screenshot of the first page of the table. We should consider how to improve this.
Possibilities to discuss: - Option to export data tables (not saved search tables) to CSV (to accompany the PDF) - Option to export a full data table as a formatted PDF table (limited to N results, IBM Qradar does max of 500)
This is similar to the problem we have with saved search export LINK REDACTED
Original comment by @tbragin:
We discussed this live and decided on the following approach in the immediate future:
- Export a full data table as a formatted PDF table, limited to N results.
- 500 seems like a reasonable default for N.
- For now, do not offer the option of making N configurable.
- Put these data tables at the end of the PDF, after the charts.
Note, this is what Splunk does as well
Reason for not going with the other option:
- Adding an option to export CSV along with PDF adds the complication of dealing with multiple files (we can zip them up, of course)
also would include saved searches in this request: a saved search added to a dashboard will be truncated, even if the number of rows within the search is a relatively low number, like 20.
We discussed this a bit in the Sharing team planning week. Here is some of the notes from that session:
Saved Search and Data Tables: treat them the same.
- Problems:
- Panel is cut off so page 1 might not show all the rows on the first page.
- Table might have 10 pages, but only first page. Even if panel shows all rows of the first page, users want to have all 10 pages of data.
- Possible Solutions
- Show just page 1 table in the pdf, but bundle data table information as .csv
- Show just page 1 table in the pdf but add tables at the bottom of the pdf, maybe as extra pages.
- [NO] auto expand rows so all rows show in a screenshot.
- We don’t want this, it could be way too much information
- [Best option] Offer an option in report PDF:
- [Yes/No]: Include zip of csvs, one for each table. Tablename.csv
- Add dashboard level option, export raw data as csv. rawDataForVis1.csv
@mbarretta
also would include saved searches in this request: a saved search added to a dashboard will be truncated, even if the number of rows within the search is a relatively low number, like 20.
For saved searches, there is a CSV export option in the "Share" menu when viewing the search in Discover.
There's a gap right now on being able to export CSV from a saved search when viewing it in the Dashboard context though, and we need to fix it.
Providing more convenient access to exporting CSV, for saved search and data table, is a general theme where we could see a lot of wins.
- Have a POST "Generation URL" to allow exporting data table (aggregated) visualization data to CSV
- Be able to export CSV from a saved search when viewing it in the Dashboard context
@tsullivan I appreciate that the CSV export allows saved search records to be exported, but not exporting everything seen in a dashboard breaks the "promise".
I've seen many dashboards that are essentially text-only: the preference is for raw data over graphics so that they can see everything in one place and avoid the extra click required to go from a visualization to the underlying data. In those cases, the dashboard is really more like a few Discover views glued together.
So yes, users could export the various data, import them into something like Excel, and then export to PDF from there, but I think it's a fair expectation for them to assume if there are 20 rows of records on a dashboard, they will be visible when that dashboard is exported.
+1 is there any more progress on this or when we will likely see it? We have a dashboard that is data tables with split tables based on how many security attacks have occured. But our PDF reports only show 1 page. Is there any temporary workarounds for this?
@Eniqmatic The App team is actively working on doing a lot of heavy lifting around how the visualizations get their data, and the fruit of that effort will be that we can get data from a saved visualization on the server-side. That will lead into options on how to implement both of the items from the description:
- Option to export data tables (not saved search tables) to CSV (to accompany the PDF)
- Option to export a full data table as a formatted PDF table
There is a workaround to export CSV from an aggregated table visualization. It's from the UI, so unfortunately it would be hard to automate: you can click the Inspector for the visualization to see the underlying data. On that screen are export controls. See: https://www.elastic.co/guide/en/kibana/6.6/vis-inspector.html
@tsullivan Excellent sounds like what I am looking for. Hopefully the guys make good progress with that! Thanks for your response.
Pinging @elastic/kibana-reporting-services (Team:Reporting Services)
Enterprise users of Elastic need to be able to share pdf reports. The current limitation of only generating a pdf image for the first page of tabular data massively undermines the value of Elastic and KIbana in the eyes of these customers. It's fantastic that Kibana will do all the really funky new stuff (Lens etc) but, ultimately, if it cant deliver the basics, enterprise users wont adopt it because it doesnt meet their needs (mainstream adopters vs early adopters).
I fully agree with @mikeh688 Then, I have several users that ask me to have alerting with a Kibana table visualization as CSV attachment. And today, I can't address this need, that is very useful to be able to open data in Excel, and even, simply copy the text of the data table. Formatted data is necessary for me. Particularly to use number format locale. Microsoft Excel expects number format locale from current country. This is important for decimal numbers to us the good decimal separator. Same thing for dates.
Same as above comments..
I ran into this issue with table data on a dashboard. I then tried just a pdf report at the table visualization, same issue.
I then saved the data table filters as a saved search to try a standalone csv report, but cannot do things like GROUP BY/aggregation/etc at the discovery layer that are necessary for this data.
Overall reporting still feels like a third-party addon or external cog, having to setup things like email configuration directly in the config. I would hope things are going in the same direction as alerting is going, with use of "connectors" managed internally and available to use with scheduled reports as well (in addition to fixing the issues with PDF generation on data tables).
With SearchSource from @elastic/kibana-app-services available on the server side, it is now possible to re-create the query and tabify the search results using the reporting framework. The formatting of the data can easily(?) be the same in a report as it is seen in the browser. Once Reporting is able to access the raw table data, it could support CSV or PDF exports for the table data.
I think that is what has been needed all along. It has taken a long time to get here because it depended on having SearchSource available on the server-side.
SearchSource sounds like something internal. So, your post sounds like great news and what does it mean in terms of something being available in release form that enables paged tables to be distributed in pdf reports? thanks Mike
@mikeh688 Hi, thanks for asking!
Feature:Reporting is owned by a larger team of devs that also maintain "SearchSource" which is a configuration scheme for searches that is becoming more and more ubiquitous in Kibana. The larger pool of resources and expertise means this can be worked on in the short term.
I have added this issue to a project board for App Services, and it will be discussed in a planning session so that an owner can take it on.
As you say @tsullivan, most of my Kibana clients ask to be able to have a Data Table content sent by mail every day, in CSV format, as an attachment.
It looks like this can be started now, as #88303 has been merged.
Being able to email daily reports from a lens visualization via watcher and a POST URL would solve an immediate business need for my company.
Hi guys - great to see this moving forwards (and thanks). Is there any idea of a release ETA? thanks
Looks like there is a greater need for exporting CSV from table-based visualizations in an automation-friendly way. Therefore, we can close this issue in favor of https://github.com/elastic/kibana/issues/30982
Can i challenge this being closed please? Customers who create dashboards and want to share (for example, management reports), require data table content to be contained in the pdf generated. The issue at the moment is that only the first page of the table is contained in the pdf output and this is very limiting.
Hi @mikeh688 I agree we still need an issue to provide what you wrote. This issue was a little vague because the description didn't put it as succinctly. I am going to re-open and edit the description to focus on "Customers who create dashboards... require data table content to be contained in the pdf generated"
The other issue is for allowing full CSV export from data table visualizations in an automation-friendly way, which solves the "full export" problem, but it isn't in PDF context.
Thank you for contributing to this issue, however, we are closing this issue due to inactivity as part of a backlog grooming effort. If you believe this feature/bug should still be considered, please reopen with a comment.
Hi @petrklapka, I stand with @mikeh688 in this case and would also oppose this being closed.
This is definitely still a requirement for us, and something we would love to see coming in the future. People are always interested in the dashboard export, and then think "ah..." when we tell them the limitations, especially for the data in their tables.
