obsidian-dataview
obsidian-dataview copied to clipboard
blacksmithgu
"Burning out" dataviews
I've started using dataviews in my daily notes via the daily note template. One dataview I'm using lists all the notes I edit that day:
list from "some-folder"
where file.mtime.day = date({{date:YYYY-MM-DD}}).day and file.mtime.month = date({{date:YYYY-MM-DD}}).month and file.mtime.year = date({{date:YYYY-MM-DD}}).year
That ends up inserted in my daily note as a dataview that matches any files edited today. It's very useful. However, if I edit a file after that day, the list stops showing it, even though I edited a file on that day. Because I'd love to have a history of the files I've worked on for any given day, I need some way to "burn out" a dataview—effectively replace its code fence with its markdown result.
I can see this being a button that appears on hovering the code fence when in preview mode, or even a command in the command palette that operates on the dataview under the cursor. Even better would be a way to bake this into the dataview so that it happens automatically after a certain threshold, e.g.
list from "some-folder"
where file.mtime.day = date({{date:YYYY-MM-DD}}).day and file.mtime.month = date({{date:YYYY-MM-DD}}).month and file.mtime.year = date({{date:YYYY-MM-DD}}).year
burnout after {{date:YYYY-MM-DD}}
but I have no idea if such a thing is technically feasible, as I haven't dug too deeply into Obsidian's internals. Either way, a way to go from a dataview query to its hardcoded result would be very useful to freeze documents in time.
It's doable, and I've had a few people ask about it for another application (Obsidian Publish). Will add support for it in an upcoming release (likely by having it replace or append the fully rendered HTML for tables, and Markdown for everything else).
I was writing a separate feature request but I think this fits what I'd like. I think the possibility to "permanently expand dataview's queries" in the markdown source it is useful in many scenarios.
- Not all dataview queries needs to be dynamically updated. Sometimes I just want to generate a list/table of notes on the fly. Like in kevboh use case.
- It will make dataview more in agreement with the philosophy of "everything is just markdown" and "future-proof". Users should not worry that in 10 years dataview will be not working/maintained anymore. In any moment I could "make permanent" any query.
- Better integrations with third party apps. For instance, I may use dataview to populate some tables (or whatever new amazing view we will have in the future), and then complete the editing of my work on Ulysses (or other editors). In this case, the ability to copy the "markdown version" of my query would be a must.
So, in short, I fully agree with the need of a feature like this.
(btw, just for completeness, this feature request may be related to #99)
Yeah, I would love for this to exist—keep finding uses for it. @blacksmithgu if you have an idea of how it could trigger, I'd be happy to try and PR it myself—been meaning to dig into the codebase anyway…
This has been on my backlog for a while but I have never managed to get around to it yet - @kevboh if you find the time to take a look at it I'd be happy to help scope it and review it.
I'd love to have this feature and, if possible, be able to use it "automatically" when my notes are committed to Github.
As cool as this plugin is, it's not very helpful for me unless I can use it to publish results (unless there is a way to include rendering from dataview as part of the build step in a js based framework like nextjs/gatsby/sveltekit?)
@blacksmithgu @kevboh I'm about to release a large vault with several dataview queries that I'd love to include. Curious to hear if there's any progress on this feature. Thanks!
I would also like to voice support for dataview to have a "static output" option. This could be particularly useful for use within templates, where the dataview query could be run so that the note results with the output of the query within it. For some use cases (archival), it's in keeping with the spirit of markdown to have note contents that will be readable in any context, not just Obsidian/Dataview, for years to come. In any case, thank you for the incredibly useful tool you're providing for the community. It's extremely helpful.
I think you can combine Templater + DataviewJS to do queries and dynamically generate Markdown tables + lists for this, but first class support is definitely a good idea.
The ways I'm planning on this interaction being possible are via:
- A context menu in preview mode for views that allows you to replace a view with it's output in either (1) fancy dataview output via HTML or (2) as a plain dataview table / list.
- A special view property which Dataview understands to be a plain Markdown output; the query block will remain, but outputs will be given in plain Markdown in a specially marked region directly above or below the codeblock.
Any updates or workaround for this issue?
This is my current workflow:
Lists
- Copy the display mode version of the list.
- Copy into editor as links in URI format.
- Use the wikilinks to mdlinks converter plugin to manually adjust links to be markdown links.
- Find and replace all
app://obsidian.md/and manually change any remaining issues to get normal wikilinks.
Tables
- Open Dev-tools and navigate to the
HTML Object and copy its contents.
- Utilize this online tool: Convert HTML Table to Markdown Table - Table Convert Online to then convert the HTML to a Markdown/HTML table.
- Copy Back into Vault as Markdown or HTML (but losing all wikilink functionality).
It's not ideal but for my use case this functionality is crucial (i.e. need the vault to render properly both on GitHub for navigation as well as show the tables and lists on Obsidian Publish).
Can we revert back to the avirut/obsidian-query2table Plugin for referencing the desired functionality?
I’d love to bump this idea. I do a lot of generating summaries in quarterly / weekly notes… burning in the note to markdown gives the file more permanence and portability.
Thanks for the workaround @jimbrig. Any ideas on a more permanent solution @blacksmithgu ?
In case anyone is interested, this is a working Templater template that uses DataView:
<%*
const dv = this.DataviewAPI
const escapePipe = s => new String(s).replace(/\|/, '\\|') // required for links in Markdown table
/*
You may want to collect your utilities in a central Javascript module below
the Obsidian vault, e.g. in a file lib/utils.js.
The file can be included via
const { escapePipe } = require(app.vault.adapter.basePath + "/lib/utils.js")
*/
%>
| Project | Tasks |
| ------- | ----- |
<%
// Note that dv.table() cannot be used as it creates HTML but we want Markdown.
dv.pages("!#template")
.where(p => p.doctype == 'project') // a custom attribute, specified like "doctype:: project"
.map(p => {
let project = escapePipe(dv.fileLink(p.file.path))
let tasklist = p.file.tasks.map(t => `${t.text} ${escapePipe(t.link)}`).join('<br>')
return `|${project}|${tasklist}|`
})
.join("\n")
%>
Here's my approach of getting files created during two dates, it can be used to replace https://github.com/trydalch/obsidian-daily-activity, which does not have the ability to specify a certain date.
Just like the previous reply, it's an example for anyone who's interested in a work-around before the "Buring out" feature is out.
<%*
/* config */
const IGNORE_DIR_REGEXS = [
/^40 Daily Notes/,
]
/* utils */
const linkTailRegex = /\.md\|[^\]]+/
const removeLinkTail = s => new String(s).replace(linkTailRegex, '')
/* main logic */
const dateFormat = 'yyyy-MM-dd'
const dv = this.DataviewAPI
let startDate = await tp.system.prompt('Start date', dv.date('today').toFormat(dateFormat))
if (!startDate) {
return
}
startDate = dv.date(startDate)
let endDate = await tp.system.prompt('End date (optional)')
console.log('endDate', endDate)
if (endDate) {
endDate = dv.date(endDate)
} else {
endDate = startDate.plus(dv.duration('1 day'))
}
console.log('dv', dv, startDate, endDate)
tR += `Files created at ${startDate.toFormat(dateFormat)}~${endDate.toFormat(dateFormat)}:\n`
const out = dv.pages()
.where(p => {
if (IGNORE_DIR_REGEXS.filter(v => v.test(p.file.path)).length > 0) {
return false
}
return p.file.ctime >= startDate && p.file.ctime < endDate
})
.map(p => {
return `- ${removeLinkTail(p.file.link)}`
})
.join("\n")
tR += out
%>
I used some methods that are not documented in DataView, including date.toFormat, date.plus, dv.duration, there's much more to find in console.log(dv, date)
The result is like:
+1
I have read so much about DataView and I really really want to start using it, but as I want my notes to be software-agnostic this is the one feature I am waiting for.
P.S. I'm an IT professional and I do realise how cliché that sounds, but in this case it is genuinely true 😄
Alright, so I'm finally around to looking at this again. I have two modes which I am planning on supporting directly right away:
- Preview mode: All dataviews will support both "Export to Markdown" and "Freeze as Markdown", which (1) copy markdown for you to paste, and (2) directly replace the query block with markdown. This requires manual action to freeze but at least enables the core functionality.
- Bulk Freezing: Dataview will provide a simple UI for replacing ALL queries in your vault with their static results in one atomic operation - best for if you are migrating away from Obsidian or Dataview but want to keep all of your tables and indices. This will also support the ability to keep the existing Dataview block and just append the result as Markdown after the block.
These two scenarios should unblock the most important use-cases: data archiving, basic view support in publish and GitHub, migrating away from Dataview, and even being able to see important Dataview-based links in the graph. I can also consider implementing "Bulk Freezing" as a simple automatic job that runs at user-defined intervals or on important operations.
I was just searching for this an hour ago and was disappointed not to find anything. I am glad so glad I tried re-searching and finding this posted about 1/2 an hour after my first search. Your plugin is what makes obsidian what it is for so many people and this was the only shortcoming I had found with it. So glad to see you picked up this feature request again!
@blacksmithgu that sounds ideal. Im interested in the interval-based action, specifically if there's a way to narrow it to specific files, folders, or tags. But what you describe here is exciting!
Only applying the freezer to a specific dataview source (i.e., anything you can put in FROM) seems like a good feature.
That sounds excellent @blacksmithgu - especially having the ability to run a simple automatic job on specific intervals would be helpful because some dataviews would be time dependent and it would be great to be able to 'burn them' at specific times.
Only applying the freezer to a specific dataview source (i.e., anything you can put in
FROM) seems like a good feature.
Oh, interesting—I'm talking about the opposite, only applying the freezer to specific destination notes. My use case is in the original ticket; I want to be able to freeze a list of files I've edited in my entire vault (source "/") in a single specific note.
Just a genuine question: what would make this feature hard to implement? You generate the results, and you write it down. The first part is hard, but you have done it. Why would the second part be hard as well?
Oh, interesting—I'm talking about the opposite, only applying the freezer to specific destination notes. My use case is in the original ticket; I want to be able to freeze a list of files I've edited in my entire vault (source "/") in a single specific note.
That is what I mean by "source"; you would spefiy which files you want to freeze via a dataview source query. So if I want to freeze all of the contents of my daily notes, I could do it by the source "daily".
Just a genuine question: what would make this feature hard to implement? You generate the results, and you write it down. The first part is hard, but you have done it. Why would the second part be hard as well?
A simple implementation which just verbatim replaced your query codeblocks with their results would not be that hard, but it is also an inflexible solution. A much better and more general solution is to output the results above/below the codeblock - but then when I run the freezer again, how do I figure out what is plain Markdown that you wrote (or modified), and what is Markdown that Dataview generated? There needs to be some way of indicating "this was autogenerated by Dataview by this specific query and can be safely overwritten with new results".
Why not just using some comments to let users know that this portion will be overwritten in the next run?
- Very interested in this feature, would genuinely be transformative for several of my workflows.
BlacksmithGU, I am watching for this new feature with excitement!
I cannot tell, will I be able to somehow trigger obsidian from outside in order to generate a markdown output from a Dataview computation?
My use case is the generation of markdown (and then ultimately CSV) from Obsidian data for auto upload. I need to keep a public summary computed from my vault current and online. I am hoping I in one way or another I will be able to compute this data from Dataview.
Awesome plugin!!
@oblinger you can already do this with some of the scripts mentioned in this issue. It's not particularly difficult but won't be very portable as you'd have to use node modules to write to a CSV, to my understanding.
Why not just using some comments to let users know that this portion will be overwritten in the next run?
I have seen something similar to this before with content that is generated and will be replaced. Surrounding the generated code in comments immediately before and then immediately after. Really looking forward to seeing something like this implemented! It would certainly simplify some workflows for me!
