Support for blockmap file generation for custom repositories

[baron1405]

I am new to the blockmap file and only came across it because my IntelliJ IDEA 2020.3.2 log file has a number of failed attempts to download a *blockmap.zip file for my plugin from our Enterprise repository. The failure is logged at INFO level but has a stack trace. For example:

com.intellij.util.io.HttpRequests$HttpStatusException: Request failed with status code 404. Status=404, Url=http://someserver/someplugin-15.0.7.zip.blockmap.zip
	at com.intellij.util.io.HttpRequests.throwHttpStatusError(HttpRequests.java:671)
	at com.intellij.util.io.HttpRequests.openConnection(HttpRequests.java:649)
	at com.intellij.util.io.HttpRequests$RequestImpl.getConnection(HttpRequests.java:381)
	at com.intellij.util.io.HttpRequests$RequestImpl.getInputStream(HttpRequests.java:390)
	at com.intellij.ide.plugins.marketplace.MarketplacePluginDownloadService$downloadPluginViaBlockMap$newBlockMap$1.process(MarketplacePluginDownloadService.kt:69)
	at com.intellij.ide.plugins.marketplace.MarketplacePluginDownloadService$downloadPluginViaBlockMap$newBlockMap$1.process(MarketplacePluginDownloadService.kt:23)
	at com.intellij.util.io.HttpRequests.doProcess(HttpRequests.java:541)
	at com.intellij.util.io.HttpRequests.process(HttpRequests.java:523)
	at com.intellij.util.io.HttpRequests$RequestBuilderImpl.connect(HttpRequests.java:355)
	at com.intellij.ide.plugins.marketplace.MarketplacePluginDownloadService.downloadPluginViaBlockMap(MarketplacePluginDownloadService.kt:68)
	at com.intellij.ide.plugins.marketplace.MarketplaceRequests.downloadPluginViaBlockMap(MarketplaceRequests.kt:414)
	at com.intellij.openapi.updateSettings.impl.PluginDownloader.downloadPlugin(PluginDownloader.java:296)
	at com.intellij.openapi.updateSettings.impl.PluginDownloader.prepareToInstallAndLoadDescriptor(PluginDownloader.java:175)
	at com.intellij.openapi.updateSettings.impl.PluginDownloader.prepareToInstallAndLoadDescriptor(PluginDownloader.java:143)
	at com.intellij.openapi.updateSettings.impl.PluginDownloader.prepareToInstall(PluginDownloader.java:139)
	at com.intellij.openapi.updateSettings.impl.UpdateChecker.checkAndPrepareToInstall(UpdateChecker.kt:500)
...

From what I read about blockmap files they are metadata that allow efficient patch updates. Is the gradle-intellij-plugin supposed to generate the file for me to upload to my Enterprise repository?

As always, thanks for your insight and help!

[baron1405]

Hi @zolotov, can you provide any guidance on this?

[zolotov]

Sorry, missed the notification :(

I'm not aware of blockmap files. As far as I remember, @piotrtomiak implemented enterprise repos support, maybe he knows something about the error?

[chrkv]

Hello! At the moment gradle-intellij-plugin does not generate blockmaps for your plugins, but it makes sense to provide this feature for custom repositories.

If you want to try it now then you can save output of following command to file blockmap.json and then zip it to archive called <yourPluginFile>.blockmap.zip and place near your plugin file: com.fasterxml.jackson.databind.ObjectMapper().writeValueAsBytes(BlockMap(<yourPluginFileInputStream>))

[baron1405]

Thanks @zolotov and @chrkv for the info. I will use that Java command. At this point, consider this issue a feature request for adding blockmap support to the gradle-intellij-plugin.

[piotrtomiak]

@zolotov @baron1405 - I've added support for referencing custom repositories in gradle config to build dependencies, but this issue indeed seems to be related to repository generation phase.

[baron1405]

@chrkv Thanks for the information. I have a few questions:

1. When you say <yourPluginFile> does that include the extension? For example, if my plugin is called foo.zip, the blockmap archive should be called foo.zip.blockmap.zip.
2. Is there a URL for the blockmap file of an existing plugin so that I can have a look? I tried to get one for the Scala plugin but it just kept downloading the plugin.
3. I see there is also a hash file that needs to be generated and uploaded. I assume that can be accomplished using the FileHash class. FOr that the naming convention appears to be foo.zip.hash.json. Is that correct?

[chrkv]

1. Yes, if plugin is called foo.zip, the blockmap archive should be called foo.zip.blockmap.zip.
2. For example, latest Scala update is available at https://plugins.jetbrains.com/files/1347/109868/scala-intellij-bin-2020.3.19.zip and it's blockmap file is available at https://plugins.jetbrains.com/files/1347/109868/scala-intellij-bin-2020.3.19.zip.blockmap.zip
3. Yes, you are correct; hash file is also required for blockmap downloads. For the same Scala's update hash file is available at https://plugins.jetbrains.com/files/1347/109868/scala-intellij-bin-2020.3.19.zip.hash.json. You can create it with ObjectMapper().writeValueAsString(FileHash(yourPluginFileInputStream))

[brian-mcnamara]

I was looking at implementing blockmap support into https://github.com/brian-mcnamara/plugin_uploader to solve the warning, however I did come across a bug https://youtrack.jetbrains.com/issue/IDEA-293222/non-dynamic-plugin-installs-do-not-use-blockmap-partial-download which (at least for my plugin, which is not dynamic) means the plugin is still downloaded fully. At least the exception no longer occurs, so I will get around to cleaning up the code and merging it in, but it would be nice to get the bug fixed.

[brian-mcnamara]

blockmap support added to plugin_uploader as part of version 1.3.0