Taier icon indicating copy to clipboard operation
Taier copied to clipboard

[Bug] [Flink Jar] Download hdfs jar fail when running flink jar task:xxxx is not a valid DFS filename

Open weaksloth opened this issue 2 years ago • 3 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

When running flink jar task with a resource which is stored in hdfs,taier can not download jar from hdfs successfully.And it makes flink jobs run fail. 图片

What you expected to happen

I have seen the source code and debug,found that the reason for this problem is related to StreamSqlFormatService#generateAddJarSql(String,String) and FileSystemManager#downloadFile(...)

StreamSqlFormatService will replace hdfs://localhost:9820 to hdfs:/localhost:9820,

When I return resource url directly, here comes the second question: FileSystemManager will remove prefix(hdfs://), which make url to localhost:9820.Of course,HdfsFileManage download fail

How to reproduce

follow below steps can produce this problem:

  1. upload a flink application jar and choose hdfs storage resource-info
  2. create flink jar task and run.

Anything else

No response

Version

master

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

weaksloth avatar Jul 04 '23 10:07 weaksloth

@vainhope 请教下大佬为什么要去将//替换掉,是出于什么原因考虑的呢?此外,对于hdfs前缀的去除是不是没有考虑完善,只去除了hdfs://,实际上还可能会有ip/host:port信息没有被去除

weaksloth avatar Jul 04 '23 10:07 weaksloth

flink的udf 资源目前都是仅支持sftp hdfs的支持暂时还没开发 这个问题是前端入口未屏蔽导致的

vainhope avatar Jul 04 '23 11:07 vainhope

了解了大佬

weaksloth avatar Jul 04 '23 14:07 weaksloth