[Bug] [Flink Jar] Download hdfs jar fail when running flink jar task:xxxx is not a valid DFS filename
Search before asking
- [X] I had searched in the issues and found no similar issues.
What happened
When running flink jar task with a resource which is stored in hdfs,taier can not download jar from hdfs successfully.And it makes flink jobs run fail.
What you expected to happen
I have seen the source code and debug,found that the reason for this problem is related to StreamSqlFormatService#generateAddJarSql(String,String) and FileSystemManager#downloadFile(...)
StreamSqlFormatService will replace hdfs://localhost:9820 to hdfs:/localhost:9820,
When I return resource url directly, here comes the second question:
FileSystemManager will remove prefix(hdfs://), which make url to localhost:9820.Of course,HdfsFileManage download fail
How to reproduce
follow below steps can produce this problem:
- upload a flink application jar and choose hdfs storage
- create flink jar task and run.
Anything else
No response
Version
master
Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
@vainhope 请教下大佬为什么要去将//替换掉,是出于什么原因考虑的呢?此外,对于hdfs前缀的去除是不是没有考虑完善,只去除了hdfs://,实际上还可能会有ip/host:port信息没有被去除
flink的udf 资源目前都是仅支持sftp hdfs的支持暂时还没开发 这个问题是前端入口未屏蔽导致的
了解了大佬