compass icon indicating copy to clipboard operation
compass copied to clipboard

Published 20 hours ago • verified publisher icon cubefs

[Question]: Can I diagnose abnormal jobs directly from the yarn spark app without collecting scheduling metadata?

Open liangrui1988 opened this issue 10 months ago • 5 comments

Contact Details

[email protected]

What would you like to ask or discuss?

你好！请问，是否可以不依懒调度系统元数据，直接从yarn和spark元数据和日志进行自动触发作业诊断呢？ 在我没有采集调度元数据的情况下，配置了spark和yarn的采集，并且能够实现离线作业的诊断，根据作业id来诊断。 但从compass页面上看不到任何作业信息，应该是compass依懒调度元数据触发自动诊断是吧。 但我们调度系统是自已开发的，目前不想集成进来，先直接诊断yarn spark作业。 这种情况是否可以，从那里触发自动诊断呢？

liangrui1988 avatar Apr 18 '24 03:04 liangrui1988

@liangrui1988 You can refer to the document

nilnon avatar Apr 18 '24 06:04 nilnon

@nilnon

@liangrui1988 你可以参考文档

我的意思不是对一个作业进行离线诊断，而是对所有yarn上的作业进行自动诊断哈，然后再web页面上可以看到每天诊断出异常的作业信息。 这个需要怎么调整或配置？

liangrui1988 avatar Apr 19 '24 02:04 liangrui1988

It is feasible to implement a synchronization module that pulls down completed YARN jobs and invokes diagnostic interfaces, making some customizations on top of the source code.

meijing123 avatar Apr 23 '24 08:04 meijing123

It is feasible to implement a synchronization module that pulls down completed YARN jobs and invokes diagnostic interfaces, making some customizations on top of the source code.

Ok, thank you. Let me see. How do we do that

liangrui1988 avatar Apr 25 '24 07:04 liangrui1988

Excuse me, has it been realized?

xiaomaisuii avatar Sep 26 '24 09:09 xiaomaisuii