求救，我DNS无法运行

[wuchen4588]

问题现象
简要描述问题出现的现象

运行环境

1. 软路由 N3500

2. 电信

重现步骤

1. 114.114.114.114.。

2. google.com。

信息收集

1. 将/var/log/smrtdns.log日志作为附件上传。
2. 如进程异常，请将coredump功能开启，上传coredump信息文件。
   在自定义界面，开启设置->自定义设置->生成coredump配置，重现问题后提交coredump文件 coredump文件在/tmp目录下

2021-01-14 12:59:06,004][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 12:59:06,005][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 12:59:06,005][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 12:59:06,005][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 12:59:11,396][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 12:59:11,396][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 12:59:11,396][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 12:59:11,396][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 12:59:16,866][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 12:59:16,866][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 12:59:16,866][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 12:59:16,866][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 12:59:22,290][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 12:59:22,290][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 12:59:22,290][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 12:59:22,290][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 12:59:27,793][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 12:59:27,793][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 12:59:27,793][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 12:59:27,794][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 12:59:32,409][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 12:59:32,409][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 12:59:32,409][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 12:59:32,409][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 12:59:37,902][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 12:59:37,902][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 12:59:37,902][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 12:59:37,903][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 12:59:43,322][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 12:59:43,322][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 12:59:43,322][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 12:59:43,322][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 12:59:48,775][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 12:59:48,775][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 12:59:48,775][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 12:59:48,775][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:00:00,822][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:00:00,822][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:00:00,823][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:00:00,823][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:00:06,259][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:00:06,259][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:00:06,259][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:00:06,259][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:00:11,710][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:00:11,710][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:00:11,710][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:00:11,710][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:00:17,165][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:00:17,166][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:00:17,166][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:00:17,166][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:00:22,636][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:00:22,636][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:00:22,636][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:00:22,636][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:00:28,104][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:00:28,104][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:00:28,104][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:00:28,104][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:00:56,202][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:00:56,202][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:00:56,202][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:00:56,202][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:01:01,651][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:01:01,651][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:01:01,651][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:01:01,651][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:01:07,134][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:01:07,135][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:01:07,135][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:01:07,135][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:01:12,623][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:01:12,624][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:01:12,624][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:01:12,624][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:01:18,102][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:01:18,102][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:01:18,102][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:01:18,102][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:01:23,550][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:01:23,550][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:01:23,550][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:01:23,550][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:01:35,276][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:01:35,276][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:01:35,276][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 8, Invalid argument [2021-01-14 13:01:35,276][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:01:35,276][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:01:40,747][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:01:40,747][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:01:40,747][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 8, Invalid argument [2021-01-14 13:01:40,747][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:01:40,747][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:01:46,203][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:01:46,203][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:01:46,203][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 8, Invalid argument [2021-01-14 13:01:46,203][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:01:46,203][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:01:51,680][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:01:51,680][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:01:51,680][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 8, Invalid argument [2021-01-14 13:01:51,680][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:01:51,680][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:01:57,150][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:01:57,150][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:01:57,150][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 8, Invalid argument [2021-01-14 13:01:57,150][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:01:57,150][ERROR][ smartdns.c:292 ] start dns server failed. [2021-01-14 13:02:02,600][ERROR][ dns_server.c:3425] bind service :7913 failed, Address in use [2021-01-14 13:02:02,600][ERROR][ dns_server.c:3668] create server socket failed. [2021-01-14 13:02:02,600][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 8, Invalid argument [2021-01-14 13:02:02,600][ERROR][ dns_server.c:252 ] epoll ctl failed, fd = 7, Invalid argument [2021-01-14 13:02:02,600][ERROR][ smartdns.c:292 ] start dns server failed.

[PikuZheng]

看起来你的smartdns配置的端口有其他程序占用了。smartdns想用但没法用

[1phalley]

bind service :7913 failed, Address in use 端口7913被占用了 netstat -ntlp | grep 7913 看看是什么进程，pkill掉或者在网页端永久禁用，或者你换个端口

[chenyajunai]

在Openwrt后台界面我也遇到这个问题，就是你如果修改了配置，但是端口啥都没有变，只是保存应用也提示端口被占用 要么换端口，要么就去进程里结束进程，然后再重新提交保存应用就可以了

[xxxsen]

smartdns 版本号: 1.2021.24

同样碰到这个问题，不清楚是openwrt固件问题还是smartdns的问题。。 出问题的时候，smartdns进程还在，但是日志里面会显示smartdns启动失败，之后dns完全无法解析了，通过dig，每次都是超时

另外除了这个问题，发现smartdns隔一天就会出现大量的dns解析出现SOA问题(请求国外域名的时候)，目前也不确定是怎么产生的。。

[PikuZheng]

.......(恕删) smartdns 版本号: 1.2021.24

同样碰到这个问题，不清楚是openwrt固件问题还是smartdns的问题。。 出问题的时候，smartdns进程还在，但是日志里面会显示smartdns启动失败，之后dns完全无法解析了，通过dig，每次都是超时

另外除了这个问题，发现smartdns隔一天就会出现大量的dns解析出现SOA问题(请求国外域名的时候)，目前也不确定是怎么产生的。。

图1看显然是6053端口被占用了，smartdns起不来。但看图2图3，怀疑你在试图同时运行两个smartdns。 另外版本号不对，请先试试最新的版本看有没有改善。 soa可能和缓存有关，是旧版的已知bug。也可能是其他服务抢占了dns请求，也可能是上游没有得到有效的结果（比如上游查询结果都在黑名单

[xxxsen]

刚刚又试了下，发现有可能是openwrt的问题，保持启用状态然后在webui上进行保存，发现并不会先杀进程后重启，而是直接启动了smartdns进程。。然后日志里面又刷了那个服务启动失败的错误。。但是那个查询dns无响应是确实存在的问题，开了debug日志，手动发了dns包后日志也没打出来。。

版本号确实是通过-v生成的。。。

第三个，我这边试下新版本，因为我这边默认的dns都是海外dns，走tls/https，所以不大可能是黑名单结果。。

[PikuZheng]

刚刚又试了下，发现有可能是openwrt的问题，保持启用状态然后在webui上进行保存，发现并不会先杀进程后重启，而是直接启动了smartdns进程。。然后日志里面又刷了那个服务启动失败的错误。。但是那个查询dns无响应是确实存在的问题，开了debug日志，手动发了dns包后日志也没打出来。。

版本号确实是通过-v生成的。。。

第三个，我这边试下新版本，因为我这边默认的dns都是海外dns，走tls/https，所以不大可能是黑名单结果。。

BusyBox v1.30.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 19.07.8, r11364-ef56c85848
 -----------------------------------------------------
root@OpenWrt:~# smartdns -v
smartdns 1.2021.11.15-1500
root@OpenWrt:~#

[xxxsen]

使用新版后, 问题再次出现。。。

问题跟之前一样，smartdns未退出，但是新的smartdns又继续启动，导致启动失败，同时原先的smartdns看着是处于挂起状态，对dns查询请求无反应。

出现的原因可能是宽带重拨号有关，在异常的时间点附近，宽带出现重连的。

[xxxsen]

补充下我这边的配置

• /var/etc/smartdns/smartdns.conf

server-name smartdns
bind :6053
dualstack-ip-selection yes
prefetch-domain yes
serve-expired yes
cache-size 500000
rr-ttl 3600
rr-ttl-min 5
log-size 64K
log-num 1
log-level error
conf-file /etc/smartdns/address.conf
conf-file /etc/smartdns/blacklist-ip.conf
conf-file /etc/smartdns/custom.conf

• /etc/smartdns/address.conf

nameserver /sz.aaaa.bbb.com/bootstrap

• /etc/smartdns/custom.conf

log-level debug
log-size 128k
log-file /var/log/smartdns.log
log-num 2
cache-size 500000
prefetch-domain yes
serve-expired yes
cache-persist no
speed-check-mode tcp:80,ping
conf-file /etc/smartdns/smartdns-domains.china.conf
server-https https://dns-unfiltered.adguard.com/dns-query
server-tls dns-unfiltered.adguard.com
server-https https://dns.cloudflare.com/dns-query
server-tls 1dot1dot1dot1.cloudflare-dns.com
server-https https://1.0.0.1/dns-query
server-https https://1.1.1.1/dns-query
server-https https://8.8.4.4/dns-query
server 223.5.5.5 -group bootstrap -no-cache -exclude-default-group
server 223.6.6.6 -group bootstrap -no-cache -exclude-default-group
server 114.114.114.114 -group bootstrap -no-cache -exclude-default-group

另外上面的截图的3:49不准，实际情况就是在3:44左右宽带重连的，只是ddns更新脚本没那么及时。。

[xxxsen]

gdb堆栈信息, 打印堆栈的时候发现一直在循环同一个栈帧, 看不到栈底信息, 但是最终的调用是调用到__lock 函数，然后程序就挂着了。。

[xxxsen]

最后还有一个，处于这种状态的进程无法通过sigterm 杀死。。

[xxxsen]

https://cmd.sendev.cc/api/share/getshare?code=O0Peo80lr0_SsPJaBAmIFTHZt3UcN8wXaWOsPwVLADCSCrd7kQwmOesvxg015VcqmzdpS9u_HmkKGDVeGn0e8H7QvAgUPe_PB04iuybVTcfGc0Vco_UYEKnUGrCMo9sryhI_wSqFSU289w&key=hello

最后，补一个core文件下载地址。。 对应的机器架构: aarch64, 版本: smartdns 1.2021.08.27-1923

[PikuZheng]

试试修改 /etc/init.d/smartdns 的 start_service()

start_service()
{
        kill -9 $(ps |grep smartdns |grep -v grep |awk '{print $1}')
	config_load "smartdns"
	config_foreach load_service "smartdns"
}

另外试一下如果不开第二dns服务 是不是就正常了

[xxxsen]

pr

强杀进程可以解决问题，但是没搞错的话，smartdns本身有一些逻辑是要在退出前处理的，代码里面的sigterm逻辑，所以最好的方式还是要有2步逻辑，sigterm后检查进程的退出情况，如果等10s后，进程还在，之后才使用sigkill杀死。

[xxxsen]

补了个检查脚本。。

wait_exit_and_try_forcekill()
{
        inst=$(ps -ef | grep smartdns | grep -v grep| grep '.conf' | awk '{print $1}')
        kill -15 $inst > /dev/null 2>&1
        for i in $(seq 10)
        do
                cnt=$(ps -ef | grep smartdns | grep -v grep| grep '.conf' | wc -l)
                if [ "$cnt" = "0" ]; then
                        echo "already exist, skip force kill..."
                        return
                fi
                sleep 1
        done
        echo "force kill smartdns inst:$inst"
        kill -9 $inst
        sleep 1
}

start_service()
{
        wait_exit_and_try_forcekill
        config_load "smartdns"
        config_foreach load_service "smartdns"
}

[pymumu]

最新版本验证，有问题reopen issue