n3dr icon indicating copy to clipboard operation
n3dr copied to clipboard
verified publisher icon 030

Logging changes (fine tuning)

Open 3eka opened this issue 3 years ago • 8 comments

What would you like to be added: I would like to have an option to fine tune logging (like: INFO, WARN, ERROR, etc -- with NONE option too). Or, some errors I want to see on the screen (like ERROR), other can go into file (like INFO ones)

Why is this needed: Because, it is really annoying. Few examples:

  • This is output (on the screen) when --basePathPrefix switch is used
++++++++++++++++++INFO[0240]/n3dr/internal/pkg/connection/connection.go:23 github.com/030/n3dr/internal/pkg/connection.(*Nexus3).Client() adding 'nexus' as a prefix to the basePath
*********++++++++++INFO[0240]/n3dr/internal/pkg/connection/connection.go:23 github.com/030/n3dr/internal/pkg/connection.(*Nexus3).Client() adding 'nexus' as a prefix to the basePath
*INFO[0240]/n3dr/internal/pkg/connection/connection.go:23 github.com/030/n3dr/internal/pkg/connection.(*Nexus3).Client() adding 'nexus' as a prefix to the basePath
*******+++++++++++++*++**INFO[0241]/n3dr/internal/pkg/connection/connection.go:23 github.com/030/n3dr/internal/pkg/connection.(*Nexus3).Client() adding 'nexus' as a prefix to the basePath
++++++INFO[0241]/n3dr/internal/pkg/connection/connection.go:23 github.com/030/n3dr/internal/pkg/connection.(*Nexus3).Client() adding 'nexus' as a prefix to the basePath
*********++++++++++INFO[0241]/n3dr/internal/pkg/connection/connection.go:23 github.com/030/n3dr/internal/pkg/connection.(*Nexus3).Client() adding 'nexus' as a prefix to the basePath
*INFO[0241]/n3dr/internal/pkg/connection/connection.go:23 github.com/030/n3dr/internal/pkg/connection.(*Nexus3).Client() adding 'nexus' as a prefix to the basePath
...

As you might see n3dr keeps repeating following, every time when it advances to the next item, which make it hard to follow:

  • INFO[xxxx]/n3dr/internal/pkg/connection/connection.go:23 github.com/030/n3dr/internal/pkg/connection.(*Nexus3).Client() adding 'nexus' as a prefix to the basePath
:; n3dr --version
n3dr version 6.7.3
:; go version
go version go1.19.1 illumos/amd64

3eka avatar Sep 28 '22 11:09 3eka

6.8.0

030 avatar Oct 30 '22 12:10 030

@3eka Did you try the logging option?

030 avatar Nov 04 '22 14:11 030

Tests done with version 6.8.1...

:; n3dr --help
...
      --logFile                       write the log to syslog
      --logLevel string               change the log level
...

Q1: Why --logFile goes to syslog (I would expect options, like syslog or /path/log_file_name.log)? Q2: Would be nice to display options for --logLevel (had to dig into code to find that info, debug and trace are allowed options)

  1. Using --basePathPrefix and --logLevel=info 
INFO[2022-11-04T16:15:36+01:00]/n3dr/cmd/n3dr/root.go:80 main.n3drHiddenHome() n3drHomeDir: '/export/home/predrag_zecevic/.n3dr'
INFO 2022/11/04 16:15:36 trying to resolve absolute path path=/export/home/predrag_zecevic/.n3dr
INFO 2022/11/04 16:15:36 adding path to search paths path=/export/home/predrag_zecevic/.n3dr
INFO 2022/11/04 16:15:36 attempting to read in config file
INFO 2022/11/04 16:15:36 searching for config in paths paths=[/export/home/predrag_zecevic/.n3dr]
INFO[2022-11-04T16:15:36+01:00]/n3dr/cmd/n3dr/root.go:202 main.parseConfig() Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml'
INFO[2022-11-04T16:15:36+01:00]/n3dr/internal/app/n3dr/artifactsv2/download.go:212 github.com/030/n3dr/internal/app/n3dr/artifactsv2.(*Nexus3).Backup() backing up 'npm_group', 'group', npm
...
INFO[2022-11-04T16:25:10+01:00]/n3dr/internal/app/n3dr/artifactsv2/download.go:212 github.com/030/n3dr/internal/app/n3dr/artifactsv2.(*Nexus3).Backup() backing up 'test-raw', 'hosted', raw
*******%%++++++++++++++++++++++++++++++%+*+++++++*+**?*******?*??*?*?????+++????????+++++++++++++++++

Timestamp output is not consolidated, but all seems OK (except legend what all those characters, displayed during backup means)

  1. Backup of second Nexus instance (no prefix) with --logLevel=info 
INFO[2022-11-04T16:30:10+01:00]/n3dr/cmd/n3dr/root.go:80 main.n3drHiddenHome() n3drHomeDir: '/export/home/predrag_zecevic/.n3dr'
INFO 2022/11/04 16:30:10 trying to resolve absolute path path=/export/home/predrag_zecevic/.n3dr
INFO 2022/11/04 16:30:10 adding path to search paths path=/export/home/predrag_zecevic/.n3dr
INFO 2022/11/04 16:30:10 attempting to read in config file
INFO 2022/11/04 16:30:10 searching for config in paths paths=[/export/home/predrag_zecevic/.n3dr]
INFO[2022-11-04T16:30:10+01:00]/n3dr/cmd/n3dr/root.go:202 main.parseConfig() Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml'
INFO[2022-11-04T16:30:10+01:00]/n3dr/internal/app/n3dr/artifactsv2/download.go:212 github.com/030/n3dr/internal/app/n3dr/artifactsv2.(*Nexus3).Backup() backing up 'my-yum', 'hosted', yum
INFO[2022-11-04T16:30:10+01:00]/n3dr/internal/app/n3dr/artifactsv2/download.go:212 github.com/030/n3dr/internal/app/n3dr/artifactsv2.(*Nexus3).Backup() backing up 'nuget-group', 'group', nuget
INFO[2022-11-04T16:30:10+01:00]/n3dr/internal/app/n3dr/artifactsv2/download.go:212 github.com/030/n3dr/internal/app/n3dr/artifactsv2.(*Nexus3).Backup() backing up 'maven-snapshots', 'hosted', maven2
INFO[2022-11-04T16:30:10+01:00]/n3dr/internal/app/n3dr/artifactsv2/download.go:212 github.com/030/n3dr/internal/app/n3dr/artifactsv2.(*Nexus3).Backup() backing up 'maven-central', 'proxy', maven2
INFO[2022-11-04T16:30:10+01:00]/n3dr/internal/app/n3dr/artifactsv2/download.go:212 github.com/030/n3dr/internal/app/n3dr/artifactsv2.(*Nexus3).Backup() backing up 'nuget.org-proxy', 'proxy', nuget
INFO[2022-11-04T16:30:10+01:00]/n3dr/internal/app/n3dr/artifactsv2/download.go:212 github.com/030/n3dr/internal/app/n3dr/artifactsv2.(*Nexus3).Backup() backing up 'maven-releases', 'hosted', maven2
INFO[2022-11-04T16:30:10+01:00]/n3dr/internal/app/n3dr/artifactsv2/download.go:212 github.com/030/n3dr/internal/app/n3dr/artifactsv2.(*Nexus3).Backup() backing up 'my-docker', 'hosted', docker
FATA[2022-11-04T16:30:10+01:00]/n3dr/cmd/n3dr/repositoriesV2.go:70 main.glob..func7() parse ":0/v2/_catalog": missing protocol scheme

Died? But logging is OK (same notice as for above attempt is true)

There is no difference between debug and trace log levels. I have few observations: A) I would expect in either debug or trace log level explanation where and why it has died in more details (for example, I am not sure if it has died on my-docker backup or trying to fetch info for next one (in my case ``my-raw` repository) B) Specifying either log level (debug or trace) following is displayed on screen

DEBU[2022-11-04T16:31:55+01:00]/export/home/predrag_zecevic/go/pkg/mod/github.com/030/[email protected]/internal/pkg/http/http.go:20 github.com/030/p2iwd/internal/pkg/http.(*Auth).RequestAndResponse() Auth: '&{Accept  GET PASSWORD_SHOWN_HERE! :0/v2/_catalog predrag_zecevic}'
FATA[2022-11-04T16:31:55+01:00]/n3dr/cmd/n3dr/repositoriesV2.go:70 main.glob..func7() parse ":0/v2/_catalog": missing protocol scheme

password is visible, should be masked.

Thanks for update. Best regards.

P.S. It seems that I still have 2 problems: Case 1) - it dies often after some period if running time w/o finishing job (maybe Nexus kicks it out, because it thinks someone is DDOS-ing?). Not so important, because if I repeat it several times, it finishes Case 2) - seems raw repository causes error "missing protocol schema"

This is my config file:

:; cat /export/home/predrag_zecevic/.n3dr/config.yml
removeLocalFile: true

3eka avatar Nov 04 '22 15:11 3eka

@3eka Could you create another issue to address "the raw repository and FATA[2022-11-04T16:30:10+01:00]/n3dr/cmd/n3dr/repositoriesV2.go:70 main.glob..func7() parse ":0/v2/_catalog": missing protocol scheme" issue?

030 avatar Dec 17 '22 17:12 030

Example: n3dr repositoriesV2 --backup -u some-user -p some-pass -n localhost:9000 --https=false --directory-prefix /tmp/some-dir --dockerPort 9001 --dockerHost http://localhost --logFile some-file.log --logLevel trace --syslog

030 avatar Dec 17 '22 17:12 030

7.0.0

030 avatar Dec 17 '22 22:12 030

Sorry, I have missed your latest update completely.

Let us focus on logging:

  • if I choose logFile and logLevel, I expected that everything is logged there
  • in reality, tool still logs to stdout

Thanks and regards

3eka avatar Jan 04 '23 15:01 3eka

HI @030 ,

can you describe a bit logging logic?

I am expecting following:

  • When --logFile=/path/to/log/file is specified, ALL output should end there. Right now, tool sends to stdout (even with --logLevel=none settings). This is when I was running backup, then Nexus service restart has stopped it, and I have started same command again:
INFO[0000] n3drHomeDir: '/export/home/predrag_zecevic/.n3dr' 
INFO[0000] Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml' 
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++....

Rest is sent to log file

  • If I stop n3dr and start again, with same logging options (e.g. logging to the same file), new content is added to the existing file (I would expect overwrite -- or another switch which can be used for automating this, like --logMethod={new,overwrite})

I still have to check all logging level options Regards.

3eka avatar Jan 05 '23 11:01 3eka