resticprofile icon indicating copy to clipboard operation
resticprofile copied to clipboard

JSON Schema

Open jkellerer opened this issue 2 years ago • 3 comments

Implements #103

  • [x] Dictionary of restic commands and options (with descriptions, defaults, type etc.)
  • [ ] Config info API (descriptions, defaults, restic(profile) option, etc.)
  • [x] JSON schema for V2 config
  • [ ] JSON schema for V1 config
  • [ ] Test it
  • [ ] Add command for the functionality

jkellerer avatar Apr 24 '22 16:04 jkellerer

Codecov Report

Base: 71.76% // Head: 75.72% // Increases project coverage by +3.96% :tada:

Coverage data is based on head (54507d2) compared to base (39bb5dd). Patch coverage: 90.41% of modified lines in pull request are covered.

:exclamation: Current head 54507d2 differs from pull request most recent head b924330. Consider uploading reports for the commit b924330 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #107      +/-   ##
==========================================
+ Coverage   71.76%   75.72%   +3.96%     
==========================================
  Files          77       89      +12     
  Lines        7681     9583    +1902     
==========================================
+ Hits         5512     7256    +1744     
- Misses       1947     2069     +122     
- Partials      222      258      +36     
Impacted Files Coverage Δ
schedule/handler.go 100.00% <ø> (ø)
restic/downloader.go 22.73% <22.73%> (ø)
restic/source.go 44.12% <44.12%> (ø)
config/flag.go 89.19% <74.07%> (-1.09%) :arrow_down:
commands.go 56.71% <75.00%> (+1.67%) :arrow_up:
wrapper.go 81.87% <80.90%> (-1.41%) :arrow_down:
config/path.go 85.90% <84.21%> (+4.65%) :arrow_up:
restic/commands.go 89.83% <89.83%> (ø)
restic/files.go 91.30% <91.30%> (ø)
shell/args.go 90.00% <91.67%> (-0.76%) :arrow_down:
... and 28 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

codecov[bot] avatar Apr 24 '22 16:04 codecov[bot]

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 17 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

sonarqubecloud[bot] avatar May 09 '22 08:05 sonarqubecloud[bot]

Just a small update on the next steps:

Will create JSON schema directly from the info API as this is easier to test and implement than trying to use the lib that I introduced before. After that commands and all tests will be implemented.

Scope of this PR will be:

  • JSON schema and command to generate it
  • Command to process a template against the info API (to create a config file reference)
  • Simplify profile.go further, now that info API knows the structure already

Out of scope (to be delivered later):

  • Profile validation
  • More intelligent processing of restic options (only pass options that a command really understands)

jkellerer avatar May 15 '22 20:05 jkellerer

Depends on https://github.com/creativeprojects/go-selfupdate/pull/33

jkellerer avatar Nov 20 '22 16:11 jkellerer

Added "More intelligent processing of restic options" to the scope of this PR (can be toggled in global)

jkellerer avatar Jan 14 '23 17:01 jkellerer

This is getting huge 😄 I suspect you didn't realise it would get out of control when you started it 😛

I'm moving it to 0.21.0 as I'd like to push a new docker image with the new restic in it 👍🏻 (I might want to find a way to push a docker image without doing a build but it doesn't look too easy with goreleaser for now)

creativeprojects avatar Jan 21 '23 23:01 creativeprojects

Yes true, it got a bit out of control :) .. but I'm almost there, just need to restore test coverage and do some final tests with real configurations.

Nevertheless it may still take a few more weeks before it is final, so yes moving it to 0.21 makes sense if you want to release now.

jkellerer avatar Jan 22 '23 10:01 jkellerer

PR is finally implemented. I'd like to test it manually for another week before I label it ready for review.

jkellerer avatar Feb 05 '23 16:02 jkellerer

@creativeprojects , I've left all individual commits for now, but will squash everything together before merging.

Take your time and let me know what you think.

Btw. to get on with testing jsonschema you can:

make generate-jsonschema generate-config-reference      
cd docs
hugo server

And then use VSCode with a new YAML doc:

# yaml-language-server: $schema=http://localhost:1313/resticprofile/jsonschema/config-2.json
...

(Details are in http://localhost:1313/resticprofile/configuration/jsonschema/)

jkellerer avatar Feb 11 '23 15:02 jkellerer

Thanks, it looks awesome!

85 files changed... it's going to take me a while though 😆

creativeprojects avatar Feb 11 '23 17:02 creativeprojects

Well I have to say I'm very pleased with this! There was a lot to do behind the scene to load everything from the restic man pages.

Also I have to admit I haven't gone through everything, mostly the parts that could break the configuration.

I'll have a few follow ups afterwards (I don't want to delay this PR), like the reference page in the documentation which is too big. I'd like to cut it in parts, but we'll do that later.

Having said that, I found 2 things so far.

  1. the profile prom from dev.yaml (where I removed the tags template) is showing an error on the prometheus labels section:
prom:
    force-inactive-lock: true
    initialize: true
    inherit: default
    prometheus-save-to-file: "self.prom"
    prometheus-push: "http://localhost:9091/"
    prometheus-labels:
      - host: "{{ .Hostname }}"
    status-file: /Volumes/RAMDisk/status.json
    backup:
        extended-status: true
        no-error-on-warning: true
        source:
          - "{{ .CurrentDir }}"
Screenshot 2023-02-22 at 21 32 59

The profile runs well though.

Edit: Have I got it wrong all along? the config is like an array of map by using a - in front 🤔

  1. The run-before and run-after requires an array:
run-before-after:
    inherit: default

    snapshots:
        host: true
        run-before: "echo Before snapshots"
        run-after: "echo After snapshots"

There's no error reported in vscode, but running the profile fails:

Screenshot 2023-02-22 at 21 38 33

creativeprojects avatar Feb 22 '23 21:02 creativeprojects

To 1.) Yes as you wrote: - key: value means []map[string]string but we just have a single map not an array. So the json schema should not allow it.

To 2.) Very good catch, again. The schema validation is correct but I missed to pass the correct mapstructure.DecoderConfig for dynamic sections like snapshot, therefore decoding fails. Will fix it.

jkellerer avatar Feb 23 '23 10:02 jkellerer

Right, so I've been using this version for a few days now and it all looks good to me.

I think it's ready to merge 👍🏻

Thanks again for the tremendous work! ❤️

creativeprojects avatar Feb 25 '23 14:02 creativeprojects

Great, I'll squash and merge now. Have used this implementation for several weeks and had no issues (but didn't use the 2 failing features 😄)

jkellerer avatar Feb 25 '23 17:02 jkellerer