output to both git and file

[wiad]

It would be useful for me to be able to store the collected config files both as plain file and in git. This does not seem to be supported, or am i missing something?

The file storage is useful to be able to easily provide the config files as input to other applications and scripts. For example, we have a script for parsing and checking for ACL rules in our switch configs. We could of course remodel this to get the switch configuration via git instead, but it's more convenient to just get it from a filepath.

We really want the git output too, for version control and backup, so it would be nice if we could have both.

[ytti]

It's not supported, but would be quite easy to add. Essentially what we'd need to do to support is, split output by ,\* into an array, essentially in https://github.com/ytti/oxidized/blob/master/lib/oxidized/node.rb#L157 copy the logic from above resolve_input which supports multiple inputs.

Then in https://github.com/ytti/oxidized/blob/master/lib/oxidized/worker.rb#L42 do something like

node.output.new.each do |output|
   # existing code verbatim
end

I think that's all that's needed. But don't have time or motivation to do it or test it, not any time soon either.

In the mean time, if you're not ready to write ruby code, why not just have crontab pull your git repo every 5min, to have plain files?

[wiad]

I'll give it a go and add a PR if I end up with something useful. Thanks for the pointers!

[davama]

@waid I was able to do exactly what you are trying but using the hooks section. Hope it's useful https://github.com/ytti/oxidized/issues/396

[wiad]

@davama Thanks for the tip!

[wiad]

@davama If you have the time it would be very helpful if you could test the patch for multiple outputs (ytti/oxidized#426) that I submitted.

[davama]

@wiad yes i was looking to try it since this will probably be a more elegant solution than the hooks method. will probably have an answer by next week though. It's read-only-friday :-) ill have more time next week Thanks Dave

[davama]

I thought i had replied. Well either way it seems there was a git bug with v0.14.2. Now running v0.14.3. I made the changes to the node{s}.rb and worker.rb files. and attempted to use multiple outputs but failed. Last default takes precedence:

output:
   default: file
   file:
     directory: /home/config/repository/.
   default: git
   git:
    user: config
    email: [email protected]
    repo: /home/config/repository/.git

The git in this instance takes precedence. If i comment the git default line i only get file as my output but no git. The reverse is also true.

output:
   default: git
   git:
    user: config
    email: [email protected]
    repo: /home/config/repository/.git
   #default: file
   file:
     directory: /home/config/repository/.

Like this, only git works. According to the notes i read. The first "default" line will be used by the API. Anywhere else to look?

Thanks

[wiad]

@davama You use it the same as with input, i.e:

default: git, file

In this example git will be used by the api.

[davama]

@wiad That was the issue.

Would it be possible to control how git is initialized? i.e: With these changes the outputs looks like this, which is expected:

~/repository/GroupA/subgroup{a,b,c}/node{a,b,c}/node{a,b,c}_flat_file ~/repository/GroupA/subgroup{a,b,c}/node{a,b,c}.git

This essentially creates two directories of the same node, repo and flat. Granted it's all files and folders but if repo and flat file can be in the same node folder that would be nice. We would like to look like so:

~/repository/GroupA/subgroup{a,b,c}/node{a,b,c}/node_flat_file ~/repository/GroupA/subgroup{a,b,c}/node{a,b,c}/.git

This is why i was using a file as output and the Hooks block to initialize a ".git" directory repo in the node folder post_store. Had a secondary hook for git updates. Note: I also do an update of the git repo description.

events: [post_store]
     cmd: 'cd /home/config/repository/$OX_NODE_GROUP && git init && echo "$OX_NODE_NAME commit" > .git/description'

Hope it's clear.

Thank you for the support.

[wiad]

Glad it works!

Your suggestion regarding how git and file placements are managed feels like a separate feature request once this one is accepted and merged. Since I myself am happy with the current behaviour I probably wont have the time to work at it, though. But maybe yourself or someone else can look into it.

[davama]

Yeah i thought that too. I guess this can be closed. Thanks!

[ytti]

Multiple outputs is supported now.

[wiad]

@ytti how is it implemented? It's not documented in the readme.

[ytti]

crap, my bad. It's not, PR was not merged, and now I can't merge it due to other changes. I'll try to get this done.

[davama]

@wiad I was about to ask that question. Thank you @ytti

[mattschwen]

Hey, is there somewhere where this is documented?

[devicenull]

Multiple outputs would definitely be a nice thing to have

[andreas1o]

Anyone found out how to do this ?

[wiad]

There is a proposed patch (#426) that should give you an idea of how to implement this yourself. May not work with the current code base though. I've been using it for a year now to output both to flat files and to git, but I'm on an older version.

[bonigeek]

I'm testing and does not work:

default: git, file

[tbennett6421]

Is this feature still unsupported?

[andreas1o]

I would like this feature to be added also.

[howardjones]

I just installed yesterday and it looks like this never did happen... just curious though: what is the intended meaning of the 'default' field and listing multiple outputs if you can only ever use one?

[lplazaparis]

I would like this feature to be added also :)

[andreas1o]

Bump =)

[laf]

If you would like to express interest in this feature please use the 👍 in the first post so you aren't spamming comments which people get notified for.

[ewysong]

For anyone wanting a workaround for this, this is what I am using and seems to work well. Create a hook in your config file that runs on post_store to retrieve the most recent copy of the device config from the git repo and store it as a plain text file in a different directory. This way the plain text file is always updated as soon as a config change is committed.

hooks:
  export_git_to_file:
    type: exec
    events: [post_store]
    cmd: 'git archive master --remote /home/oxidized/configs.git/ ${OX_NODE_NAME} | tar -x -C /home/oxidized/configs/'
    async: true
    timeout: 40

The above example is for a git repo at /home/oxidized/configs.git/, and the plain text files will be stored in /home/oxidized/configs/

[ytti]

Why don't you clone it once and 'git pull' on hook? What you're doing seems expensive in comparison.

[ewysong]

I don't know git that well and just cobbled it together. Thought I'd share it just in case someone else found it useful.

Looks like your idea would definitely be the more efficient way to do it.

[FlorianB06]

Thank you @ewysong for sharing, I have been trying the hook trick for days & days without success. because this solution is not documented and many people seems to have the struggle, could you @ytti put somewhere a bit of clear documentation and how to implement ? for instance a full config file would help.

Thank you for your awesome job anyway.

[astenger]

Thank you @ewysong and @ytti for putting me on the right track.

I was able to make it work with a hook and git pull. I am running oxidized using the docker container.

I used the straight forward git output configuration. I also added the hook section for the pull. This hook will fail initially since we have not done the git clone yet.

output:
  default: git
  git:
      user: Oxidized
      email: [email protected]
      repo: "/root/.config/oxidized/configs.git"
hooks:
  export_git_to_file:
    type: exec
    events: [post_store]
    cmd: 'cd /root/.config/oxidized/configs && git pull'
    async: true
    timeout: 40

Then I made sure to get the first devices config downloaded which lets oxidized initialize the git repo.

With the container up, I did the clone from within the container to ensure that the git pull on update can be done from within the container as well.

docker exec -it oxidized /bin/bash
cd /root/.config/oxidized/
git clone configs.git

Hope this helps!

[lucdnb]

I've read already all topics about this but, after all, Oxidized not a dual output of configs yet. I have tried to configure these hooks described here, but none has works... I could have done another solution, but my repository as I currently use (git) files doesn't come in text, so it's not just copying them. Anyone can help me with this?

[davama]

What have been doing for a long time:

hooks:
   first_hook:
     type: exec
     events: [post_store]
     cmd: 'echo "$OX_NODE_NAME commit" > /home/config/oxidized/$OX_NODE_GROUP.git/description'
     timeout: 10
   second_hook:
     type: exec
     events: [post_store]
     cmd: 'mkdir -p /home/config/repository/$OX_NODE_GROUP && curl -X GET http://127.0.0.1:9002/oxidized/node/fetch/$OX_NODE_GROUP/$OX_NODE_NAME > /home/config/repository/$OX_NODE_GROUP/$OX_NODE_NAME 2> /dev/null'

[lucdnb]

Thanks for the answer! It worked, but only with 7 devices ... I didn't quite understand, if it worked with these 7 then in theory the configuration is correct for the others too.

My output is git, and my directory is called "devices.git". Within it, there are other folders and files in which the "objects" folder contains the extracted settings, but with several other subfolders.

In this line: cmd: 'echo "$OX_NODE_NAME commit" > /opt/librenms/.config/oxidized/devices.git/$OX_NODE_GROUP.git/description' Maybe I would have to add some more subdirectory?

[davama]

Not sure how your source file is setup but we group our nodes.

perhaps you are not using groups? In which case you would have to update the cmd accordingly

[lucdnb]

When I sent my last comment, I went back to the tests. I discovered the secret, the line below needs to be false: single_repo: false

After that, all devices are being copied in plain text as I would like. Thank you very much, helped me with this case!

[alditechz]

Hi @lucdnb, I'm new on developing something like this and i want to ask how to download config version files ? i already can enable versioning function and displayed on web gui. but i want to download every devices config update. if i use file as default output the versioning not showing and i can download config file from every host using sftp but the versioning not showing. but if i use git as default output, i can't download the config file from every host.

what should i do ? Thank you

[lucdnb]

Hi, sorry for lately response. Did you solve your problem? Is very simple, use "git" output to have versioning and when you want to download the text files, just follow the steps that we spoke above.