octocatalog-diff icon indicating copy to clipboard operation
octocatalog-diff copied to clipboard
verified publisher icon github

Catalogs 'from-puppetdb' contain differences to catalogs from puppetserver

Open alexjfisher opened this issue 8 years ago • 2 comments

Hi! I'm just starting out with the tool and have been trying to use the --from-puppetdb option. (Should this option be documented here?)

octocatalog-diff  -n node1.example.com \
                  --from-puppetdb \
                  -t production \
                  --to-puppet-master puppet.example.com

In the above example, the node is already in the production environment and running puppet. I would therefore expect there to be no differences. Unfortunately, I get quite a few. I think I could filter them out with some ignore options, but I was planning to use the tool in a CI setup (seeing which nodes will be affected by a change), so I don't think this would be practical.

On my servers I'm using CentOS 7 with... puppetserver-2.7.2 puppetdb-4.4.0 (with postgresql 9.6) puppet-agent-1.10.4

I'm running octocatalog-diff 1.4.0 on ruby 2.4.1 via rvm on ubuntu.

Many thanks, Alex

  Apache::Listen[443] =>
   parameters =>
     alias =>
      - ["443"]
   title =>
    - "443"
    + 443
*******************************************
  File[/etc/httpd/conf.d/ssl.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.d/ssl.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/access_compat.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/access_compat.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/actions.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/actions.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/alias.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/alias.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/alias.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/alias.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/auth_basic.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/auth_basic.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/auth_digest.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/auth_digest.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authn_anon.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authn_anon.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authn_core.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authn_core.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authn_dbm.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authn_dbm.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authn_file.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authn_file.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authnz_ldap.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authnz_ldap.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/authnz_ldap.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authnz_ldap.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authz_core.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authz_core.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authz_dbm.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authz_dbm.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authz_groupfile.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authz_groupfile.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authz_host.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authz_host.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authz_owner.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authz_owner.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/authz_user.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/authz_user.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/autoindex.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/autoindex.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/autoindex.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/autoindex.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/cache.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/cache.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/cgi.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/cgi.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/dav.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/dav.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/dav_fs.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/dav_fs.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/dav_fs.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/dav_fs.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/deflate.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/deflate.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/deflate.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/deflate.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/dir.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/dir.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/dir.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/dir.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/env.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/env.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/expires.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/expires.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/ext_filter.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/ext_filter.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/filter.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/filter.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/include.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/include.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/ldap.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/ldap.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/ldap.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/ldap.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/log_config.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/log_config.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/logio.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/logio.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/mime.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/mime.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/mime.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/mime.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/mime_magic.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/mime_magic.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/mime_magic.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/mime_magic.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/negotiation.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/negotiation.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/negotiation.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/negotiation.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/proxy.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/proxy.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/proxy.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/proxy.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/proxy_http.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/proxy_http.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/rewrite.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/rewrite.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/setenvif.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/setenvif.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/setenvif.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/setenvif.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/socache_shmcb.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/socache_shmcb.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/speling.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/speling.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/ssl.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/ssl.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/substitute.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/substitute.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/suexec.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/suexec.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/systemd.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/systemd.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/unixd.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/unixd.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/usertrack.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/usertrack.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/version.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/version.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/vhost_alias.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/vhost_alias.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/wsgi.conf] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/wsgi.conf"]
*******************************************
  File[/etc/httpd/conf.modules.d/wsgi.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/wsgi.load"]
*******************************************
  File[/etc/httpd/conf.modules.d/xsendfile.load] =>
   parameters =>
     alias =>
      - ["/etc/httpd/conf.modules.d/xsendfile.load"]
*******************************************
  File[/etc/pulp/server.conf] =>
   parameters =>
     content =>
      @@ -35,5 +35,5 @@
       #                    the ca_cert parameter must point to a file of CA certificates used to
       #                    validate the connection.
      -# ca_path:           The ca_certs file contains a set of concatenated “certification authority”
      +# ca_path:           The ca_certs file contains a set of concatenated �certification authority�
       #                    certificates, which are used to validate certificates passed from the other end
       #                    of the connection.
*******************************************
  File[/etc/pulp/vhosts80] =>
   parameters =>
     alias =>
      - ["/etc/pulp/vhosts80"]
*******************************************
  Service[mongodb] =>
   parameters =>
     alias =>
      - ["mongod"]
*******************************************
  Service[ntp] =>
   parameters =>
     alias =>
      - ["ntpd"]
*******************************************
  Service[squid3_service] =>
   parameters =>
     alias =>
      - ["squid"]
*******************************************
  Stage[main] =>
   parameters =>
     alias =>
      - ["main"]
*******************************************

alexjfisher avatar Aug 04 '17 08:08 alexjfisher

Perhaps getting the from catalog from puppetdb can't ever work out that well. :(

https://docs.puppet.com/puppetdb/4.4/api/wire_format/catalog_format_v9.html

PuppetDB receives catalogs from Puppet masters in the following wire format. This format is subtly different from the internal format used by Puppet, so catalogs are converted by the PuppetDB catalog terminus before they are sent.

alexjfisher avatar Aug 04 '17 08:08 alexjfisher

I think I ran into something similar this while attempting to diff a from-catalog right from puppetdb 4.4 with a to-catalog downloaded from puppetdb 5.1 and living in a file under /tmp.

octocatalog-diff --from-puppetdb --to-catalog /tmp/5_myhost.json --hostname myhost.me.com

BUG: catalog has no data::resources or ::resources array. Please report this.

I notice that in both versions of the puppetdb catalog "resources" is a hash (resources[data] is an array), but in the version used by the puppet agent for the same server "resources" is an array.

christopherwood avatar Nov 02 '17 21:11 christopherwood