Remove duplicate full_name/version items

[johnccfm]

The win_pkg module identifies software by matching on the full_name and version data only. It seems to scan its list of know packages until it finds the first match. If there are duplicates in this "index", then the match you get seems to be indeterminate.

The results of this is that if you have two packages defined with the same full_name and version ... say npp and npp_x86 ... then calling pkg.install npp might fail. doing pkg.list_pkgs might show "npp_x86" as installed. This means that salt will always treat that install as failed, even though the software has installed successfully.

A side-effect of this (for the old pre-2015.8 repo at least) is that you can't have separate packages for x86 and x64 software unless their installed full_name differs, e.g. "Foobar" and "Foobar (64-bit)". For certain cases where I really, really need this I've been forced to use a free MSI editor to change the name.

I've deduplicated the whole repo, removing multiple cases of duplicates.

In some cases where there is only a 32-bit version, the regular and _x86 versions were identical apart from the uninstall path being "C:\Program Files" instead of "C:\Program Files (x86)". This looks like an attempt to fix a known problem, but it won't work. I've just removed the _x86 versions.

In other cases where there are 64-bit and 32-bit versions of the software, but the full_name is identical, I've removed the 32-bit version.

There were a couple of places where the versions I was removing looked to have better URLs or installs flags, so I've retained those.

Without this de-duplication, it seems to me that it's impossible to install many packages in the repo on pre-2015.8 minions.

I realise this is probably low priority now repo-ng and 2015.8 are available, but until bugs in 2015.8 are fixed I've had to fall back to 2015.5.5.

[johnccfm]

For future reference, this was the helper script I used from ipython to detect duplicates. It would be nice if the Salt master detected and warned about these problems.

import yaml
import collections
import os

check_installer = False

def stream_yaml():
    for file in os.listdir('.'):
        if file.endswith('.sls'):
            f = open(file)
            yield (file, yaml.load(f))
            f.close()

y = stream_yaml()
y = list(y)

pkgs = []

for f_name, pkg_grp in y:
    for pkg, pkg_data in pkg_grp.items():
        for ver, ver_data in pkg_data.items():

            new = {'file_name': f_name, 'name': pkg, 'version': ver, 'full_name':
                ver_data['full_name'], 'installer' : ver_data['installer']}
            if check_installer:
                new['key'] = ver_data['full_name'] + "|" + ver + '|' + ver_data['installer']
            else:
                new['key'] = ver_data['full_name'] + "|" + ver

            pkgs.append(new)

keys = [ pkg['key'] for pkg in pkgs ]

duplicates = sorted([ k for k,n in collections.Counter(keys).items() if n > 1 ])

result = {}

for d in duplicates:
    result[d] = [ pkg for pkg in pkgs if pkg['key'] == d ]

[ghost]

@johncfm, I think that everyone has noticed the same thing and you are expected just to deal with it as best you can. I've removed firefox-esr-x86 and salt-minion-x86 from my own copy of the repo for this reason. The problem doesn't fully go away with winrepo-ng. For instance, because of a pull request of mine a new sls file called adobreader-dc-classic.sls has been added because of a new adobe reader versioning system. That won't really cooperate with adobereader-dc.sls because if you have adobe reader classic installed there is nothing to distinguish it from adobe reader continuous besides the version number, but a separate sls file had to be made because the version numbers in "classic" are lower than the ones in "continuous" (pkg.latest and other things would break if just added the new classic entries to the adobereader-dc.sls file). So if you want to actually use the adobereader-dc-classic.sls file you also have to remove adobereader-dc.sls. At least one contributor to salt is indicating he's got some big improvements up his sleave so let's wait and see.

[TheBigBear]

@johnccfm can we close this PR it is no longer relevant, right?