Add Telemetry (tracking usage)

[sayedihashimi]

Currently for usage the only data that we have is the download count from the VS gallery. We currently have no clue which templates are getting used. It would be good if we could track this info somehow. I am not sure what all our available options are here. I'm also not sure if anyone would mind if we added this. Having this data would help us prioritize new features as well as understand in which areas we should continue to invest.

Proposal

Here are my initial thoughts.

We should be able to track usage for SideWaffle as well as other template packs created by TemplateBuilder. One way that I can think of to enable this is to create a custom wizard that will hit a tracking URL and pass query parameters with the following info.

1. Package ID for the VSIX containing the template
2. TemplateID if in the .vstemplate
3. Template Name
4. Template type (project template or item template)

We can update the TemplateBuilder build process to inject the custom wizard when the vsix is built. For those who are interested in "opting out" we can add an MSBuild parameter which can be used to disable the functionality (it will be on by default).

Questions

1. Any concerns with tracking this data? If so is the opt out proposal I have good enough?
2. Is there any (hopefully free) services available that can help us track/report on this?

How to implement

We will need to do the following to get this implemented.

• [x] Ensure that all project & item templates have a TemplateID
• [x] Update project & item template template files to have a TemplateID
• [x] Create a custom wizard that will send the data to google analytics
• [x] Figure out how to add the custom wizard to projects/items by default
• [x] Update project & item template template files to include that wizard by default

[codewithtyler]

So just to clarify, when a user chooses to a template you want to pass the info to a web server telling it what template they chose?

[sayedihashimi]

@RandomlyKnighted yes that's the proposal.

[sayedihashimi]

Here is a blog post that has some details about how to add this http://igrali.com/2013/09/04/add-google-analytics-to-your-windows-phone-and-windows-store-apps-in-3-easy-steps/.

[sayedihashimi]

OK so it seems like there is no concerns with us adding telemetry to SideWaffle/TemplateBuilder. I'll start working on this soon. Currently we have no idea what templates are being used. It would be very helpful to have an understanding of which templates are getting used, and which are not. We will have an option to disable the telemetry as well.

My initial thoughts are to use google analytics.

[codewithtyler]

In regards to Step 1, it looks like the "SideWaffle Item Template" item template does not add a TemplateId when it adds the .vstemplate files.

Were you able to find out how to add a custom wizard to project/item templates by default?

[sayedihashimi]

@RandomlyKnighted

it looks like the "SideWaffle Item Template" item template does not add a TemplateId

I just updated them in b6548b29c6d72a207d879575090599aca7ac4630, item and project template template now both have TemplateId.

Were you able to find out how to add a custom wizard to project/item templates by default?

All we have to do is to update the _project.vstemplate.xml or *.vstemplate file to have an additional WizardExtension element (example). In this case instead of adding one by default we just have to update the files in this repo to use this new custom wizard. And when we receive new templates make sure they have the custom wizard in the files.

We also will need to update every template file to have a TemplateId. We can do this with PowerShell pretty easily.

[codewithtyler]

Correct me if I'm wrong, but I'm understanding this as we can add something like the following to each vstemplate file

  <WizardExtension>
    <Assembly>LigerShark.Templates, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null</Assembly>
    <FullClassName>LigerShark.Templates.SideWaffleTelemetryWizard</FullClassName>
  </WizardExtension>

Where the FullClassName element points at a class in some project that we create. Is that right?

[sayedihashimi]

Correct

[codewithtyler]

Does it have to be a specific kind of project? (i.e. Console Application, Class Library, etc.)

[codewithtyler]

@sayedihashimi In the HTML5 UP! Wizard that you created I noticed that that each function contained an argument for a project. How is the project being passed to each of the functions? The reason I ask is because I'm wanting to set up the telemetry wizard so that the TemplateID is the argument.

[codewithtyler]

I've created a function that uses the Google Analytics SDK to track which template is created. There are still three things left that need to be done.

• [x] Somehow get the template ID from the vstemplate file (is there a way to read information from the vstemplate file when the project is created?)
• [x] Add the WizardExtension element to the vstemplate file of all project and item templates
• [x] Testing it to make sure it works as expected

Whenever you go to store the information just reading the Template ID isn't going tell you which template that is unless you look it up. So do we want to store more than just the Template ID in order to make reading the information easier?

[sayedihashimi]

@RandomlyKnighted I think we should send TemplateID and Name. I'll look into how to get those values, hopefully tonight.

[sayedihashimi]

@RandomlyKnighted I looked around to see if I could figure out how to get the TemplateID/Name inside a custom wizard and didn't make much progress. I've emailed a few people who should be able to give me more info. I'll update this when they reply.

[sayedihashimi]

@RandomlyKnighted OK I've heard back. I was wrong in my approach to solve this, evidently you cannot access the .vstemplate metadata from a custom wizard. Instead in the .vstemplate file we can define a Custom parameter for the ID and Name. Then when the custom wizard is invoked we can use that data to track the usage.

I think we should create the following custom params for each template;

• TelemetryID - a new unique guid for each template
• TemplateName - copies the name field from the .vstemplate file

[codewithtyler]

So we replace the TemplateID with the Custom Parameters? If that's the case, then we could probably do that with PowerShell.

[sayedihashimi]

There's no benefit in removing existing IDs. We can just create a script to add the new elements, and the guids can/should be different.

[codewithtyler]

If we create a new guid for this then the old TemplateID would become an identifier for Visual Studio and this one would be an identifier for us, correct?

[sayedihashimi]

Yes. The issue with copying it is that it will get out of sync and there's no benefit to syncing them.

[codewithtyler]

Makes sense. So I guess our next step is to update all the vstemplate files with the TemplateExtension and Custom Parameters. I'm still pretty new to PowerShell so I could use some help with this part.

[sayedihashimi]

See my script to add template id https://github.com/ligershark/side-waffle/blob/master/add-templateid.ps1 that should get you started

[codewithtyler]

Thanks, I was able to get my script to work. Now, I just need to get the TemplateName and TemplateID from the wizard parameters and pass it to my function then I'll be ready to upload a VSIX to Dropbox so we can test it.

[sayedihashimi]

@RandomlyKnighted I looked at the wizard code and the values are not coming in via customParams but instead they are in replacementsDictionary. See the image below.

[codewithtyler]

How were you able to step through the code like that? Anytime I try to debug it using the Experimental Instance it just skips my breakpoint?

Update: I was able to get the values from replacementsDictionary by using $TemplateName$. However, in the MSDN documentation it shows that you use $TemplateName to get the value. Is that the way it used to be? If so, then it would seem that the documentation is outdated.

[sayedihashimi]

@RandomlyKnighted yeah I see that too not sure what causes it. What I did was add System.Diagnostics.Debugger.Launch(); to the line that I wanted to debug and then F5 on that.

Not sure about the docs.

[codewithtyler]

@sayedihashimi did we ever update SideWaffle so that new project and item templates automatically add the Wizard?

[sayedihashimi]

I thought that I added that a while back, are you not seeing it in the result?

[codewithtyler]

Where would it have been added at?

[codewithtyler]

@sayedihashimi Ok, I've done some digging and I know why it was throwing the error

The given key was not present in the dictionary.

It appears that the TemplateName and TemplateID are not being added to the replacementsDictionary. I'm still trying to figure out why they are not being added.

[codewithtyler]

@sayedihashimi When you get some time could you take a look at this and make sure I'm not missing anything. I compared the xUnit project template to the Caliburn.Micro Project Template. The wizard nodes and custom parameter nodes for both projects look identical. Yet, the Caliburn.Micro project goes through the wizard with no problem. When I create a xUnit project it gives me the error mentioned above, which you saw a couple weeks ago when we did our Skype call. I don't see why it's not adding the TemplateName and TemplateID to the replacementsDictionary for some templates.

[sayedihashimi]

@RandomlyKnighted I figured out the exception. The issue is that TemplateBuilder is removing the CustomParameters entry. I filed a new bug for it at https://github.com/ligershark/side-waffle/issues/321. I'll take care of that. I'll also wrap the RunStarted code in a try/catch so that the exception is never raised.