Files are missing.

[kenorb]

Can you commit the files, please? Thanks.

[ryanaslett]

At the conclusion of this blog post: http://bdgregg.blogspot.com/2013/09/dtracetoolkit-0xx-mistakes.html

Learning From Mistakes

I need to:

Get the DTraceToolkit on github.

[brendangregg]

I'm curious to hear why people want this on github. What is it hoped to accomplish? What are peoples desires or needs here?

Yes, I said in the blog post it should be on github, so I already know my own desires. I'm interested to hear yours. :-)

(If the reason is to facilitate contributions: please read point 19 of the post.)

[ryanaslett]

Im not necessarily a fan of githubs Pull Request system, or the githubization of everything, however code on github is granted many advantages (though any publicly hosted repository with a sufficiently modern interface would work- https://gitorious.org, https://bitbucket.org etc).

1. Code review : To be able to inspect the code in a web interface see the code displayed and syntax highlighted without downloading the tarball, unzipping it and opening it up in a CLI editor that may or may not be configured to interpret a D script with proper highlighting and formatting. My favorite editor of choice has limited support for D.
2. Communication and referencing existing code : In talking with somebody about how to put together a proper script, its much easier to send them a hyperlink to a couple lines in a file that demonstrate what they are trying to accomplish vs telling them something like "go download Dtracetoolkit-0.99, untar it, and look in the php folder at the flowtime.d script for an example of capturing function timings in php".
3. History visibility : much of this blog post (http://bdgregg.blogspot.com/2013/09/dtracetoolkit-0xx-mistakes.html) might be evident in the commit history of how it has changed over time. There are many lessons to be learned from somebody elses change history. I'll flip open a blame log on a repo to see if I might find out why something is the way it is.
4. Contributions : It facilitates contributions by changing the topology of the project from a star network to a mesh. So less #19 kinds of contributions and more of the #18 " allowed others to easily participate in testing these scripts, accelerating their development". If somebody is sends a rookie pull request with ill advised changes, its miles easier to check the diffs in a PR, instead of digging through the entire script or having to test it to see if it runs. Additionally the pull request is visible which means that somebody else who isn't a rookie can come in and, see the errors in the PR and address the shortcomings without your involvement - spreads out the load.
5. Distribution, Findability, Signals : If Im looking for a library or some code, stars and forks are a fuzzy metric for understanding a library/toolkits maturity or value. Granted when pre-github powerhouses like this land on github, they dont necessarily garner the same kind of star fork frenzy that something like the tech dujour does: (e.g. https://github.com/dotcloud/docker) But its still a useful metric. If I do a search on github for Dtrace (which is now the first place I look for code), I can see which projects stand out, and which forks are not really worth investigating.

[kenorb]

I'd like to have DTraceToolkit files on GitHub as it's easy to manage hundreds of my favorites repos in one place (e.g. by starring it, managing, backing them up), or searching for the code or files (key: t), forking for own customization (without keeping everything on every laptop/computer by risking lose of data), or creating collection for dtrace scripts by creating repo with submodules to other repos for education purposes and having all dtrace scripts in one place. I don't know even where the official repository for DTraceToolkit is, as I couldn't find it easily, but I could find this one more easily despite of being empty. Found one on SourceForge, but based on the last update and Downloads count doesn't look like official, and on the official pages there are no much links to any repositories.

[brendangregg]

Thanks for the comments; I'm also interested in specifically what people want to do with the DTrace scripts. Run them? On what OSes? And do you expect them to work?

The DTraceToolkit was created as a collection of working scripts for everyone to use, without needing to know DTrace. At the time it only needed to support Solaris 10 FCS. To play it safe, I tried to focus on what I thought was a stable interface -- the syscall provider. That turned out to be untrue, and was much more tied to kernel implementation than I hoped (because of the way it instruments the trap table).

If it helps, don't think of the DTraceToolkit as 230 scripts. Think of it as 230 kernel patches that were mostly written in 2005, for the Solaris kernel.

So do you want to run them today? On what OSes? And do you expect these kernel patches to work?

Is such a project even feasable today?

Or do you want a collaboration ground where performance engineers can share scripts, where there is no expectation that they will work wherever DTrace exists. For such a model, the most useful of the scripts (the core "toolkit") is shipped with the operating system, and maintained by the vendors (eg, as happens with OS X).

[kenorb]

I'd like to learn more about dtrace syntax with live examples how to use it, even they're not working properly for all OSes. It's good to learn directly from some live examples, rather than from tutorials. Mainly for OSX which already has some example dtrace scripts in /usr/bin/*.d (around 28) e.g. to monitor new processes (newproc.d), files (filebyproc.d), but more available then better.

You can already find some examples on GitHub: https://github.com/search?l=d&q=%23%21%2Fusr%2Fsbin%2Fdtrace&ref=searchresults&type=Code but most of them are very simple.

In general I think there is lack of some examples, you can rarely find some good dtrace coding on net. I don't expect them to work.

[ryanaslett]

"they are examples of solving problems (even if they don't work), not ready-to-go tools." - and that is all I would expect from them.

Personally Im less interested in the syscall tracing space as I am in the application level instrumentation. I've been using the OSX dtrace scripts for a while to troubleshoot local issues, and was floored to find out that mysql and php both had instrumentation hooks in them, and additionally there are projects like https://github.com/davepacheco/mod_usdt to add hooks to apache.

While there are profiling tools for PHP that work (xhprof, xdebug/kcachegrind, etc) There isnt really a toolset that can give lateral views of a combination of applications interacting with the system. To be able to profile function call times at the same time as Im seeing the jump to the database is gold coupled with the actual query string, and then to see disk thrashing happen because my key buffers are too small, especially if I were to integrate dTrace into our CI system to pick up on application performance gotchas as part of the build process.

But to address the 'kernel patch' nature of the script - I see them as 230 solutions to specific information gathering problems. Some of those problems don't even even exist in other platforms, or may have solutions in other tools. And many of those solutions are definitely not going to work when going from Solaris 10 FCS to a BSD box, or even to something like https://github.com/dtrace4linux/linux , but they definitely shine a light on whats possible to achieve, even if whats there is only considered production quality on an OS from nine years ago.

"Or do you want a collaboration ground where performance engineers can share scripts, where there is no expectation that they will work wherever DTrace exists" - This, definitely. I've gotten cynical enough that I don't expect any code to work at this point, much less have accurate man pages.

[brendangregg]

Thanks for the comments! To start with, I've published the DTrace book scripts on github, so that they are findable here: https://github.com/brendangregg/DTrace-book-scripts

I think they are the best examples out there. These were written much after I wrote the DTraceToolkit, and I had much more experience by that point, which is reflected in the scripts. I also focused on creating short textbook examples rather than robust tools. They also have much wider observability scope that the DTraceToolkit. And, they've had more help from others to refine (Jim Mauro, etc).

Many of the DTraceToolkit scripts are written as tools, and get hard to read and learn from due to the option processing (in shell) and error checking involved. You'll see from the script length that the DTrace book scripts are neat short examples. They had to be -- we were printing them on paper, and wanted them to mostly fit on 1 page! Other problems with the DTraceToolkit I discussed on that blog post. By the time I wrote the book scripts, I had learned a lot from these mistakes.

I have further ideas for creating a larger repository of examples, but this is a quick start to get more DTrace scripts on github.

[ryanaslett]

What did you just do to my weekend? Super awesome, thank you for doing that!

[diimdeep]

I wish someone would update scripts, shellsnoop for example not work on Yosemite.