Add ability to change sampling rate

[petethepig]

Currently it's hardcoded at 100 Hz.

Multiple people asked for this in both Slack and here on Github, e.g #153

[petethepig]

I’m thinking, when people run pyroscope exec python they should be able to set sample rate, e.g pyroscope exec -sample-rate 50 python

Same with the go profiler:

profiler.Start(profiler.Config{
    ApplicationName: "simple.golang.app",
    ServerAddress:   "http://pyroscope:4040", // this will run inside docker-compose, hence `pyroscope` for hostname
    SampleRate:      50,
})

[qzaidi]

Using the custom profiler has somehow resulted in all of the cpu profiles vanishing from the display. Memory profiling data does show up.

[alonlong]

can u help to show more details about the problem?

[qzaidi]

changed sampling to 20 from 100 hz as it seems profiler had increased cpu usage significantly - this switched it to the custom profiler internally I guess, and now only alloc and inuse traces are shown, no cpu traces are present.

[qzaidi]

This issue happens always when custom profiler is used. In both cases, the agent seems to be sending the cpu samples correctly to server, but the server won't display anything except when a sample rate of 100 hz is used.

Here's some example data that is being sent when sample rate is 20

b'\x00\x00\x03\x90\x01github.com/pyroscope-io/pyroscope/pkg/agent.(*ProfileSession).takeSnapshots;github.com/pyroscope-io/pyroscope/pkg/agent/gospy.(*GoSpy).Snapshot;\x00\x02\xb3\x03github.com/pyroscope-io/pyroscope/pkg/agent/gospy.getHeapProfile;github.com/pyroscope-io/pyroscope/pkg/convert.ParsePprof;google.golang.org/protobuf/proto.Unmarshal;google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal;google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal;google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer;google.golang.org/protobuf/internal/impl.consumeStringSliceValidateUTF8\x00\x00\xb1\x01runtime/pprof.writeHeap;runtime/pprof.writeHeapInternal;runtime/pprof.writeHeapProto;runtime/pprof.(*profileBuilder).build;compress/gzip.(*Writer).Write;compress/flate.NewWriter\x00\x01";compress/flate.(compressor).init\x00\x00\x0bnet/http.(\x00\x02\xbe\x01Transport).dialConnFor;net/http.(*Transport).dialConn;net/http.(*Transport).dial;net.(*Dialer).DialContext;net.(*Resolver).resolveAddrList;net.(*Resolver).internetAddrList;net.filterAddrList\x00\x00\xcc\x04persistConn).addTLS.func2;crypto/tls.(*Conn).Handshake;crypto/tls.(*Conn).clientHandshake;crypto/tls.(*clientHandshakeState).handshake;crypto/tls.(*clientHandshakeState).doFullHandshake;crypto/tls.(*Conn).verifyServerCertificate;sync.(*Once).Do;sync.(*Once).doSlow;crypto/x509.initSystemRoots;crypto/x509.loadSystemRoots;crypto/x509.(*CertPool).AppendCertsFromPEM;encoding/asn1.Unmarshal;encoding/asn1.UnmarshalWithParams;encoding/asn1.parseField;encoding/asn1.parseField;encoding/asn1.parseField;encoding/asn1.parseField;encoding/asn1.parseField;reflect.MakeSlice;reflect.unsafe_NewArray\x00\x00\xbe\x04runtime.main;main.main;github.com/spf13/cobra.(*Command).Execute;github.com/spf13/cobra.(*Command).ExecuteC;github.com/spf13/cobra.(*Command).execute;gitlab.snapp.ir/simulation/locsim/cmd.runsim;gitlab.snapp.ir/simulation/locsim/app.(*Simulator).StartProfiler;github.com/pyroscope-io/pyroscope/pkg/agent/profiler.Start;github.com/pyroscope-io/pyroscope/pkg/agent.(*ProfileSession).Start;github.com/pyroscope-io/pyroscope/pkg/agent/gospy.Start;github.com/pyroscope-io/pyroscope/pkg/agent/gospy.startCPUProfile;github.com/pyroscope-io/pyroscope/pkg/agent/pprof.StartCPUProfile\x98\x82J\x00'

[alonlong]

Got it, will check it, thank u

[petethepig]

@qzaidi What's your go version / os? Can you send us the output of running go env?

[qzaidi]

This is my local go env (go version 1.15.3 on WSL/Ubuntu) - but I have observed this issue both locally and also on a linux server.
go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/home/qasim/.cache/go-build" GOENV="/home/qasim/.config/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/home/qasim/go/pkg/mod" GONOPROXY="" GONOSUMDB="[redacted]" GOOS="linux" GOPATH="/home/qasim/go" GOPRIVATE="" GOPROXY="[redacted]" GOROOT="/usr/local/go" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64" GCCGO="gccgo" AR="ar" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="/home/qasim/Source/snapp/locsim/go.mod" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build776046727=/tmp/go-build -gno-record-gcc-switches"

[hiyanxu]

The same problem happen with me. Change sampling with 10、20、50、100 hz, only 100hz can display CPU profiles.

[abeaumont]

I cannot reproduce the problem with profiles vanishing anymore :confetti_ball: But different sampling rates are not properly taken into account when calculating the actual time spent. For example, I run the go adhoc example two times, first sampling at 100Hz, then sampling at 50Hz, these are the results:

As it can be seen in the graph, it looks like the second takes (roughly) half the time, while in reality they take the same amount. The problem is thus that the normalization doesn't take into account the actual same rate.