TensorFlowSharp

TensorFlowSharp copied to clipboard

i'm trying to set the limit of mem usage on the GPU so TF can coexist with other processes on GPU

as I did not find a wrapper for ConfigProto in TensorFlowSharp I did the folowing ugly workaround :

In python config = tf.ConfigProto() config.gpu_options.per_process_gpu_memory_fraction = 0.20 config.gpu_options.allow_growth = False string1 = config.SerializeToString() f = open('MyProto.txt','wb') f.write(string1) f.close()

2. In C# , when starting my session var _graph = new TFGraph(); var file = new FileReader(name_of_previous_saved_protofile); byte[] byteProto = File.ReadAllBytes(fileProtoName); IntPtr unmanagedPointer = Marshal.AllocHGlobal(byteProto.Length); Marshal.Copy(byteProto, 0, unmanagedPointer, byteProto.Length); _graph.Import(File.ReadAllBytes(modelFilename)); var _session = new TFSession(_graph); TensorFlow.TFSessionOptions options = new TensorFlow.TFSessionOptions(); TFStatus tfStat = new TFStatus(); options.SetConfig(unmanagedPointer, byteProto.Length, tfStat);

Although in python I see that the memory settles at 30% (probably some 10% is used by windows), when I try in CSharp, TF gets all the memory on both memory cards .... The tfStat is "OK", and the libtensorflow.dll is the one from here : https://github.com/migueldeicaza/TensorFlowSharp/issues/109

Thanks in advance for any help on this issue

Hi @intelpen

You can use protobuf by generating codes from .proto definitions. Here is instruction: document

I already convert some of .proto of TF to C# to use StatSummary, and you can see files here. After include files to your project, you can use ConfigProto just like following codes.

using Google.ProtoBuf;
using Vision.Tensorflow.Proto;

var config = new Proto.ConfigProto();
config.GpuOptions.AllowGrowth = false;
config.GpuOptions.PerProcessGpuMemoryFraction = 0.20;
using(MemoryStream stream = new MemoryStream())
{
    config.WriteTo(stream);
    byte[] configBuffer = stream.ToArray();
}

thanks @gmlwns2000 ! is there a nugget to install the Vision.Tensorflow ? (What is the name of the nugget ? ) ?

Yohoo, I figured out the problem.

In my case, the first (and easy to spot afterwards) issue was that I did not pass the options to the session creation. The fix is : _session = new TFSession(_graph, options, tfStat1);

I have also tried the option you gave me (I copy-pasted all your Proto folder in my project .. I din not have the nerve to try to generate with protobuf ...) and it worked as well, with one correction , create a GpuOptions() before .

//var config = new ConfigProto(); //config.GpuOptions = new GPUOptions(); //config.GpuOptions.AllowGrowth = false; //config.GpuOptions.PerProcessGpuMemoryFraction = 0.2; //using (MemoryStream stream = new MemoryStream()) //{ // config.WriteTo(stream); // byte[] configBuffer = stream.ToArray(); // IntPtr unmanagedPointer = Marshal.AllocHGlobal(configBuffer.Length); // Marshal.Copy(configBuffer, 0, unmanagedPointer, configBuffer.Length); // options.SetConfig(unmanagedPointer, configBuffer.Length, tfStat); //} //var _session = new TFSession(_graph, options, tfStat);

In the long term, would be nice if we can have this ConfigProto() directly in the Tensorflow nugget, if possible.

Here's what I did to get it to work

TFSessionOptions TFOptions = new TFSessionOptions ();
	unsafe {
		byte [] GPUConfig = new byte[] { 0x32, 0x0b, 0x09, 0x33, 0x33, 0x33, 0x33, 0x33, 0x33, 0xd3, 0x3f, 0x20, 0x01 };
		fixed (void* ptr = &GPUConfig [0]) {
			TFOptions.SetConfig (new IntPtr (ptr), GPUConfig.Length);
		}
	}
				
//using (var session = new TFSession (graph, TFOptions)) {

You can get the GPUConfig bytes from python like this (horrible code, I know, but it works)

    config = tf.ConfigProto()
    config.gpu_options.allow_growth = True
    config.gpu_options.per_process_gpu_memory_fraction=0.3
    sess = tf.Session(config=config)

    import binascii
    text_file = open("gpuproto.dat", "w")
    bytes = config.SerializeToString()
    hexstring = " ".join("%02x" % b for b in bytes)
    text_file.write(hexstring)
    text_file.close()

Edit: If you have the tensorflow-gpu pip package installed, you can find a copy of the dll at %localappdata%\Programs\Python\Python36\Lib\site-packages\tensorflow\python - simply copy _pywrap_tensorflow_internal.pyd and rename it to libtensorflow.dll

@hwvs when I do your approach I get an error in C#:

TensorFlow.TFException: Unparseable ConfigProto

This is how I generate my array:

import binascii
import tensorflow as tf

config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.10
config.gpu_options.allow_growth = False

bytes = config.SerializeToString()
print(" ".join("0x%02x," % b for b in bytes))

This is how the array looks like:

var gpuConfig = new[] { 0x32, 0x09, 0x09, 0x9a, 0x99, 0x99, 0x99, 0x99, 0x99, 0xb9, 0x3f };

It's a bit shorter than yours...

OK my problem was that gpuConfig was automatically declared as int[] instead of byte[].

Using ideas from the previous posts, looking at the byte arrays generated, I noticed that embedded in the byte array was the bytes used to represent the IEEE double for the gpu memory fraction limit. For example, in the 11-byte array shown above by @turowicz, the first 3 bytes appear to be some sort of code or flags...possibly in part due to the "allow_growth = false". The last 8 bytes are the data needed to represent the double used for the gpu memory fraction limit. I ran @turowicz's python code with several double values to make sure, and for the settings given (allow_growth = false, and the memory fraction value), the first 3 bytes were constant and the last 8 changed with the percent setting. With that in mind, I wrote the following code to create this options array with any given percent value.

Would be great to figure out the first 3 bytes, or truly how to build this array. Regardless, this worked great for me. Thanks to everyone above.

double gpuMemoryFractionLimit = 0.5;

// this seems to be the header for allow_growth=false
byte[] header = new byte[] { 0x32, 0x09, 0x09 };  

// this is the byte array for the double value
byte[] ba = BitConverter.GetBytes(gpuMemoryFractionLimit);  

// build the concatenated array
byte[] config = new byte[header.Length + ba.Length]; 
header.CopyTo(config, 0);
ba.CopyTo(config, 3);

// create the TFSessionOptions instance using the 
TFSessionOptions TFOptions = new TFSessionOptions();

// you could choose to pin here, but I just copied to unmanaged space
IntPtr unmanagedPointer = Marshal.AllocHGlobal(config.Length);
Marshal.Copy(config, 0, unmanagedPointer, config.Length);

// set the config
TFOptions.SetConfig(unmanagedPointer, config.Length);

// build your graph
_graph = new TFGraph();

	// add your custom stuff to graph

// create the session
TFSession_session = new TFSession(_graph, TFOptions);


// don't forget to free the unmanaged allocation made above
Marshal.FreeHGlobal(unmanagedPointer);

@rbgreenway this is great!

I can confirm @rbgreenway's solution works well. I wrapped it in a SetGpuRatio method.

Using ideas from the previous posts, looking at the byte arrays generated, I noticed that embedded in the byte array was the bytes used to represent the IEEE double for the gpu memory fraction limit. For example, in the 11-byte array shown above by @turowicz, the first 3 bytes appear to be some sort of code or flags...possibly in part due to the "allow_growth = false". The last 8 bytes are the data needed to represent the double used for the gpu memory fraction limit. I ran @turowicz's python code with several double values to make sure, and for the settings given (allow_growth = false, and the memory fraction value), the first 3 bytes were constant and the last 8 changed with the percent setting. With that in mind, I wrote the following code to create this options array with any given percent value.

Would be great to figure out the first 3 bytes, or truly how to build this array. Regardless, this worked great for me. Thanks to everyone above.

double gpuMemoryFractionLimit = 0.5;

// this seems to be the header for allow_growth=false
byte[] header = new byte[] { 0x32, 0x09, 0x09 };  

// this is the byte array for the double value
byte[] ba = BitConverter.GetBytes(gpuMemoryFractionLimit);  

// build the concatenated array
byte[] config = new byte[header.Length + ba.Length]; 
header.CopyTo(config, 0);
ba.CopyTo(config, 3);

// create the TFSessionOptions instance using the 
TFSessionOptions TFOptions = new TFSessionOptions();

// you could choose to pin here, but I just copied to unmanaged space
IntPtr unmanagedPointer = Marshal.AllocHGlobal(config.Length);
Marshal.Copy(config, 0, unmanagedPointer, config.Length);

// set the config
TFOptions.SetConfig(unmanagedPointer, config.Length);

// build your graph
_graph = new TFGraph();

	// add your custom stuff to graph

// create the session
TFSession_session = new TFSession(_graph, TFOptions);


// don't forget to free the unmanaged allocation made above
Marshal.FreeHGlobal(unmanagedPointer);

According to your settings, or prompt me CPU, not GPU??

I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2