flowdec
flowdec copied to clipboard
hammerlab
Lightsheet Processing
Hi Eric,
I manage to established the pipeline using flowdec to deconvolved the lighthsheet images, but I am observing that for each channel and block the tensorflow enviorment is reload. Is there a way to avoid this?
Script:
data = [] total_data = []
algo = fd_restoration.RichardsonLucyDeconvolver(3).initialize()
for n in range(4): for k in range(4): datao = dapi[:, n*480:(n+1)480, k480:(k+1)*480] res = algo.run(fd_data.Acquisition(data=datao, kernel=psf_single), niter=70) data_deconvolved = res.data data.append(data_deconvolved) total_data.append(data) data = [] combined_image = da.block(total_data) combined_image.shape
Verbose:
2020-04-14 11:51:31.772567: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-04-14 11:51:31.796488: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599990000 Hz 2020-04-14 11:51:31.796918: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55806f8d8a70 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-04-14 11:51:31.796938: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2020-04-14 11:51:31.863707: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-14 11:51:31.864106: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55806fa87b00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2020-04-14 11:51:31.864120: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2070 with Max-Q Design, Compute Capability 7.5 2020-04-14 11:51:31.864332: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-14 11:51:31.864642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce RTX 2070 with Max-Q Design computeCapability: 7.5 coreClock: 1.185GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 357.69GiB/s 2020-04-14 11:51:31.864701: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2020-04-14 11:51:31.864727: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2020-04-14 11:51:31.864748: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2020-04-14 11:51:31.864769: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2020-04-14 11:51:31.864790: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2020-04-14 11:51:31.864811: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2020-04-14 11:51:31.864832: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-04-14 11:51:31.864879: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-14 11:51:31.865210: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-14 11:51:31.865520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0 2020-04-14 11:51:31.865559: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2020-04-14 11:51:31.866593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-04-14 11:51:31.866603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0 2020-04-14 11:51:31.866607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N 2020-04-14 11:51:31.866686: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-14 11:51:31.867018: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-14 11:51:31.867325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7010 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 7.5) 2020-04-14 11:51:32.577218: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2020-04-14 11:51:32.813305: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2020-04-14 11:51:37.893802: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-14 11:51:37.894299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce RTX 2070 with Max-Q Design computeCapability: 7.5 coreClock: 1.185GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 357.69GiB/s 2020-04-14 11:51:37.894369: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2020-04-14 11:51:37.894390: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2020-04-14 11:51:37.894407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2020-04-14 11:51:37.894424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2020-04-14 11:51:37.894442: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2020-04-14 11:51:37.894460: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2020-04-14 11:51:37.894478: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-04-14 11:51:37.894538: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-14 11:51:37.894779: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-14 11:51:37.894979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0 2020-04-14 11:51:37.894998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-04-14 11:51:37.895003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0 2020-04-14 11:51:37.895007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N 2020-04-14 11:51:37.895075: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-14 11:51:37.895318: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-14 11:51:37.895524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7010 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 7.5) 2020-04-14 11:51:43.133267: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-04-14 11:51:43.133516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce RTX 2070 with Max-Q Design computeCapability: 7.5 coreClock: 1.185GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 357.69GiB/s
Hi @jesusdpa1 , it is definitely not normal to see that, but it's hard to say for sure without the whole script. Is dapi a dask or numpy array?
Hi Eric, I loaded the image using the lazy dask arrays but before each loop I compute each indexed image (move to memory dask to numpy array). Here is the whole script,
Ah, well I can't say that's definitely the issue but support for TF 1.x was added in https://github.com/hammerlab/flowdec/pull/28 though it's really only tested against 1.14.
I would try installing tensorflow-gpu==1.14.0 first and see if it goes away.
Tensorflow 2.2.0 runs on my linux machine with flowdec without a problem, or so I thought until reading this (the windows one somehow can't and I need to use an older version).
I do something very similar to your example (well, visit points loop, time points loop, and channel loop). I just realized that I have the same issue as you, I have all the cuda libraries being loaded in every cycle. However it doesn't seem to affect speed.
**Edit: tensorflow 1.14 in the windows machine does the same **
