scCODA
scCODA copied to clipboard
Published
20 hours ago •
theislab
theislab
Which tensorflow version to use?
Hi sccoda team,
While trying to also run sccoda on my laptop (MacBook Air M1), and downloading all relevant packages for my project, I've come across a version discrepancy. (Note: when I installed sccoda/tensorflow on an iMac a few weeks back the issue did not arise)
When installing tensorflow:
sccoda 0.1.7 requires numpy>=1.21, but you have numpy 1.19.5 which is incompatible. sccoda 0.1.7 requires tensorflow>=2.8, but you have tensorflow 2.5.0 which is incompatible. sccoda 0.1.7 requires tensorflow-probability>=0.16.0, but you have tensorflow-probability 0.12.0 which is incompatible.
On the description for sccoda installation however, other package versions are described.
This package uses the
tensorflow (>=2.4, <2.6)andtensorflow-probability (==0.12)packages.
Which package versions ones should I be using?
Also, I've been having issues importing sccoda into my jupyter notebook, as the kernel stops running. Could that issue be related to the package versions? (Note: I do not have this issue using sccoda in jupyter notebook on an iMac, only on my macbook air)
Thanks!
Hi @c-westhoven,
when we updated scCODA recently to support the latest tensorflow (>=2.8) and tensorflow-probability (>=0.16) versions, we missed updating the readme. The correct version requirements should be displayed there now.
Regarding your issues with the macbook: If this isn't solved with upgrading to a later tensorflow version, I suspect that this is related to the fact that your macbook uses a M1 CPU, which might need a special installment of tensorflow to work (see here for example). Since I don't have access to such a CPU right now, I'm unfortunately not able to test this.
It would be great if you could report back on your experiences with running scCODA on your macbook! Thanks!
Hello there, following up on this as I am facing a similar issue. I've installed tf>2.8 (2.12.0) and tf-probability > 0.16 (0.19.0) and I cannot reproduce the "getting started" tutorial. In particular:
>>> model_salm = mod.CompositionalAnalysis(data_salm, formula="Condition", reference_cell_type="Goblet")
Metal device set to: Apple M2
systemMemory: 16.00 GB
maxCacheSize: 5.33 GB
>>> sim_results = model_salm.sample_hmc()
WARNING:tensorflow:From /Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/internal/batched_rejection_sampler.py:102: calling while_loop_v2 (from tensorflow.python.ops.control_flow_ops) with back_prop=False is deprecated and will be removed in a future version.
Instructions for updating:
back_prop=False is deprecated. Consider using tf.stop_gradient instead.
Instead of:
results = tf.while_loop(c, b, vars, back_prop=False)
Use:
results = tf.nest.map_structure(tf.stop_gradient, tf.while_loop(c, b, vars))
2023-04-20 16:50:54.493289: W tensorflow/tsl/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
2023-04-20 16:50:54.864903: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : NOT_FOUND: could not find registered platform with id: 0x13ca32970
0%| | 0/20000 [00:00<?, ?it/s]---------------------------------------------------------------------------
NotFoundError Traceback (most recent call last)
Cell In[31], line 1
----> 1 sim_results = model_salm.sample_hmc()
File ~/miniforge3/envs/sccoda/lib/python3.10/site-packages/scCODA-0.1.9-py3.10.egg/sccoda/model/scCODA_model.py:314, in CompositionalModel.sample_hmc(self, num_results, num_burnin, num_adapt_steps, num_leapfrog_steps, step_size, verbose)
306 return {
307 'target_log_prob': pkr.inner_results.inner_results.accepted_results.target_log_prob,
308 'diverging': (pkr.inner_results.inner_results.log_accept_ratio < -1000.),
309 'is_accepted': pkr.inner_results.inner_results.is_accepted,
310 'step_size': pkr.inner_results.inner_results.accepted_results.step_size,
311 }
313 # The actual HMC sampling process
--> 314 states, kernel_results, duration = self.sampling(num_results, num_burnin,
315 hmc_kernel, self.init_params, trace_fn)
317 if verbose:
318 pbar.bar.close()
File ~/miniforge3/envs/sccoda/lib/python3.10/site-packages/scCODA-0.1.9-py3.10.egg/sccoda/model/scCODA_model.py:165, in CompositionalModel.sampling(self, num_results, num_burnin, kernel, init_state, trace_fn)
163 # The actual sampling process
164 start = time.time()
--> 165 states, kernel_results = sample_mcmc(num_results, num_burnin, kernel, init_state, trace_fn)
166 duration = time.time() - start
167 print("MCMC sampling finished. ({:.3f} sec)".format(duration))
File ~/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow/python/util/traceback_utils.py:153, in filter_traceback.<locals>.error_handler(*args, **kwargs)
151 except Exception as e:
152 filtered_tb = _process_traceback_frames(e.__traceback__)
--> 153 raise e.with_traceback(filtered_tb) from None
154 finally:
155 del filtered_tb
File ~/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:52, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
50 try:
51 ctx.ensure_initialized()
---> 52 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
53 inputs, attrs, num_outputs)
54 except core._NotOkStatusException as e:
55 if name is not None:
NotFoundError: Graph execution error:
Detected at node 'mcmc_sample_chain/reduced_kernel_bootstrap_results/simple_step_size_adaptation___init__/_bootstrap_results/transformed_kernel_bootstrap_results/mh_bootstrap_results/hmc_kernel_bootstrap_results/maybe_call_fn_and_grads/value_and_gradients/value_and_gradient/PartitionedCall' defined at (most recent call last):
File "/Users/cittaro.davide/miniforge3/envs/sccoda/bin/ipython", line 8, in <module>
sys.exit(start_ipython())
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/__init__.py", line 128, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/traitlets/config/application.py", line 1043, in launch_instance
app.start()
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/terminal/ipapp.py", line 318, in start
self.shell.mainloop()
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/terminal/interactiveshell.py", line 888, in mainloop
self.interact()
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/terminal/interactiveshell.py", line 881, in interact
self.run_cell(code, store_history=True)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-31-719266c03b83>", line 1, in <module>
sim_results = model_salm.sample_hmc()
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/scCODA-0.1.9-py3.10.egg/sccoda/model/scCODA_model.py", line 314, in sample_hmc
states, kernel_results, duration = self.sampling(num_results, num_burnin,
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/scCODA-0.1.9-py3.10.egg/sccoda/model/scCODA_model.py", line 165, in sampling
states, kernel_results = sample_mcmc(num_results, num_burnin, kernel, init_state, trace_fn)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/scCODA-0.1.9-py3.10.egg/sccoda/model/scCODA_model.py", line 155, in sample_mcmc
return tfp.mcmc.sample_chain(
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/sample.py", line 330, in sample_chain
previous_kernel_results = kernel.bootstrap_results(current_state)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/experimental/mcmc/with_reductions.py", line 146, in bootstrap_results
inner_results = self.inner_kernel.bootstrap_results(init_state)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/simple_step_size_adaptation.py", line 443, in bootstrap_results
inner_results = self.inner_kernel.bootstrap_results(init_state)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/transformed_kernel.py", line 492, in bootstrap_results
inner_results=self._inner_kernel.bootstrap_results(
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/hmc.py", line 561, in bootstrap_results
kernel_results = self._impl.bootstrap_results(init_state)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/metropolis_hastings.py", line 273, in bootstrap_results
pkr = self.inner_kernel.bootstrap_results(init_state)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/hmc.py", line 775, in bootstrap_results
] = mcmc_util.maybe_call_fn_and_grads(self.target_log_prob_fn, init_state)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/internal/util.py", line 297, in maybe_call_fn_and_grads
result, grads = _value_and_gradients(fn, fn_arg_list, result, grads)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/internal/util.py", line 265, in _value_and_gradients
return tfp_math_value_and_gradients(fn, fn_arg_list)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/math/gradient.py", line 108, in value_and_gradient
return _value_and_grad_impl(
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/math/gradient.py", line 378, in _value_and_grad_impl
y, dydx, aux = grad_fn(lambda: f(*args, **kwargs) if _has_args(f) else f(),
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/math/gradient.py", line 330, in _gradient_old
y, aux = f()
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/math/gradient.py", line 378, in <lambda>
y, dydx, aux = grad_fn(lambda: f(*args, **kwargs) if _has_args(f) else f(),
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/math/gradient.py", line 375, in <lambda>
f = lambda *args, **kwargs: (real_f(*args, **kwargs) # pylint: disable=g-long-lambda
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/transformed_kernel.py", line 123, in transformed_log_prob_fn
tlp = log_prob_fn(*fn(state_parts))
Node: 'mcmc_sample_chain/reduced_kernel_bootstrap_results/simple_step_size_adaptation___init__/_bootstrap_results/transformed_kernel_bootstrap_results/mh_bootstrap_results/hmc_kernel_bootstrap_results/maybe_call_fn_and_grads/value_and_gradients/value_and_gradient/PartitionedCall'
Detected at node 'mcmc_sample_chain/reduced_kernel_bootstrap_results/simple_step_size_adaptation___init__/_bootstrap_results/transformed_kernel_bootstrap_results/mh_bootstrap_results/hmc_kernel_bootstrap_results/maybe_call_fn_and_grads/value_and_gradients/value_and_gradient/PartitionedCall' defined at (most recent call last):
File "/Users/cittaro.davide/miniforge3/envs/sccoda/bin/ipython", line 8, in <module>
sys.exit(start_ipython())
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/__init__.py", line 128, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/traitlets/config/application.py", line 1043, in launch_instance
app.start()
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/terminal/ipapp.py", line 318, in start
self.shell.mainloop()
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/terminal/interactiveshell.py", line 888, in mainloop
self.interact()
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/terminal/interactiveshell.py", line 881, in interact
self.run_cell(code, store_history=True)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-31-719266c03b83>", line 1, in <module>
sim_results = model_salm.sample_hmc()
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/scCODA-0.1.9-py3.10.egg/sccoda/model/scCODA_model.py", line 314, in sample_hmc
states, kernel_results, duration = self.sampling(num_results, num_burnin,
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/scCODA-0.1.9-py3.10.egg/sccoda/model/scCODA_model.py", line 165, in sampling
states, kernel_results = sample_mcmc(num_results, num_burnin, kernel, init_state, trace_fn)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/scCODA-0.1.9-py3.10.egg/sccoda/model/scCODA_model.py", line 155, in sample_mcmc
return tfp.mcmc.sample_chain(
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/sample.py", line 330, in sample_chain
previous_kernel_results = kernel.bootstrap_results(current_state)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/experimental/mcmc/with_reductions.py", line 146, in bootstrap_results
inner_results = self.inner_kernel.bootstrap_results(init_state)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/simple_step_size_adaptation.py", line 443, in bootstrap_results
inner_results = self.inner_kernel.bootstrap_results(init_state)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/transformed_kernel.py", line 492, in bootstrap_results
inner_results=self._inner_kernel.bootstrap_results(
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/hmc.py", line 561, in bootstrap_results
kernel_results = self._impl.bootstrap_results(init_state)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/metropolis_hastings.py", line 273, in bootstrap_results
pkr = self.inner_kernel.bootstrap_results(init_state)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/hmc.py", line 775, in bootstrap_results
] = mcmc_util.maybe_call_fn_and_grads(self.target_log_prob_fn, init_state)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/internal/util.py", line 297, in maybe_call_fn_and_grads
result, grads = _value_and_gradients(fn, fn_arg_list, result, grads)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/internal/util.py", line 265, in _value_and_gradients
return tfp_math_value_and_gradients(fn, fn_arg_list)
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/math/gradient.py", line 108, in value_and_gradient
return _value_and_grad_impl(
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/math/gradient.py", line 378, in _value_and_grad_impl
y, dydx, aux = grad_fn(lambda: f(*args, **kwargs) if _has_args(f) else f(),
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/math/gradient.py", line 330, in _gradient_old
y, aux = f()
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/math/gradient.py", line 378, in <lambda>
y, dydx, aux = grad_fn(lambda: f(*args, **kwargs) if _has_args(f) else f(),
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/math/gradient.py", line 375, in <lambda>
f = lambda *args, **kwargs: (real_f(*args, **kwargs) # pylint: disable=g-long-lambda
File "/Users/cittaro.davide/miniforge3/envs/sccoda/lib/python3.10/site-packages/tensorflow_probability/python/mcmc/transformed_kernel.py", line 123, in transformed_log_prob_fn
tlp = log_prob_fn(*fn(state_parts))
Node: 'mcmc_sample_chain/reduced_kernel_bootstrap_results/simple_step_size_adaptation___init__/_bootstrap_results/transformed_kernel_bootstrap_results/mh_bootstrap_results/hmc_kernel_bootstrap_results/maybe_call_fn_and_grads/value_and_gradients/value_and_gradient/PartitionedCall'
2 root error(s) found.
(0) NOT_FOUND: could not find registered platform with id: 0x13ca32970
[[{{node mcmc_sample_chain/reduced_kernel_bootstrap_results/simple_step_size_adaptation___init__/_bootstrap_results/transformed_kernel_bootstrap_results/mh_bootstrap_results/hmc_kernel_bootstrap_results/maybe_call_fn_and_grads/value_and_gradients/value_and_gradient/PartitionedCall}}]]
[[mcmc_sample_chain/trace_scan/while/exit/_277/_64]]
(1) NOT_FOUND: could not find registered platform with id: 0x13ca32970
[[{{node mcmc_sample_chain/reduced_kernel_bootstrap_results/simple_step_size_adaptation___init__/_bootstrap_results/transformed_kernel_bootstrap_results/mh_bootstrap_results/hmc_kernel_bootstrap_results/maybe_call_fn_and_grads/value_and_gradients/value_and_gradient/PartitionedCall}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_sample_mcmc_3547]
Hi @dawe! We've regularly faced similar issues in the past, which were usually related to breaking changes in tensorflow or tensorflow-probability. Therefore, we recently re-implemented scCODA as part of the pertpy package, which does not rely on tensorflow anymore. You can try out the new implementation or use the known stable versions (tensorflow 2.8 and tf-probability 0.16) of these dependencies.