New lockfile `/tmp/clvm_compile/p2_conditions.clvm.lock` prevents second harvester being started

[neurosis69]

Would it be possible to add the node_id to the path to get uniqueness? https://github.com/Chia-Network/chia-blockchain/blob/093115311176e56388fb40334003a20e440da721/chia/wallet/puzzles/load_clvm.py#L72

I run two harvester instances on the same machine, both are connected to different farmers to simulate kind of H/A. With versions < 1.5.1, this was possible by just changing the port of the secondary harvester and farmers to let them connect.

Since 1.5.1 I cannot start a second harvester instance on the same machine and the reason is that both want to acquire the same lock file.

Traceback (most recent call last):
  File "/home/chia/chia-blockchain/venv/bin/chia", line 5, in <module>
    from chia.cmds.chia import main
  File "/home/chia/chia-blockchain/chia/cmds/chia.py", line 18, in <module>
    from chia.cmds.plotters import plotters_cmd
  File "/home/chia/chia-blockchain/chia/cmds/plotters.py", line 2, in <module>
    from chia.plotters.plotters import call_plotters
  File "/home/chia/chia-blockchain/chia/plotters/plotters.py", line 5, in <module>
    from chia.plotters.bladebit import get_bladebit_install_info, plot_bladebit
  File "/home/chia/chia-blockchain/chia/plotters/bladebit.py", line 10, in <module>
    from chia.plotting.create_plots import resolve_plot_keys
  File "/home/chia/chia-blockchain/chia/plotting/create_plots.py", line 10, in <module>
    from chia.daemon.keychain_proxy import KeychainProxy, connect_to_keychain_and_validate, wrap_local_keychain
  File "/home/chia/chia-blockchain/chia/daemon/keychain_proxy.py", line 8, in <module>
    from chia.cmds.init_funcs import check_keys
  File "/home/chia/chia-blockchain/chia/cmds/init_funcs.py", line 10, in <module>
    from chia.consensus.coinbase import create_puzzlehash_for_pk
  File "/home/chia/chia-blockchain/chia/consensus/coinbase.py", line 6, in <module>
    from chia.wallet.puzzles.p2_delegated_puzzle_or_hidden_puzzle import puzzle_for_pk
  File "/home/chia/chia-blockchain/chia/wallet/puzzles/p2_delegated_puzzle_or_hidden_puzzle.py", line 68, in <module>
    from .p2_conditions import puzzle_for_conditions
  File "/home/chia/chia-blockchain/chia/wallet/puzzles/p2_conditions.py", line 17, in <module>
    MOD = load_clvm("p2_conditions.clvm")
  File "/home/chia/chia-blockchain/chia/wallet/puzzles/load_clvm.py", line 105, in load_clvm
    return Program.from_bytes(bytes(load_serialized_clvm(clvm_filename, package_or_requirement=package_or_requirement)))
  File "/home/chia/chia-blockchain/chia/wallet/puzzles/load_clvm.py", line 91, in load_serialized_clvm
    compile_clvm(full_path, output, search_paths=[full_path.parent])
  File "/home/chia/chia-blockchain/chia/wallet/puzzles/load_clvm.py", line 71, in compile_clvm
    with Lockfile.create(pathlib.Path(tempfile.gettempdir()) / "clvm_compile" / full_path.name):
  File "/home/chia/chia-blockchain/chia/util/lock.py", line 29, in __enter__
    self.acquire(timeout=self.timeout, poll_interval=self.poll_interval)
  File "/home/chia/chia-blockchain/chia/util/lock.py", line 42, in acquire
    self._lock.acquire(timeout=timeout, poll_interval=poll_interval)
  File "/home/chia/chia-blockchain/venv/lib/python3.10/site-packages/filelock/_api.py", line 173, in acquire
    self._acquire()
  File "/home/chia/chia-blockchain/venv/lib/python3.10/site-packages/filelock/_unix.py", line 35, in _acquire
    fd = os.open(self._lock_file, open_mode)
PermissionError: [Errno 13] Permission denied: '/tmp/clvm_compile/p2_conditions.clvm.lock'

As workaround I hardcoded the path to /tmp/clvm_compile_1, but I think that could be solved better.

[arvidn]

@xdustinface could this have been introduced by https://github.com/Chia-Network/chia-blockchain/pull/12541 ? It looks odd that any locking of these files would have to happen outside of compiling them

[altendky]

We should not be recompiling these on every start. Per my discussions that seems to be a developer quality of life feature that leaks out to not only non-chialisp devs but even users. Previously we had issues around this where the lock file wouldn't get removed and you would get a dead silent hang with no stdout nor logging, what with it happening during import. I planned on stopping this behavior entirely after https://github.com/Chia-Network/chia-blockchain/pull/12050.

But... for now, this should be a real temporary directory not a hard coded one. Then they won't both go into the same directory.

[github-actions[bot]]

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

[altendky]

I think my suggestion of a 'real' temp directory defeats the point. Sorry for the detour there. I'm bringing up another alternative that I wanted to do for other reasons anyways over in https://github.com/Chia-Network/chia-blockchain/pull/13478 to just stop recompiling the Chialisp for most cases.

[github-actions[bot]]

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

[altendky]

https://github.com/Chia-Network/chia-blockchain/pull/13478 has been merged and may have side-stepped this issue. @neurosis69, did you have a chance to try either that branch or now main?

[neurosis69]

haven't had time yet. I plan to test this during the upgrade of my farms components to 1.6, probably until next weekend.

[neurosis69]

I can give the following update:

• harvester1 still running with 1.5.1 using /tmp/clvm_compile as path for recompiling
• harvester2 was running with 1.5.1 using /tmp/clvm_compile_1 as hardcoded path for recompiling
• harvester2 is now updated to main:71d33eb86b81dd29282ec6001bdeda020e0635ab

running install.sh for harvester2 completed successfully. but starting the harvester process resulted in the following error:

Traceback (most recent call last):
  File "/home/chia/chia-blockchain/venv/bin/chia", line 5, in <module>
    from chia.cmds.chia import main
  File "/home/chia/chia-blockchain/chia/cmds/chia.py", line 13, in <module>
    from chia.cmds.peer import peer_cmd
  File "/home/chia/chia-blockchain/chia/cmds/peer.py", line 7, in <module>
    from chia.cmds.cmds_util import NODE_TYPES
  File "/home/chia/chia-blockchain/chia/cmds/cmds_util.py", line 13, in <module>
    from chia.rpc.wallet_rpc_client import WalletRpcClient
  File "/home/chia/chia-blockchain/chia/rpc/wallet_rpc_client.py", line 3, in <module>
    from chia.data_layer.data_layer_wallet import Mirror, SingletonRecord
  File "/home/chia/chia-blockchain/chia/data_layer/data_layer_wallet.py", line 52, in <module>
    from chia.wallet.wallet import Wallet
  File "/home/chia/chia-blockchain/chia/wallet/wallet.py", line 9, in <module>
    from chia.full_node.bundle_tools import simple_solution_generator
  File "/home/chia/chia-blockchain/chia/full_node/bundle_tools.py", line 4, in <module>
    from chia.full_node.generator import create_compressed_generator
  File "/home/chia/chia-blockchain/chia/full_node/generator.py", line 7, in <module>
    from chia.wallet.puzzles.rom_bootstrap_generator import get_generator
  File "/home/chia/chia-blockchain/chia/wallet/puzzles/rom_bootstrap_generator.py", line 7, in <module>
    MOD = load_serialized_clvm("rom_bootstrap_generator.clvm")
  File "/home/chia/chia-blockchain/chia/wallet/puzzles/load_clvm.py", line 105, in load_serialized_clvm
    compile_clvm(full_path, output, search_paths=search_paths)
  File "/home/chia/chia-blockchain/chia/wallet/puzzles/load_clvm.py", line 75, in compile_clvm
    with Lockfile.create(pathlib.Path(tempfile.gettempdir()) / "clvm_compile" / full_path.name):
  File "/home/chia/chia-blockchain/chia/util/lock.py", line 29, in __enter__
    self.acquire(timeout=self.timeout, poll_interval=self.poll_interval)
  File "/home/chia/chia-blockchain/chia/util/lock.py", line 42, in acquire
    self._lock.acquire(timeout=timeout, poll_interval=poll_interval)
  File "/home/chia/chia-blockchain/venv/lib/python3.10/site-packages/filelock/_api.py", line 173, in acquire
    self._acquire()
  File "/home/chia/chia-blockchain/venv/lib/python3.10/site-packages/filelock/_unix.py", line 35, in _acquire
    fd = os.open(self._lock_file, open_mode)
PermissionError: [Errno 13] Permission denied: '/tmp/clvm_compile/rom_bootstrap_generator.clvm.lock'

[github-actions[bot]]

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

[neurosis69]

the issue is still not fixed

[peter92427]

Hello, had the same problem, probably because running multiple instances... You can avoid it by setting alternate temp dir before starting node.

machina:~$ echo $TMPDIR machina:~$ export TMPDIR=/tmp/chia1 machina:~$ echo $TMPDIR /tmp/chia1

Set another temp dir for the second harvester before starting it, etc...

[trepca]

@neurosis69 did you try from main branch?

[altendky]

They said they switched their harvester2 to https://github.com/Chia-Network/chia-blockchain/commit/71d33eb86b81dd29282ec6001bdeda020e0635ab which seems to include the initial relevant changes from https://github.com/Chia-Network/chia-blockchain/commit/a20ee08fce6d608bf0f886b8edba35031dabb1c1. But... I had missed the serialized CLVM load usages which were addressed later by https://github.com/Chia-Network/chia-blockchain/commit/5ee4c398bfa2a10461ea76cb1a8dd33f5e84f703. So, there is reason to try main again.

[neurosis69]

I have cloned from main and it is working now! Thx for your support.

[altendky]

Thanks for the report and testing. :]