RosettaCommons
RF3 error "TypeError("FabricTrainer.__init__() got an unexpected keyword argument 'cleanup_guideposts'")"
Hello there,
I ran RFD3 and RF3 perfectly fine with the demo and a regular protein. However, RF3 returned a TypeError("FabricTrainer.init() got an unexpected keyword argument 'cleanup_guideposts'") when I tried to design a binder for a cup-shaped protein. Could this be because the binder is designed to be inside the target protein?
Many thanks! :)
The error message is:
│ │ │ │ │ 'param_groups': [ │ │ │ │ │ │ │ { │ │ │ │ │ │ │ │ 'lr': 0.0009, │ │ │ │ │ │ │ │ 'betas': [0.9, 0.95], │ │ │ │ │ │ │ │ 'eps': 1e-08, │ │ │ │ │ │ │ │ 'weight_decay': 0, │ │ │ │ │ │ │ │ 'amsgrad': False, │ │ │ │ │ │ │ │ 'maximize': False, │ │ │ │ │ │ │ │ 'foreach': None, │ │ │ │ │ │ │ │ 'capturable': False, │ │ │ │ │ │ │ │ 'differentiable': False, │ │ │ │ │ │ │ │ 'fused': None, │ │ │ │ │ │ │ │ ... +3 │ │ │ │ │ │ │ } │ │ │ │ │ │ ] │ │ │ │ │ }, │ │ │ │ │ 'scheduler_cfg': { │ │ │ │ │ │ 'scheduler': { │ │ │ │ │ │ │ 'base_lr': 0.0009, │ │ │ │ │ │ │ 'warmup_steps': 1000, │ │ │ │ │ │ │ 'decay_factor': 0.95, │ │ │ │ │ │ │ 'decay_steps': 50000, │ │ │ │ │ │ │ 'base_lrs': [0], │ │ │ │ │ │ │ 'last_epoch': 1482, │ │ │ │ │ │ │ '_step_count': 1483, │ │ │ │ │ │ │ '_get_lr_called_within_step': False, │ │ │ │ │ │ │ '_last_lr': [0.0009] │ │ │ │ │ │ }, │ │ │ │ │ │ 'interval': 'step', │ │ │ │ │ │ 'frequency': 1 │ │ │ │ │ }, │ │ │ │ │ 'global_step': 84672, │ │ │ │ │ 'current_epoch': 922, │ │ │ │ │ 'train_cfg': {'callbacks': {'log_af3_training_losses_callback': {'target': │ │ │ │ 'foundry.callbacks.train_logging.LogAF3TrainingLossesCallback', 'log_every_n': 1, 'log_full_batch_losses': True}, │ │ │ │ 'log_learning_rate_callback': {'target': 'foundry.callbacks.train_logging.LogLearningRateCallback', │ │ │ │ 'log_every_n': 10}, 'log_model_parameters_callback': {'target': │ │ │ │ 'foundry.callbacks.train_logging.LogModelParametersCallback'}, 'log_dataset_sampling_ratios_callback': │ │ │ │ {'target': 'foundry.callbacks.train_logging.LogDatasetSamplingRatiosCallback'}, │ │ │ │ 'store_validation_metrics_in_df_callback': {'target': │ │ │ │ 'rf3.callbacks.metrics_logging.StoreValidationMetricsInDFCallback', 'save_dir': │ │ │ │ '/net/scratch/dimaio/training/logs/train/af3/2025-10-10_09-54_JOB_48324318/val_metrics', 'metrics_to_save': │ │ │ │ 'all'}, 'log_af3_validation_metrics_callback': {'target': │ │ │ │ 'rf3.callbacks.metrics_logging.LogAF3ValidationMetricsCallback', 'metrics_to_log': 'all'}, │ │ │ │ 'dump_validation_structures_callback': {'target': │ │ │ │ 'rf3.callbacks.dump_validation_structures.DumpValidationStructuresCallback', 'save_dir': │ │ │ │ '/net/scratch/dimaio/training/logs/train/af3/2025-10-10_09-54_JOB_48324318/val_structures', 'dump_predictions': │ │ │ │ True, 'one_model_per_file': False, 'dump_trajectories': False}}, 'logger': {'csv': {'target': │ │ │ │ 'lightning.fabric.loggers.CSVLogger', 'root_dir': │ │ │ │ '/net/scratch/dimaio/training/logs/train/af3/2025-10-10_09-54_JOB_48324318', 'flush_logs_every_n_steps': 1}}, │ │ │ │ 'trainer': {'strategy': 'ddp', 'accelerator': 'gpu', 'devices_per_node': 8, 'num_nodes': 2, 'loss': │ │ │ │ {'confidence_loss': {'target': 'rf3.loss.af3_confidence_loss.ConfidenceLoss', 'weight': 1.0, 'plddt': {'weight': │ │ │ │ 1.0, 'n_bins': 50, 'max_value': 1.0}, 'pae': {'weight': 1.0, 'n_bins': 64, 'max_value': 32}, 'pde': {'weight': │ │ │ │ 1.0, 'n_bins': 64, 'max_value': 32}, 'exp_resolved': {'weight': 1.0, 'n_bins': 2, 'max_value': 1}, │ │ │ │ 'log_statistics': True, 'rank_loss': {'use_listnet_loss': False, 'weight': 0.0}}}, 'metrics': {'by_type_lddt': │ │ │ │ {'target': 'rf3.metrics.lddt.ByTypeLDDT'}, 'all_atom_lddt': {'target': 'rf3.metrics.lddt.AllAtomLDDT'}, │ │ │ │ 'distogram': {'target': 'rf3.metrics.distogram.DistogramLoss'}, 'distogram_comparisons': {'target': │ │ │ │ 'rf3.metrics.distogram.DistogramComparisons'}, 'distogram_entropy': {'target': │ │ │ │ 'rf3.metrics.distogram.DistogramEntropy'}, 'chiral_loss': {'target': 'rf3.metrics.chiral.ChiralLoss'}, │ │ │ │ 'unresolved_rasa': {'target': 'rf3.metrics.rasa.UnresolvedRegionRASA'}, 'ptm': {'target': │ │ │ │ 'rf3.metrics.predicted_error.ComputePTM'}, 'iptm': {'target': 'rf3.metrics.predicted_error.ComputeIPTM'}, │ │ │ │ 'count_clashing_chains': {'target': 'rf3.metrics.clashing_chains.CountClashingChains'}}, 'target': │ │ │ │ 'rf3.trainers.rf3.RF3TrainerWithConfidence', 'validate_every_n_epochs': 9999, 'max_epochs': 10000, │ │ │ │ 'n_examples_per_epoch': 8000, 'prevalidate': False, 'n_recycles_train': 4, 'clip_grad_max_norm': 10.0, │ │ │ │ 'output_dir': '/net/scratch/dimaio/training/logs/train/af3/2025-10-10_09-54_JOB_48324318', │ │ │ │ 'checkpoint_every_n_epochs': 1, 'precision': 'bf16-mixed', 'grad_accum_steps': 16}, 'paths': {'root_dir': │ │ │ │ '/home/dimaio/RF2-allatom-foundry', 'log_dir': '/net/scratch/dimaio/training/logs', 'output_dir': │ │ │ │ '/net/scratch/dimaio/training/logs/train/af3/2025-10-10_09-54_JOB_48324318', 'work_dir': │ │ │ │ '/home/dimaio/RF2-allatom-foundry/src/foundry', 'data': {'pdb_data_dir': │ │ │ │ '/projects/ml/datahub/dfs/af3_splits/2025_07_13', 'monomer_distillation_data_dir': │ │ │ │ '/squash/af2_distillation_facebook', 'monomer_distillation_parquet_dir': │ │ │ │ '/projects/ml/datahub/dfs/distillation/af2_distillation_facebook', 'mgnify_distillation_data_dir': │ │ │ │ '/squash/mgnify_distill_rf3/', 'mgnify_distillation_parquet_dir': '/projects/ml/mgnify_distill_rf3/', │ │ │ │ 'na_complex_distillation_data_dir': '/projects/ml/prot_dna/rf3_newDL', 'na_complex_distillation_parquet_dir': │ │ │ │ '/projects/ml/prot_dna', 'disorder_distill_parquet_dir': '/projects/ml/disorder_distill', 'protein_msa_dirs': │ │ │ │ [{'dir': '/projects/msa/rf2aa_af3/rf2aa_paper_model_protein_msas', 'extension': '.a3m.gz', 'directory_depth': 2}, │ │ │ │ {'dir': '/projects/msa/rf2aa_af3/missing_msas_through_2024_08_12', 'extension': '.msa0.a3m.gz', 'directory_depth': │ │ │ │ 2}, {'dir': '/projects/msa/mmseqs_gpu', 'extension': '.a3m.gz', 'directory_depth': 2}], 'rna_msa_dirs': [{'dir': │ │ │ │ '/projects/msa/rf2aa_af3/rf2aa_paper_model_rna_msas', 'extension': '.afa', 'directory_depth': 0}], │ │ │ │ 'failed_examples_dir': None}}, 'datasets': {'diffusion_batch_size_train': 32, 'diffusion_batch_size_inference': 4, │ │ │ │ 'n_recycles_train': 4, 'n_recycles_validation': 10, 'run_confidence_head': True, 'p_unconditional': 0.9, │ │ │ │ 'p_dropout_atom_level_embeddings': 0.333, 'p_dropout_ref_conf': 0.333, 'n_msa': 1024, 'crop_size': 768, │ │ │ │ 'max_atoms_in_crop': 8000, 'key_to_balance': 'n_tokens_total', 'take_first_chiral_subordering': False, │ │ │ │ 'use_element_for_atom_names_of_atomized_tokens': False, 'mirror_prob': 0.02, 'atomization_prob': 0.02, │ │ │ │ 'ligand_dropout_prob': 0.0, 'add_residue_is_paired_feature': True, 'train': {'pdb': {'sub_datasets': {'interface': │ │ │ │ {'dataset': {'target': 'atomworks.ml.datasets.datasets.StructuralDatasetWrapper', 'save_failed_examples_to_dir': │ │ │ │ None, 'cif_parser_args': {'cache_dir': '/net/scratch/dimaio/rf3_cache/', 'load_from_cache': True, 'save_to_cache': │ │ │ │ True}, 'dataset': {'target': 'atomworks.ml.datasets.datasets.PandasDataset', 'id_column': 'example_id', 'name': │ │ │ │ 'interface', 'data': '/projects/ml/datahub/dfs/af3_splits/2025_07_13/interfaces_df_train.parquet', 'filters': │ │ │ │ ["deposition_date < '2024-01-01'", 'resolution < 9.0', 'num_polymer_pn_units <= 300', 'cluster.notnull()', │ │ │ │ "~(pn_unit_1_non_polymer_res_names.notnull() and │ │ │ │ pn_unit_1_non_polymer_res_names.str.contains('(?:^|,)\s*(?:144|15P|1PE|2F2|2JC|3HR|3SY|7N5|7PE|9JE|AAE|ABA|ACE|A… │ │ │ │ regex=True))", "~(pn_unit_2_non_polymer_res_names.notnull() and │ │ │ │ pn_unit_2_non_polymer_res_names.str.contains('(?:^|,)\s*(?:144|15P|1PE|2F2|2JC|3HR|3SY|7N5|7PE|9JE|AAE|ABA|ACE|A… │ │ │ │ regex=True))", 'is_inter_molecule', "pdb_id not in ['1lqt','7wno']"], 'columns_to_load': ['example_id', 'pdb_id', │ │ │ │ 'assembly_id', 'deposition_date', 'resolution', 'num_polymer_pn_units', 'method', 'cluster', 'n_prot', 'n_nuc', │ │ │ │ 'n_ligand', 'n_peptide', 'total_num_atoms_in_unprocessed_assembly', 'pn_unit_1_iid', 'pn_unit_2_iid', │ │ │ │ 'pn_unit_1_non_polymer_res_names', 'pn_unit_2_non_polymer_res_names', 'is_inter_molecule', │ │ │ │ 'all_pn_unit_iids_after_processing', 'involves_loi']}, 'return_key': None, 'transform': {'target': │ │ │ │ 'rf3.data.pipelines.build_af3_transform_pipeline', 'is_inference': False, 'protein_msa_dirs': [{'dir': │ │ │ │ '/projects/msa/rf2aa_af3/rf2aa_paper_model_protein_msas', 'extension': '.a3m.gz', 'directory_depth': 2}, {'dir': │ │ │ │ '/projects/msa/rf2aa_af3/missing_msas_through_2024_08_12', 'extension': '.msa0.a3m.gz', 'directory_depth': 2}, │ │ │ │ {'dir': '/projects/msa/mmseqs_gpu', 'extension': '.a3m.gz', 'directory_depth': 2}], 'rna_msa_dirs': [{'dir': │ │ │ │ '/projects/msa/rf2aa_af3/rf2aa_paper_model_rna_msas', 'extension': '.afa', 'directory_depth': 0}], │ │ │ │ 'add_residue_is_paired_feature': True, 'n_recycles': 4, 'crop_size': 768, 'n_msa': 1024, 'diffusion_batch_size': │ │ │ │ 32, 'max_atoms_in_crop': 8000, 'run_confidence_head': True, 'p_unconditional': 0.9, │ │ │ │ 'p_dropout_atom_level_embeddings': 0.333, 'p_dropout_ref_conf': 0.333, 'take_first_chiral_subordering': False, │ │ │ │ 'use_element_for_atom_names_of_atomized_tokens': False, 'mirror_prob': 0.02, 'atomization_prob': 0.02, │ │ │ │ 'ligand_dropout_prob': 0.0, 'add_cyclic_bonds': True, 'crop_contiguous_probability': 0.0, │ │ │ │ 'crop_spatial_probability': 1.0}, 'dataset_parser': {'target': │ │ │ │ 'atomworks.ml.datasets.parsers.default_metadata_row_parsers.InterfacesDFParser', 'base_dir': │ │ │ │ '/projects/ml/frozen_pdb_copies/2025_07_13_pdb'}}, 'weights': {'target': │ │ │ │ 'atomworks.ml.samplers.calculate_weights_for_pdb_dataset_df', 'alphas': {'a_prot': 3.0, 'a_nuc': 3.0, 'a_ligand': │ │ │ │ 1.0, 'a_loi': 5.0}, 'beta': 1.0}}, 'pn_unit': {'dataset': {'target': │ │ │ │ 'atomworks.ml.datasets.datasets.StructuralDatasetWrapper', 'save_failed_examples_to_dir': None, 'cif_parser_args': │ │ │ │ {'cache_dir': '/net/scratch/dimaio/rf3_cache/', 'load_from_cache': True, 'save_to_cache': True}, 'dataset': │ │ │ │ {'target': 'atomworks.ml.datasets.datasets.PandasDataset', 'id_column': 'example_id', 'name': 'pn_unit', 'data': │ │ │ │ '/projects/ml/datahub/dfs/af3_splits/2025_07_13/pn_units_df_train.parquet', 'filters': ["deposition_date < │ │ │ │ '2024-01-01'", 'resolution < 9.0', 'num_polymer_pn_units <= 300', 'cluster.notnull()', │ │ │ │ "~(q_pn_unit_non_polymer_res_names.notnull() and │ │ │ │ q_pn_unit_non_polymer_res_names.str.contains('(?:^|,)\s*(?:144|15P|1PE|2F2|2JC|3HR|3SY|7N5|7PE|9JE|AAE|ABA|ACE|A… │ │ │ │ regex=True))", "pdb_id not in ['1lqt','7wno']"], 'columns_to_load': ['example_id', 'pdb_id', 'assembly_id', │ │ │ │ 'deposition_date', 'resolution', 'num_polymer_pn_units', 'method', 'cluster', 'n_prot', 'n_nuc', 'n_ligand', │ │ │ │ 'n_peptide', 'total_num_atoms_in_unprocessed_assembly', 'q_pn_unit_iid', 'q_pn_unit_non_polymer_res_names', │ │ │ │ 'all_pn_unit_iids_after_processing', 'q_pn_unit_is_loi']}, 'return_key': None, 'transform': {'target': │ │ │ │ 'rf3.data.pipelines.build_af3_transform_pipeline', 'is_inference': False, 'protein_msa_dirs': [{'dir': │ │ │ │ '/projects/msa/rf2aa_af3/rf2aa_paper_model_protein_msas', 'extension': '.a3m.gz', 'directory_depth': 2}, {'dir': │ │ │ │ '/projects/msa/rf2aa_af3/missing_msas_through_2024_08_12', 'extension': '.msa0.a3m.gz', 'directory_depth': 2}, │ │ │ │ {'dir': '/projects/msa/mmseqs_gpu', 'extension': '.a3m.gz', 'directory_depth': 2}], 'rna_msa_dirs': [{'dir': │ │ │ │ '/projects/msa/rf2aa_af3/rf2aa_paper_model_rna_msas', 'extension': '.afa', 'directory_depth': 0}], │ │ │ │ 'add_residue_is_paired_feature': True, 'n_recycles': 4, 'crop_size': 768, 'n_msa': 1024, 'diffusion_batch_size': │ │ │ │ 32, 'max_atoms_in_crop': 8000, 'run_confidence_head': True, 'p_unconditional': 0.9, │ │ │ │ 'p_dropout_atom_level_embeddings': 0.333, 'p_dropout_ref_conf': 0.333, 'take_first_chiral_subordering': False, │ │ │ │ 'use_element_for_atom_names_of_atomized_tokens': False, 'mirror_prob': 0.02, 'atomization_prob': 0.02, │ │ │ │ 'ligand_dropout_prob': 0.0, 'add_cyclic_bonds': True, 'crop_contiguous_probability': 0.3333333333333333, │ │ │ │ 'crop_spatial_probability': 0.6666666666666667}, 'dataset_parser': {'target': │ │ │ │ 'atomworks.ml.datasets.parsers.default_metadata_row_parsers.PNUnitsDFParser', 'base_dir': │ │ │ │ '/projects/ml/frozen_pdb_copies/2025_07_13_pdb'}}, 'weights': {'target': │ │ │ │ 'atomworks.ml.samplers.calculate_weights_for_pdb_dataset_df', 'alphas': {'a_prot': 3.0, 'a_nuc': 3.0, 'a_ligand': │ │ │ │ 1.0, 'a_loi': 5.0}, 'beta': 0.5}}}, 'probability': 0.5}, 'monomer_distillation': {'dataset': {'target': │ │ │ │ 'atomworks.ml.datasets.datasets.StructuralDatasetWrapper', 'save_failed_examples_to_dir': None, 'cif_parser_args': │ │ │ │ {'cache_dir': None, 'load_from_cache': False, 'save_to_cache': False}, 'dataset_parser': {'target': │ │ │ │ 'atomworks.ml.datasets.parsers.default_metadata_row_parsers.GenericDFParser', 'pn_unit_iid_colnames': None}, │ │ │ │ 'dataset': {'target': 'atomworks.ml.datasets.datasets.PandasDataset', 'name': 'af2fb_distillation', 'id_column': │ │ │ │ 'example_id', 'data': │ │ │ │ '/projects/ml/datahub/dfs/distillation/af2_distillation_facebook/af2_distillation_facebook.parquet', │ │ │ │ 'columns_to_load': ['example_id', 'path']}, 'return_key': None, 'transform': {'target': │ │ │ │ 'rf3.data.pipelines.build_af3_transform_pipeline', 'is_inference': False, 'protein_msa_dirs': [{'dir': │ │ │ │ '/squash/af2_distillation_facebook/msa', 'extension': '.a3m', 'directory_depth': 2}], 'rna_msa_dirs': [], │ │ │ │ 'add_residue_is_paired_feature': True, 'n_recycles': 4, 'crop_size': 768, 'n_msa': 1024, 'diffusion_batch_size': │ │ │ │ 32, 'max_atoms_in_crop': 8000, 'crop_contiguous_probability': 0.25, 'crop_spatial_probability': 0.75, │ │ │ │ 'b_factor_min': 70.0, 'run_confidence_head': True, 'take_first_chiral_subordering': False, │ │ │ │ 'use_element_for_atom_names_of_atomized_tokens': False, 'mirror_prob': 0.02, 'atomization_prob': 0.02, │ │ │ │ 'ligand_dropout_prob': 0.0, 'p_unconditional': 0.9, 'p_dropout_atom_level_embeddings': 0.333, │ │ │ │ 'p_dropout_ref_conf': 0.333}}, 'probability': 0.1}, 'mgnify_distillation_small': {'dataset': {'target': │ │ │ │ 'atomworks.ml.datasets.datasets.StructuralDatasetWrapper', 'save_failed_examples_to_dir': None, 'cif_parser_args': │ │ │ │ {'cache_dir': None, 'load_from_cache': False, 'save_to_cache': False}, 'dataset_parser': {'target': │ │ │ │ 'atomworks.ml.datasets.parsers.default_metadata_row_parsers.GenericDFParser', 'pn_unit_iid_colnames': None}, │ │ │ │ 'dataset': {'target': 'atomworks.ml.datasets.datasets.PandasDataset', 'name': 'mgnify_distillation_small', │ │ │ │ 'id_column': 'example_id', 'data': '/projects/ml/mgnify_distill_rf3//mgnify_distillation_small.parquet', │ │ │ │ 'columns_to_load': ['example_id', 'path']}, 'return_key': None, 'transform': {'target': │ │ │ │ 'rf3.data.pipelines.build_af3_transform_pipeline', 'is_inference': False, 'protein_msa_dirs': [{'dir': │ │ │ │ '/squash/mgnify_distill_rf3//msas', 'extension': '.a3m.gz', 'directory_depth': 2}], 'rna_msa_dirs': [], │ │ │ │ 'add_residue_is_paired_feature': True, 'n_recycles': 4, 'crop_size': 768, 'n_msa': 1024, 'diffusion_batch_size': │ │ │ │ 32, 'max_atoms_in_crop': 8000, 'crop_contiguous_probability': 0.25, 'crop_spatial_probability': 0.75, │ │ │ │ 'b_factor_min': 0.7, 'run_confidence_head': True, 'take_first_chiral_subordering': False, │ │ │ │ 'use_element_for_atom_names_of_atomized_tokens': False, 'mirror_prob': 0.02, 'atomization_prob': 0.02, │ │ │ │ 'ligand_dropout_prob': 0.0, 'p_unconditional': 0.9, 'p_dropout_atom_level_embeddings': 0.333, │ │ │ │ 'p_dropout_ref_conf': 0.333}}, 'probability': 0.03}, 'mgnify_distillation_big': {'dataset': {'target': │ │ │ │ 'atomworks.ml.datasets.datasets.StructuralDatasetWrapper', 'save_failed_examples_to_dir': None, 'cif_parser_args': │ │ │ │ {'cache_dir': None, 'load_from_cache': False, 'save_to_cache': False}, 'dataset_parser': {'target': │ │ │ │ 'atomworks.ml.datasets.parsers.default_metadata_row_parsers.GenericDFParser', 'pn_unit_iid_colnames': None}, │ │ │ │ 'dataset': {'target': 'atomworks.ml.datasets.datasets.PandasDataset', 'name': 'mgnify_distillation_big', │ │ │ │ 'id_column': 'example_id', 'data': '/projects/ml/mgnify_distill_rf3//mgnify_distillation_big.parquet', │ │ │ │ 'columns_to_load': ['example_id', 'path']}, 'return_key': None, 'transform': {'target': │ │ │ │ 'rf3.data.pipelines.build_af3_transform_pipeline', 'is_inference': False, 'protein_msa_dirs': [{'dir': │ │ │ │ '/squash/mgnify_distill_rf3//msas', 'extension': '.a3m.gz', 'directory_depth': 2}], 'rna_msa_dirs': [], │ │ │ │ 'add_residue_is_paired_feature': True, 'n_recycles': 4, 'crop_size': 768, 'n_msa': 1024, 'diffusion_batch_size': │ │ │ │ 32, 'max_atoms_in_crop': 8000, 'crop_contiguous_probability': 0.25, 'crop_spatial_probability': 0.75, │ │ │ │ 'b_factor_min': 0.7, 'run_confidence_head': True, 'take_first_chiral_subordering': False, │ │ │ │ 'use_element_for_atom_names_of_atomized_tokens': False, 'mirror_prob': 0.02, 'atomization_prob': 0.02, │ │ │ │ 'ligand_dropout_prob': 0.0, 'p_unconditional': 0.9, 'p_dropout_atom_level_embeddings': 0.333, │ │ │ │ 'p_dropout_ref_conf': 0.333}}, 'probability': 0.3}, 'na_complex_distillation': {'dataset': {'target': │ │ │ │ 'atomworks.ml.datasets.datasets.StructuralDatasetWrapper', 'save_failed_examples_to_dir': None, 'cif_parser_args': │ │ │ │ {'cache_dir': None, 'load_from_cache': False, 'save_to_cache': False}, 'dataset_parser': {'target': │ │ │ │ 'atomworks.ml.datasets.parsers.default_metadata_row_parsers.GenericDFParser', 'pn_unit_iid_colnames': None}, │ │ │ │ 'dataset': {'target': 'atomworks.ml.datasets.datasets.PandasDataset', 'name': 'tf_distillation', 'id_column': │ │ │ │ 'example_id', 'data': '/projects/ml/prot_dna/transcriptionFactor_distillation_rf3.newDL.csv', 'columns_to_load': │ │ │ │ ['example_id', 'path']}, 'return_key': None, 'transform': {'target': │ │ │ │ 'rf3.data.pipelines.build_af3_transform_pipeline', 'is_inference': False, 'protein_msa_dirs': [{'dir': │ │ │ │ '/projects/ml/prot_dna/rf3_newDL/a3m/', 'extension': '.a3m', 'directory_depth': 1}], 'rna_msa_dirs': [], │ │ │ │ 'add_residue_is_paired_feature': True, 'n_recycles': 4, 'crop_size': 768, 'n_msa': 1024, 'diffusion_batch_size': │ │ │ │ 32, 'max_atoms_in_crop': 8000, 'crop_contiguous_probability': 0.25, 'crop_spatial_probability': 0.75, │ │ │ │ 'pad_dna_p_skip': 0.0, 'run_confidence_head': True, 'take_first_chiral_subordering': False, │ │ │ │ 'use_element_for_atom_names_of_atomized_tokens': False, 'mirror_prob': 0.0, 'atomization_prob': 0.02, │ │ │ │ 'ligand_dropout_prob': 0.0, 'p_unconditional': 0.9, 'p_dropout_atom_level_embeddings': 0.333, │ │ │ │ 'p_dropout_ref_conf': 0.333}}, 'probability': 0.02}, 'disorder_distillation': {'dataset': {'target': │ │ │ │ 'atomworks.ml.datasets.datasets.StructuralDatasetWrapper', 'save_failed_examples_to_dir': None, 'cif_parser_args': │ │ │ │ {'cache_dir': None, 'load_from_cache': False, 'save_to_cache': False}, 'dataset_parser': {'target': │ │ │ │ 'atomworks.ml.datasets.parsers.default_metadata_row_parsers.GenericDFParser', 'pn_unit_iid_colnames': None}, │ │ │ │ 'dataset': {'target': 'atomworks.ml.datasets.datasets.PandasDataset', 'name': 'pdb_diso_distillation', │ │ │ │ 'id_column': 'example_id', 'data': '/projects/ml/disorder_distill/disorderDistillation.csv', 'columns_to_load': │ │ │ │ ['example_id', 'path']}, 'return_key': None, 'transform': {'target': │ │ │ │ 'rf3.data.pipelines.build_af3_transform_pipeline', 'is_inference': False, 'protein_msa_dirs': [{'dir': │ │ │ │ '/projects/msa/rf2aa_af3/rf2aa_paper_model_protein_msas', 'extension': '.a3m.gz', 'directory_depth': 2}, {'dir': │ │ │ │ '/projects/msa/rf2aa_af3/missing_msas_through_2024_08_12', 'extension': '.msa0.a3m.gz', 'directory_depth': 2}, │ │ │ │ {'dir': '/projects/msa/mmseqs_gpu', 'extension': '.a3m.gz', 'directory_depth': 2}], 'rna_msa_dirs': [{'dir': │ │ │ │ '/projects/msa/rf2aa_af3/rf2aa_paper_model_rna_msas', 'extension': '.afa', 'directory_depth': 0}], │ │ │ │ 'add_residue_is_paired_feature': True, 'n_recycles': 4, 'crop_size': 768, 'n_msa': 1024, 'diffusion_batch_size': │ │ │ │ 32, 'max_atoms_in_crop': 8000, 'crop_contiguous_probability': 0.25, 'crop_spatial_probability': 0.75, │ │ │ │ 'run_confidence_head': True, 'take_first_chiral_subordering': False, │ │ │ │ 'use_element_for_atom_names_of_atomized_tokens': False, 'mirror_prob': 0.02, 'atomization_prob': 0.02, │ │ │ │ 'ligand_dropout_prob': 0.0, 'p_unconditional': 0.9, 'p_dropout_atom_level_embeddings': 0.333, │ │ │ │ 'p_dropout_ref_conf': 0.333}}, 'probability': 0.02}, 'multidomain_distillation': {'dataset': {'target': │ │ │ │ 'rf3.data.paired_msa.MultiInputDatasetWrapper', 'save_failed_examples_to_dir': None, 'cif_parser_args': │ │ │ │ {'cache_dir': None, 'load_from_cache': False, 'save_to_cache': False}, 'dataset_parser': {'target': │ │ │ │ 'rf3.data.paired_msa.MultidomainDFParser'}, 'dataset': {'target': │ │ │ │ 'atomworks.ml.datasets.datasets.PandasDataset', 'name': 'multidomain_distillation', 'id_column': 'example_id', │ │ │ │ 'data': '/home/dimaio/RF3-dataset-processing/qian_dataset/domain_domain_dataset.DIGS.parquet', 'columns_to_load': │ │ │ │ ['example_id', 'pdb_path', 'msa_path']}, 'return_key': None, 'transform': {'target': │ │ │ │ 'rf3.data.pipelines.build_af3_transform_pipeline', 'is_inference': False, 'input_contains_explicit_msa': True, │ │ │ │ 'protein_msa_dirs': [], 'rna_msa_dirs': [], 'add_residue_is_paired_feature': True, 'n_recycles': 4, 'crop_size': │ │ │ │ 768, 'n_msa': 1024, 'diffusion_batch_size': 32, 'max_atoms_in_crop': 8000, 'crop_contiguous_probability': 0.25, │ │ │ │ 'crop_spatial_probability': 0.75, 'run_confidence_head': True, 'take_first_chiral_subordering': False, │ │ │ │ 'use_element_for_atom_names_of_atomized_tokens': False, 'mirror_prob': 0.0, 'atomization_prob': 0.02, │ │ │ │ 'ligand_dropout_prob': 0.0, 'p_unconditional': 0.9, 'p_dropout_atom_level_embeddings': 0.333, │ │ │ │ 'p_dropout_ref_conf': 0.333}}, 'probability': 0.02}, 'rna_monomer_distillation': {'dataset': {'target': │ │ │ │ 'atomworks.ml.datasets.datasets.StructuralDatasetWrapper', 'save_failed_examples_to_dir': None, 'cif_parser_args': │ │ │ │ {'cache_dir': None, 'load_from_cache': False, 'save_to_cache': False}, 'dataset_parser': {'target': │ │ │ │ 'atomworks.ml.datasets.parsers.default_metadata_row_parsers.GenericDFParser', 'pn_unit_iid_colnames': None}, │ │ │ │ 'dataset': {'target': 'atomworks.ml.datasets.datasets.PandasDataset', 'name': 'rna_monomer_distillation', │ │ │ │ 'id_column': 'example_id', 'data': '/projects/ml/afavor/rna_distillation/rna_distillation_filtered_df.parquet', │ │ │ │ 'columns_to_load': ['example_id', 'path', 'cluster_id', 'seq_hash', 'overall_plddt', 'overall_pde', │ │ │ │ 'overall_pae']}, 'return_key': None, 'transform': {'target': 'rf3.data.pipelines.build_af3_transform_pipeline', │ │ │ │ 'is_inference': False, 'protein_msa_dirs': [], 'rna_msa_dirs': [{'dir': │ │ │ │ '/projects/ml/afavor/rna_distillation/all_MSAs_renamed', 'extension': '.afa', 'directory_depth': 2}], │ │ │ │ 'add_residue_is_paired_feature': True, 'n_recycles': 4, 'crop_size': 768, 'n_msa': 1024, 'diffusion_batch_size': │ │ │ │ 32, 'max_atoms_in_crop': 8000, 'crop_contiguous_probability': 1.0, 'crop_spatial_probability': 0.0, │ │ │ │ 'pad_dna_p_skip': 0.0, 'b_factor_min': 0.6, 'run_confidence_head': True, 'take_first_chiral_subordering': False, │ │ │ │ 'use_element_for_atom_names_of_atomized_tokens': False, 'mirror_prob': 0.0, 'atomization_prob': 0.02, │ │ │ │ 'ligand_dropout_prob': 0.0, 'p_unconditional': 0.9, 'p_dropout_atom_level_embeddings': 0.333, │ │ │ │ 'p_dropout_ref_conf': 0.333}}, 'probability': 0.01}}, 'val': {'af3_validation': {'dataset': {'target': │ │ │ │ 'atomworks.ml.datasets.datasets.StructuralDatasetWrapper', 'save_failed_examples_to_dir': None, 'cif_parser_args': │ │ │ │ {'cache_dir': None, 'load_from_cache': False, 'save_to_cache': False}, 'dataset': {'target': │ │ │ │ 'atomworks.ml.datasets.datasets.PandasDataset', 'id_column': 'example_id', 'data': │ │ │ │ '/projects/ml/datahub/dfs/af3_splits/2025_07_13/entry_level_val_df.parquet'}, 'return_key': None, 'transform': │ │ │ │ {'target': 'rf3.data.pipelines.build_af3_transform_pipeline', 'is_inference': True, 'protein_msa_dirs': [{'dir': │ │ │ │ '/projects/msa/rf2aa_af3/rf2aa_paper_model_protein_msas', 'extension': '.a3m.gz', 'directory_depth': 2}, {'dir': │ │ │ │ '/projects/msa/rf2aa_af3/missing_msas_through_2024_08_12', 'extension': '.msa0.a3m.gz', 'directory_depth': 2}, │ │ │ │ {'dir': '/projects/msa/mmseqs_gpu', 'extension': '.a3m.gz', 'directory_depth': 2}], 'rna_msa_dirs': [{'dir': │ │ │ │ '/projects/msa/rf2aa_af3/rf2aa_paper_model_rna_msas', 'extension': '.afa', 'directory_depth': 0}], │ │ │ │ 'add_residue_is_paired_feature': True, 'n_recycles': 10, 'crop_size': None, 'n_msa': 1024, 'diffusion_batch_size': │ │ │ │ 4, 'max_atoms_in_crop': None, 'return_atom_array': True, 'run_confidence_head': True, 'p_unconditional': 1.0, │ │ │ │ 'p_dropout_atom_level_embeddings': 0.0, 'p_dropout_ref_conf': 0.0, 'take_first_chiral_subordering': False, │ │ │ │ 'use_element_for_atom_names_of_atomized_tokens': False}, 'dataset_parser': {'target': │ │ │ │ 'atomworks.ml.datasets.parsers.custom_metadata_row_parsers.ValidationDFParserLikeAF3', 'base_dir': │ │ │ │ '/projects/ml/frozen_pdb_copies/2025_07_13_pdb'}}, 'key_to_balance': 'n_tokens_total'}}, 'pipeline_target': │ │ │ │ 'foundry.data.pipelines.build_af3_transform_pipeline'}, 'dataloader': {'train': {'dataloader_params': │ │ │ │ {'batch_size': 1, 'num_workers': 2, 'prefetch_factor': 3}, 'n_fallback_retries': 4}, 'val': {'dataloader_params': │ │ │ │ {'batch_size': 1, 'num_workers': 2, 'prefetch_factor': 3}, 'n_fallback_retries': 0}}, 'model': {'optimizer': │ │ │ │ {'target': 'torch.optim.Adam', 'lr': 0, 'betas': [0.9, 0.95], 'eps': 1e-08}, 'lr_scheduler': {'target': │ │ │ │ 'rf3.training.schedulers.AF3Scheduler', 'base_lr': 0.0018, 'warmup_steps': 1000, 'decay_factor': 0.95, │ │ │ │ 'decay_steps': 50000}, 'ema': {'decay': 0.999}, 'net': {'target': 'rf3.model.RF3.RF3WithConfidence', 'c_s': 384, │ │ │ │ 'c_z': 128, 'c_atom': 128, 'c_atompair': 16, 'c_s_inputs': 449, 'feature_initializer': {'input_feature_embedder': │ │ │ │ {'features': ['restype', 'profile', 'deletion_mean'], 'atom_attention_encoder': {'c_token': 384, │ │ │ │ 'c_atom_1d_features': 393, 'c_tokenpair': 128, 'use_inv_dist_squared': True, 'atom_1d_features': ['ref_pos', │ │ │ │ 'ref_charge', 'ref_mask', 'ref_element', 'ref_atom_name_chars', 'ref_pos_ground_truth', │ │ │ │ 'has_atom_level_embedding'], 'atom_transformer': {'n_queries': 32, 'n_keys': 128, 'l_max': 40000, │ │ │ │ 'diffusion_transformer': {'n_block': 3, 'diffusion_transformer_block': {'n_head': 4, │ │ │ │ 'no_residual_connection_between_attention_and_transition': True, 'kq_norm': True}}}, 'use_atom_level_embedding': │ │ │ │ True, 'atom_level_embedding_dim': 384}}, 'relative_position_encoding': {'r_max': 32, 's_max': 2}}, 'recycler': │ │ │ │ {'n_pairformer_blocks': 48, 'pairformer_block': {'p_drop': 0.25, 'triangle_multiplication': {'d_hidden': 128}, │ │ │ │ 'triangle_attention': {'n_head': 4, 'd_hidden': 32}, 'attention_pair_bias': {'n_head': 16}}, 'template_embedder': │ │ │ │ {'n_block': 2, 'raw_template_dim': 66, 'c': 64, 'p_drop': 0.25, 'use_fourier_encoding': False}, 'msa_module': │ │ │ │ {'n_block': 4, 'c_m': 64, 'p_drop_msa': 0.15, 'p_drop_pair': 0.25, 'msa_subsample_embedder': {'num_sequences': │ │ │ │ 1024, 'dim_raw_msa': 35, 'c_s_inputs': 449, 'c_msa_embed': 64}, 'outer_product': {'c_msa_embed': 64, │ │ │ │ 'c_outer_product': 32, 'c_out': 128}, 'msa_pair_weighted_averaging': {'n_heads': 8, 'c_weighted_average': 32, │ │ │ │ 'c_msa_embed': 64, 'c_z': 128, 'separate_gate_for_every_channel': True}, 'msa_transition': {'n': 4, 'c': 64}, │ │ │ │ 'triangle_multiplication_outgoing': {'d_pair': 128, 'd_hidden': 128, 'bias': True}, │ │ │ │ 'triangle_multiplication_incoming': {'d_pair': 128, 'd_hidden': 128, 'bias': True}, 'triangle_attention_starting': │ │ │ │ {'d_pair': 128, 'n_head': 4, 'd_hidden': 32, 'p_drop': 0.0}, 'triangle_attention_ending': {'d_pair': 128, │ │ │ │ 'n_head': 4, 'd_hidden': 32, 'p_drop': 0.0}, 'pair_transition': {'n': 4, 'c': 128}}}, 'diffusion_module': │ │ │ │ {'sigma_data': 16, 'c_token': 768, 'f_pred': 'edm', 'diffusion_conditioning': {'c_s_inputs': 449, 'c_t_embed': │ │ │ │ 256, 'relative_position_encoding': {'r_max': 32, 's_max': 2}}, 'atom_attention_encoder': {'c_tokenpair': 128, │ │ │ │ 'c_atom_1d_features': 393, 'use_inv_dist_squared': True, 'atom_1d_features': ['ref_pos', 'ref_charge', 'ref_mask', │ │ │ │ 'ref_element', 'ref_atom_name_chars', 'ref_pos_ground_truth', 'has_atom_level_embedding'], 'atom_transformer': │ │ │ │ {'n_queries': 32, 'n_keys': 128, 'l_max': 40000, 'diffusion_transformer': {'n_block': 3, │ │ │ │ 'diffusion_transformer_block': {'n_head': 4, 'no_residual_connection_between_attention_and_transition': True, │ │ │ │ 'kq_norm': True}}}, 'broadcast_trunk_feats_on_1dim_old': False, 'use_chiral_features': True, │ │ │ │ 'no_grad_on_chiral_center': False, 'use_atom_level_embedding': True, 'atom_level_embedding_dim': 384}, │ │ │ │ 'diffusion_transformer': {'n_block': 24, 'diffusion_transformer_block': {'n_head': 16, │ │ │ │ 'no_residual_connection_between_attention_and_transition': True, 'kq_norm': True}}, 'atom_attention_decoder': │ │ │ │ {'atom_transformer': {'n_queries': 32, 'n_keys': 128, 'l_max': 40000, 'diffusion_transformer': {'n_block': 3, │ │ │ │ 'diffusion_transformer_block': {'n_head': 4, 'no_residual_connection_between_attention_and_transition': True, │ │ │ │ 'kq_norm': True}}}}}, 'distogram_head': {'bins': 65}, 'inference_sampler': {'solver': 'af3', 'num_timesteps': 200, │ │ │ │ 'min_t': 0, 'max_t': 1, 'sigma_data': 16, 's_min': 0.0004, 's_max': 160, 'p': 7, 'gamma_0': 0.8, 'gamma_min': 1.0, │ │ │ │ 'noise_scale': 1.003, 'step_scale': 1.5}, 'mini_rollout_sampler': {'solver': 'af3', 'num_timesteps': 20, 'min_t': │ │ │ │ 0, 'max_t': 1, 'sigma_data': 16, 's_min': 0.0004, 's_max': 160, 'p': 7, 'gamma_0': 0.8, 'gamma_min': 1.0, │ │ │ │ 'noise_scale': 1.003, 'step_scale': 1.5}, 'confidence_head': {'c_s': 384, 'c_z': 128, 'n_pairformer_layers': 4, │ │ │ │ 'pairformer': {'p_drop': 0.25, 'triangle_multiplication': {'d_hidden': 128}, 'triangle_attention': {'n_head': 4, │ │ │ │ 'd_hidden': 32}, 'attention_pair_bias': {'n_head': 16}}, 'n_bins_pae': 64, 'n_bins_pde': 64, 'n_bins_plddt': 50, │ │ │ │ 'n_bins_exp_resolved': 2, 'use_Cb_distances': False, 'use_af3_style_binning_and_final_layer_norms': True, │ │ │ │ 'symmetrize_Cb_logits': True}}}, 'name': 'af3', 'tags': ['af3'], 'task_name': 'train', 'project': 'af3', 'seed': │ │ │ │ 1, 'ckpt_path': None, 'ckpt_config': {'target': 'foundry.utils.weights.CheckpointConfig', 'path': │ │ │ │ '/home/dimaio/RF2-allatom-foundry/checkpoints/af3-run10-w-conf-ep909.pt', 'reset_optimizer': False}} │ │ │ │ } │ │ │ │ self = <rfd3.engine.RFD3InferenceEngine object at 0x7733844540e0> │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /home/jedi/miniconda3/envs/foundry/lib/python3.12/site-packages/hydra/_internal/instantiate/_instantiate2.py:226 in instantiate │ │ │ │ 223 │ │ convert = config.pop(_Keys.CONVERT, ConvertMode.NONE) │ │ 224 │ │ partial = config.pop(_Keys.PARTIAL, False) │ │ 225 │ │ │ │ ❱ 226 │ │ return instantiate_node( │ │ 227 │ │ │ config, *args, recursive=recursive, convert=convert, partial=partial │ │ 228 │ │ ) │ │ 229 │ elif OmegaConf.is_list(config): │ │ │ │ ╭──────────────────────────────────────────────────────────── locals ─────────────────────────────────────────────────────────────╮ │ │ │ convert = 'partial' │ │ │ │ partial = False │ │ │ │ recursive = False │ │ │ │ args = () │ │ │ │ config = {'strategy': 'ddp', 'accelerator': 'gpu', 'devices_per_node': 1, 'num_nodes': 1, 'loss': None, 'metrics': │ │ │ │ {'by_type_lddt': {'target': 'rf3.metrics.lddt.ByTypeLDDT'}, 'all_atom_lddt': {'target': │ │ │ │ 'rf3.metrics.lddt.AllAtomLDDT'}, 'distogram': {'target': 'rf3.metrics.distogram.DistogramLoss'}, │ │ │ │ 'distogram_comparisons': {'target': 'rf3.metrics.distogram.DistogramComparisons'}, 'distogram_entropy': │ │ │ │ {'target': 'rf3.metrics.distogram.DistogramEntropy'}, 'chiral_loss': {'target': │ │ │ │ 'rf3.metrics.chiral.ChiralLoss'}, 'unresolved_rasa': {'target': 'rf3.metrics.rasa.UnresolvedRegionRASA'}, │ │ │ │ 'ptm': {'target': 'rf3.metrics.predicted_error.ComputePTM'}, 'iptm': {'target': │ │ │ │ 'rf3.metrics.predicted_error.ComputeIPTM'}, 'count_clashing_chains': {'target': │ │ │ │ 'rf3.metrics.clashing_chains.CountClashingChains'}}, 'target': 'rf3.trainers.rf3.RF3TrainerWithConfidence', │ │ │ │ 'validate_every_n_epochs': 9999, 'max_epochs': 10000, 'n_examples_per_epoch': 8000, 'prevalidate': False, │ │ │ │ 'n_recycles_train': 4, 'clip_grad_max_norm': 10.0, 'output_dir': │ │ │ │ '/net/scratch/dimaio/training/logs/train/af3/2025-10-10_09-54_JOB_48324318', 'checkpoint_every_n_epochs': 1, │ │ │ │ 'precision': 'bf16-mixed', 'grad_accum_steps': 16, 'seed': None, 'cleanup_guideposts': True, │ │ │ │ 'cleanup_virtual_atoms': True, 'read_sequence_from_sequence_head': True, 'output_full_json': True} │ │ │ │ config_copy = {'strategy': 'ddp', 'accelerator': 'gpu', 'devices_per_node': 1, 'num_nodes': 1, 'loss': None, 'metrics': │ │ │ │ {'by_type_lddt': {'target': 'rf3.metrics.lddt.ByTypeLDDT'}, 'all_atom_lddt': {'target': │ │ │ │ 'rf3.metrics.lddt.AllAtomLDDT'}, 'distogram': {'target': 'rf3.metrics.distogram.DistogramLoss'}, │ │ │ │ 'distogram_comparisons': {'target': 'rf3.metrics.distogram.DistogramComparisons'}, 'distogram_entropy': │ │ │ │ {'target': 'rf3.metrics.distogram.DistogramEntropy'}, 'chiral_loss': {'target': │ │ │ │ 'rf3.metrics.chiral.ChiralLoss'}, 'unresolved_rasa': {'target': 'rf3.metrics.rasa.UnresolvedRegionRASA'}, │ │ │ │ 'ptm': {'target': 'rf3.metrics.predicted_error.ComputePTM'}, 'iptm': {'target': │ │ │ │ 'rf3.metrics.predicted_error.ComputeIPTM'}, 'count_clashing_chains': {'target': │ │ │ │ 'rf3.metrics.clashing_chains.CountClashingChains'}}, 'target': 'rf3.trainers.rf3.RF3TrainerWithConfidence', │ │ │ │ 'validate_every_n_epochs': 9999, 'max_epochs': 10000, 'n_examples_per_epoch': 8000, 'prevalidate': False, │ │ │ │ 'n_recycles_train': 4, 'clip_grad_max_norm': 10.0, 'output_dir': │ │ │ │ '/net/scratch/dimaio/training/logs/train/af3/2025-10-10_09-54_JOB_48324318', 'checkpoint_every_n_epochs': 1, │ │ │ │ 'precision': 'bf16-mixed', 'grad_accum_steps': 16, 'seed': None, 'cleanup_guideposts': True, │ │ │ │ 'cleanup_virtual_atoms': True, 'read_sequence_from_sequence_head': True, 'output_full_json': True} │ │ │ │ kwargs = {'convert': 'partial', 'recursive': False} │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /home/jedi/miniconda3/envs/foundry/lib/python3.12/site-packages/hydra/_internal/instantiate/_instantiate2.py:347 in │ │ instantiate_node │ │ │ │ 344 │ │ │ │ │ │ ) │ │ 345 │ │ │ │ │ kwargs[key] = _convert_node(value, convert) │ │ 346 │ │ │ │ │ ❱ 347 │ │ │ return _call_target(target, partial, args, kwargs, full_key) │ │ 348 │ │ else: │ │ 349 │ │ │ # If ALL or PARTIAL non structured or OBJECT non structured, │ │ 350 │ │ │ # instantiate in dict and resolve interpolations eagerly. │ │ │ │ ╭──────────────────────────────────────────────────────────── locals ─────────────────────────────────────────────────────────────╮ │ │ │ args = () │ │ │ │ convert = 'partial' │ │ │ │ exclude_keys = {'target', 'recursive', 'convert', 'partial'} │ │ │ │ full_key = 'trainer' │ │ │ │ is_partial = False │ │ │ │ key = 'output_full_json' │ │ │ │ kwargs = { │ │ │ │ │ 'strategy': 'ddp', │ │ │ │ │ 'accelerator': 'gpu', │ │ │ │ │ 'devices_per_node': 1, │ │ │ │ │ 'num_nodes': 1, │ │ │ │ │ 'loss': None, │ │ │ │ │ 'metrics': { │ │ │ │ │ │ 'by_type_lddt': {'target': 'rf3.metrics.lddt.ByTypeLDDT'}, │ │ │ │ │ │ 'all_atom_lddt': {'target': 'rf3.metrics.lddt.AllAtomLDDT'}, │ │ │ │ │ │ 'distogram': {'target': 'rf3.metrics.distogram.DistogramLoss'}, │ │ │ │ │ │ 'distogram_comparisons': {'target': 'rf3.metrics.distogram.DistogramComparisons'}, │ │ │ │ │ │ 'distogram_entropy': {'target': 'rf3.metrics.distogram.DistogramEntropy'}, │ │ │ │ │ │ 'chiral_loss': {'target': 'rf3.metrics.chiral.ChiralLoss'}, │ │ │ │ │ │ 'unresolved_rasa': {'target': 'rf3.metrics.rasa.UnresolvedRegionRASA'}, │ │ │ │ │ │ 'ptm': {'target': 'rf3.metrics.predicted_error.ComputePTM'}, │ │ │ │ │ │ 'iptm': {'target': 'rf3.metrics.predicted_error.ComputeIPTM'}, │ │ │ │ │ │ 'count_clashing_chains': {'target': 'rf3.metrics.clashing_chains.CountClashingChains'} │ │ │ │ │ }, │ │ │ │ │ 'validate_every_n_epochs': 9999, │ │ │ │ │ 'max_epochs': 10000, │ │ │ │ │ 'n_examples_per_epoch': 8000, │ │ │ │ │ 'prevalidate': False, │ │ │ │ │ ... +11 │ │ │ │ } │ │ │ │ node = {'strategy': 'ddp', 'accelerator': 'gpu', 'devices_per_node': 1, 'num_nodes': 1, 'loss': None, 'metrics': │ │ │ │ {'by_type_lddt': {'target': 'rf3.metrics.lddt.ByTypeLDDT'}, 'all_atom_lddt': {'target': │ │ │ │ 'rf3.metrics.lddt.AllAtomLDDT'}, 'distogram': {'target': 'rf3.metrics.distogram.DistogramLoss'}, │ │ │ │ 'distogram_comparisons': {'target': 'rf3.metrics.distogram.DistogramComparisons'}, 'distogram_entropy': │ │ │ │ {'target': 'rf3.metrics.distogram.DistogramEntropy'}, 'chiral_loss': {'target': │ │ │ │ 'rf3.metrics.chiral.ChiralLoss'}, 'unresolved_rasa': {'target': 'rf3.metrics.rasa.UnresolvedRegionRASA'}, │ │ │ │ 'ptm': {'target': 'rf3.metrics.predicted_error.ComputePTM'}, 'iptm': {'target': │ │ │ │ 'rf3.metrics.predicted_error.ComputeIPTM'}, 'count_clashing_chains': {'target': │ │ │ │ 'rf3.metrics.clashing_chains.CountClashingChains'}}, 'target': 'rf3.trainers.rf3.RF3TrainerWithConfidence', │ │ │ │ 'validate_every_n_epochs': 9999, 'max_epochs': 10000, 'n_examples_per_epoch': 8000, 'prevalidate': False, │ │ │ │ 'n_recycles_train': 4, 'clip_grad_max_norm': 10.0, 'output_dir': │ │ │ │ '/net/scratch/dimaio/training/logs/train/af3/2025-10-10_09-54_JOB_48324318', 'checkpoint_every_n_epochs': 1, │ │ │ │ 'precision': 'bf16-mixed', 'grad_accum_steps': 16, 'seed': None, 'cleanup_guideposts': True, │ │ │ │ 'cleanup_virtual_atoms': True, 'read_sequence_from_sequence_head': True, 'output_full_json': True} │ │ │ │ partial = False │ │ │ │ recursive = False │ │ │ │ value = True │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /home/jedi/miniconda3/envs/foundry/lib/python3.12/site-packages/hydra/_internal/instantiate/_instantiate2.py:97 in _call_target │ │ │ │ 94 │ │ │ msg = f"Error in call to target '{_convert_target_to_string(target)}':\n{r │ │ 95 │ │ │ if full_key: │ │ 96 │ │ │ │ msg += f"\nfull_key: {full_key}" │ │ ❱ 97 │ │ │ raise InstantiationException(msg) from e │ │ 98 │ │ 99 │ │ 100 def _convert_target_to_string(t: Any) -> Any: │ │ │ │ ╭─────────────────────────────────────────────────── locals ───────────────────────────────────────────────────╮ │ │ │ partial = False │ │ │ │ args = () │ │ │ │ full_key = 'trainer' │ │ │ │ kwargs = { │ │ │ │ │ 'strategy': 'ddp', │ │ │ │ │ 'accelerator': 'gpu', │ │ │ │ │ 'devices_per_node': 1, │ │ │ │ │ 'num_nodes': 1, │ │ │ │ │ 'loss': None, │ │ │ │ │ 'metrics': { │ │ │ │ │ │ 'by_type_lddt': {'target': 'rf3.metrics.lddt.ByTypeLDDT'}, │ │ │ │ │ │ 'all_atom_lddt': {'target': 'rf3.metrics.lddt.AllAtomLDDT'}, │ │ │ │ │ │ 'distogram': {'target': 'rf3.metrics.distogram.DistogramLoss'}, │ │ │ │ │ │ 'distogram_comparisons': {'target': 'rf3.metrics.distogram.DistogramComparisons'}, │ │ │ │ │ │ 'distogram_entropy': {'target': 'rf3.metrics.distogram.DistogramEntropy'}, │ │ │ │ │ │ 'chiral_loss': {'target': 'rf3.metrics.chiral.ChiralLoss'}, │ │ │ │ │ │ 'unresolved_rasa': {'target': 'rf3.metrics.rasa.UnresolvedRegionRASA'}, │ │ │ │ │ │ 'ptm': {'target': 'rf3.metrics.predicted_error.ComputePTM'}, │ │ │ │ │ │ 'iptm': {'target': 'rf3.metrics.predicted_error.ComputeIPTM'}, │ │ │ │ │ │ 'count_clashing_chains': { │ │ │ │ │ │ │ 'target': 'rf3.metrics.clashing_chains.CountClashingChains' │ │ │ │ │ │ } │ │ │ │ │ }, │ │ │ │ │ 'validate_every_n_epochs': 9999, │ │ │ │ │ 'max_epochs': 10000, │ │ │ │ │ 'n_examples_per_epoch': 8000, │ │ │ │ │ 'prevalidate': False, │ │ │ │ │ ... +11 │ │ │ │ } │ │ │ │ msg = 'Error in call to target 'rf3.trainers.rf3.RF3TrainerWithConfidence':\nTypeError("'+100 │ │ │ │ v = True │ │ │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ InstantiationException: Error in call to target 'rf3.trainers.rf3.RF3TrainerWithConfidence': TypeError("FabricTrainer.init() got an unexpected keyword argument 'cleanup_guideposts'") full_key: trainer
Hmm this seems like a strange error, are you loading rfd3 weights into rf3 by accident by chance?
Oops. I did exactly what you said. I loaded the wrong weight by accident. Thank you so much!!! The problem is solved now.