Adding basic input file previews when launching a calculation

[RaphaelRobidas]

Is your feature request related to a problem? Please describe. Sometimes, the input files produced are different from the expected results. To verify that they are correct, one must launch the calculations, then go to the calculation's page to see the input. If it is incorrect, the calculation must be killed and relaunched with correct parameters.

Describe the solution you'd like It would be very useful to have a preview of the input when launching a calculation. It could simply be a textarea at the bottom of the page. The preview would be updated through an AJAX call to the backend with the form data as parameter.

To generate the input, the form data must first be parsed and validated in the same manner as the submit_calculation and verify_calculation views (frontend/views.py). Some parameters are only parsed when submitting the calculation (_submit_calculation function) and I do not see any easy fix. These can be ignored for now.

Then, the parameters and calculation data are passed to the calc_to_ccinput function in frontend/tasks.py. This function normally takes a Calculation database object as parameter, but we don't want to create an object for each preview. As such, the preview could create a simple Python object with the appropriate fields to replace it.

[anchalm03]

Hello, I would like to work on this issue, can I ?

[RaphaelRobidas]

Great, go ahead!

[anchalm03]

Hello! I want to confirm, should the text area at the bottom be generated when "Submit" button is clicked, displaying the preview of the input using AJAX and then there should be one more "Confirm Calculation" button ?

Also, does this design for preview work?

[RaphaelRobidas]

There should be a separate button just to preview the input file. The top part is not really needed, since these fields will be filled in the form itself (calc_form_body.html). Just to be clear, the input file to be displayed is what gets stored in the input_file field of the Calculation class. For the xtb package, it will look something like this:

xtb in.xyz --opt tight --alpb ch2cl2 --chrg 1

Sometimes there are more lines also for specific options. This feature will be more useful for the other packages which have complex input files

[anchalm03]

Okay, I got it. Thanks for the clarification.

[anchalm03]

I need a few clarifications;

1. In my understanding, the parameters software, type, solvation_model, solvent, charge, specifications need to be validated before preview.Are there any other parameters of form data do we want to validate before preview?

2. What class does inp object belong to? (it's definition does not specify)

3. The verify_calculation and submit_calculation functions mainly depend on _submit_calculation to parse and validate form data.So, for preview_calculation, do I have to use the same _submit_calculation or make a new function similar to it ?

[RaphaelRobidas]

1. Yes, there are more parameters that need to be validated. However, you shouldn't have to worry too much about it, just use the functions that verify all the relevant parameters. If you look in calc_form_head.html, you will find the verify_form function which parses the relevant data. You can make a similar function (or refactor this one into a general data parsing part and a request-making part) in order to verify the data and get the input to display.
2. The inp object in the calc_to_ccinput function is a Calculation object from the ccinput project (https://github.com/cyllab/ccinput, in ccinput/calculation.py). It's a pure Python class that holds the parameters in a slightly different way.
3. You should reuse the same function.

[anchalm03]

Hello, Once I parse the data using _submit_calculation, from where can I obtain the validated data (Because the function returns redirect to calculations in views.py.), to generate input_file ?

[RaphaelRobidas]

Actually, you should use parse_parameters in order to get the parameters. That will give you most parameters you will need to construct a dummy Calculation object (not an actual database Calculation, but a Python object with a similar structure). This dummy object will be passed to calc_to_ccinput (in tasks.py) in order to get the input file to display.

[anchalm03]

I want to confirm; for the dummy object, a dummy class is to be created(I assume not in models.py), which file should that be done in (the one in which I am getting parameters from parse_parameters i.e views.py or somewhere else?)

[RaphaelRobidas]

@anchalm03 You can create the class in helpers.py, that seems to be the most logical location.

[anchalm03]

I need some more clarifictaions:

1. Can I define a new function instead of using calc_to_ccinput ?
2. If not, from where will the other parameters( like xyz_strucuture, calc.order, calc.local) required in the calc_to_ccinput come from as we are not using the data from the database?

[RaphaelRobidas]

@anchalm03

1. It would be best to use calc_to_ccinput, so the processing is exactly the same in the preview and in the real input generation.
2. I just had a crazy idea: in Django, you create an object in the database with obj = MyClass.objects.create(field1="blah", ...). However, you can create an object which is not saved to the database (until you explicitly save it): obj = MyClass(field1="blah"). Later, you can call obj.save() to commit it to the database, otherwise it will not be saved. And so, perhaps that _submit_calculation could create all the proper objects with the real models, but not save them to the database until the very end. When verifying the parameters or generating a preview, it would just have to return before saving the models. That would be a very elegant and efficient solution! To implement this, you would have to change all the object creations with Class.objects.create(...) by Class(...) and add the objects to an array of objects to save later, once all the processing is done. If you are creating an input file preview, you could pass a parameter to the function which makes it return the Calculation object instead of saving all the objects. You could use that instance in calc_to_ccinput, and it would all be gone after the view returns the input file. What you do think?

[anchalm03]

Yes, I think that should be a good solution. I will try working it out and then let you know difficulties (if any) in the same. Also, will I have to change all the object creations or just the Calculation objects and a few others involved in this procedure?

[RaphaelRobidas]

You should change all the object creations in _submit_calculation. You will notice that I have a verify parameter and some code blocks starting with if not verify:. I added this in order not to create the objects if we are just verifying the validity of the parameters. You can remove the condition but only create the objects in memory (MyClass(...) instead of MyClass.objects.create(...)).

At the end of the function, you should have all the objects only in memory, and then you can choose to simply return (if only verifying the parameters), generate an input file with the objects then return (if creating a preview) or saving all the objects to the database and actually launching the calculation.

[anchalm03]

There are some code blocks starting with if not verify that are not creating objects but are performing other tasks, so in that case also should I remove the condition?

[RaphaelRobidas]

Yes, but make sure that no object is created in another function call or something like that.

[anchalm03]

I tried doing it the way you explained, but it lead to an error in the initial part of the code itself, and I am facing trouble debugging it. I am attaching the screenshot here.

Can you please help?

[RaphaelRobidas]

@anchalm03 It seems like Django wants objects to be saved in a specific order. If you preserve the order in which objects were created (just appending to an array should work), you can try saving the objects in that order.

[anchalm03]

Hello I tried appending the objects in the exact same order, and still was getting the same error as above. Then I tried looking up in the postgres database, and I assume that it is a foreign key issue and some table requires data from other table while saving. Can you please help me with the order of saving the objects in the database?

[RaphaelRobidas]

Are you sure you did not miss any of the objects created in _submit_calculation? I tested a simple case and it works when the referenced object is saved:

calcus@web:/calcus$ python manage.py shell
Python 3.9.16 (main, Mar  1 2023, 15:45:29) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from frontend.models import *
>>> params = Parameters()
>>> obj = CalculationOrder(name="dummy order")
>>> obj.save()
>>> obj2 = CalculationOrder(name="dummy order", parameters=params)
>>> obj2.save()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 695, in save
    self._prepare_related_fields_for_save(operation_name='save')
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 945, in _prepare_related_fields_for_save
    raise ValueError(
ValueError: save() prohibited to prevent data loss due to unsaved related object 'parameters'.
>>> params.charge = 1
>>> params.multiplicity = 1
>>> params.save()
>>> obj2.save()
>>> obj2.refresh_from_db()
>>> obj2
<CalculationOrder: CalculationOrder object (1EvXwQvvDnady)>
>>> obj2.parameters
xtb - GFN2-xTB (vacuum)
>>> obj2.parameters.charge
1
>>> obj2.parameters.multiplicity
1

If every object is properly handled, you can push the code to your fork so I can have a look at it and test some things.