Copy Bug in Outline and Editor

[siliconserf]

This was fun. I'm running 0.6.0 released, not any in progress version. The bug showed this way:

I had a sub-folder in the Outline tree that I copied into another folder. This didn't raise any flags, no idication of lack of ID, or ID duplication. I was able to edit each of the new documents in the folder copy. Then I saved and exited. The next time I tried to open the program I got the familiar Item ID problem in Revisions showing up in the Mate Terminal. Clearing that by editing the Revisions document cleared the ID complaint. Good, except another issue showed up - the project loaded, but the program complained in terminal window about a series of "MS" documents with duplicate IDs.

After trying closing the project and reloading, I decided the problem wasn't going to go away on its own. So, I used the Debug tab to access and edit one set of the duplicate ID "MS" documents, runumbering them from a number a bit higher than the highest ID I could find. I saved this and the next load of the project showed no problems with duplicate IDs.

This was a copy operation within the same program instance, not cross program instances. I haven't seen it happen when just copying several documents in the same folder, but I could have missed it. But, the point is there is a serious hole in the copy/ID handling code.

[gedakc]

I confirmed this issue with the following steps:

1. Start manuskript and create a new project using the Novel template.
2. Go to the Outline mode and create a new folder under Chapter 1.
3. Select the New folder and create 3 new texts under the New folder.
4. Right click on the New folder and select Copy.
5. Select Chapter 2 and right click on it and select Paste.
6. Close Manusrkipt.
7. Start manuskript and open the project.
8. Note messages similar to the following:

$ bin/manuskript
Debug: Web rendering engine used: QWebView
Running manuskript version 0.6.0.
Note: No translator found or loaded for locale en_CA.
Loading: /home/gedakc/tmp/deleteme-2.msk
Detected file format version: 1. Zip: True.
WARNING ! There are some items with same IDs: ['18', '19', '20', '18', '19', '20']
Project /home/gedakc/tmp/deleteme-2.msk loaded.
Project deleteme-2.msk saved.

This problem also occurred with the latest develop branch.

Note that the same issue occurs when using Editor mode too.

[src-r-r]

@olivierkes I was looking at the design. I'd suggest, instead of using IDs, using UUIDs. This would (virtually) guarantee all items are unique. What are your thoughts? The only project overhead would be migrating the old ID code to UUIDs, which would just be a simple check on app launch. Because with the current design of assigning incremental numeric IDs I see 2 solutions for the copy problem:

1. Keep track of the last ID created within the project. This would mean updating the ID with every operation, including copying, deleting, moving, etc.
2. Check the ID with each operation, which results in n complexity.

[gedakc]

The duplicate ID problem arises if one copies and pastes a complete folder or scene because these new objects do not appear to be assigned a new ID. The problem does not arise if you copy only the written text from within a scene.

Personally I don't think that using UUID's alone will solve the problem because copying simple ID's or more complex UUID's and then pasting same should still result in duplicate values. Then again I've not dug deeply into the code so I could be wrong. This issue needs some love from someone with more knowledge of object oriented languages.

[siliconserf]

Yes, an object oriented approach would desirable. At least you'd get a constructor involved that would insure any new instance would have its own ID. I'd like to explore the subject. Is anyone using Eclipse to do development on this project?

On Sat, May 2, 2020 at 9:31 AM Curtis Gedak [email protected] wrote:

The duplicate ID problem arises if one copies and pastes a complete folder or scene because these new objects do not appear to be assigned a new ID. The problem does not arise if you copy only the text from within a scene.

Personally I don't think that using UUID's will solve the problem because copying simple ID's or more complex UUID's and then pasting same should still result in duplicate values. Then again I've not dug deeply into the code so I could be wrong. This issue needs some love from someone with more knowledge of object oriented languages.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/olivierkes/manuskript/issues/324#issuecomment-622979717, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVW6FV2XR3KOUCAG2GKUWDRPRDELANCNFSM4EUPK3XQ .

-- Ceterum censeo, delenda est Trumpo

[gedakc]

Mostly I use GNU emacs for development, but sometimes I use eclipse. For Manuskript I've found that the PyDev eclipse plugin works well for my purposes. Note that I had issues using the *ubuntu packages for eclipse so chose instead to install directly from the Eclipse web site packages.

[siliconserf]

I have Eclipse and the PyDev Eclipse plugin set up. How about a project file? It would save a lot of time.

On Sun, May 3, 2020 at 9:21 AM Curtis Gedak [email protected] wrote:

Mostly I use GNU emacs for development, but sometimes I use eclipse. For Manuskript I've found that the PyDev https://marketplace.eclipse.org/content/pydev-python-ide-eclipse eclipse plugin works well for my purposes.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/olivierkes/manuskript/issues/324#issuecomment-623134809, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVW6FSVVIIB6FU67DXOIFTRPWKX5ANCNFSM4EUPK3XQ .

-- Ceterum censeo, delenda est Trumpo

[gedakc]

I suspect you are looking for more functionality than I have configured in eclipse. The steps I used to add the manuskript project to the eclipse workspace assume that you clone the manuskript repository into a directory to use as the eclipse workspace (mine is ~/workspace/).

1. In Eclipse, select File -> New -> PyDev Project.
2. In the dialog for the Project name enter the exact same name for the manuskript git repo that you have cloned into the workspace. For example: manuskript
3. Click Finish button

This creates the following two files in the manuskript directory:

.project

<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
        <name>manuskript</name>
        <comment></comment>
        <projects>
        </projects>
        <buildSpec>
                <buildCommand>
                        <name>org.python.pydev.PyDevBuilder</name>
                        <arguments>
                        </arguments>
                </buildCommand>
        </buildSpec>
        <natures>
                <nature>org.python.pydev.pythonNature</nature>
        </natures>
</projectDescription>

.pydevproject

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?eclipse-pydev version="1.0"?><pydev_project>
    <pydev_pathproperty name="org.python.pydev.PROJECT_SOURCE_PATH">
        <path>/${PROJECT_DIR_NAME}</path>
    </pydev_pathproperty>
    <pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python interpreter</pydev_property>
    <pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>
</pydev_project>

[siliconserf]

Having an interesting time with this. Building manuskript gave an error reminding me to install pyQt5. Did that and now I get an error in version_0.py:

Line 13:from lxml import etree as ET

In the debugger: ModuleNotFoundError: No module named 'lxml'

I know that lxml showed up in a diagnostic over the file problem in the issue that kick off this adventure (thanks by the way for your indulging me with help on this.)

File "lxml.etree.pyx", line 2366, in lxml.etree._Attrib.getitem (src\lxml\lxml.etree.c:62258) KeyError: 'ID'

Since no file in the project answers to lxml I looked around and found lxml.py in a python directory. Is the IDE trying to tell me I have a configuration error?

On Sun, May 3, 2020 at 1:08 PM Curtis Gedak [email protected] wrote:

I suspect you are looking for more functionality than I have configured in eclipse. The steps I used to add the manuskript project to the eclipse workspace assume that you clone the manuskript repository into a directory to use as the eclipse workspace (mine is ~/workspace/).

1. In Eclipse, select File -> New -> PyDev Project.
2. In the dialog for the Project name enter the exact same name for the manuskript git repo that you have cloned into the workspace. For example: manuskript
3. Click Finish button

This creates the following two files in the manuskript directory:

.project

manuskript org.python.pydev.PyDevBuilder org.python.pydev.pythonNature

.pydevproject

<pydev_pathproperty name="org.python.pydev.PROJECT_SOURCE_PATH">
    <path>/${PROJECT_DIR_NAME}</path>
</pydev_pathproperty>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python interpreter</pydev_property>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>

</pydev_project>

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/olivierkes/manuskript/issues/324#issuecomment-623173090, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVW6FWXUQAVGDIOP7HB7ADRPXFMJANCNFSM4EUPK3XQ .

-- Ceterum censeo, delenda est Trumpo

[siliconserf]

Got it!!!! Found the problem was a missing install of lxml. Just ran manuskript without trouble. Thanks for your assistance and encouragement. Now I can experiment with the code and get a better understanding of the program (though if truth be told I really wish this were in C++ where I'm more comfortable.)

Regards,

Charlie H.

On Sun, May 3, 2020 at 9:17 PM Charlie H. [email protected] wrote:

Having an interesting time with this. Building manuskript gave an error reminding me to install pyQt5. Did that and now I get an error in version_0.py:

Line 13:from lxml import etree as ET

In the debugger: ModuleNotFoundError: No module named 'lxml'

I know that lxml showed up in a diagnostic over the file problem in the issue that kick off this adventure (thanks by the way for your indulging me with help on this.)

File "lxml.etree.pyx", line 2366, in lxml.etree._Attrib.getitem (src\lxml\lxml.etree.c:62258) KeyError: 'ID'

Since no file in the project answers to lxml I looked around and found lxml.py in a python directory. Is the IDE trying to tell me I have a configuration error?

On Sun, May 3, 2020 at 1:08 PM Curtis Gedak [email protected] wrote:

I suspect you are looking for more functionality than I have configured in eclipse. The steps I used to add the manuskript project to the eclipse workspace assume that you clone the manuskript repository into a directory to use as the eclipse workspace (mine is ~/workspace/).

1. In Eclipse, select File -> New -> PyDev Project.
2. In the dialog for the Project name enter the exact same name for the manuskript git repo that you have cloned into the workspace. For example: manuskript
3. Click Finish button

This creates the following two files in the manuskript directory:

.project

manuskript org.python.pydev.PyDevBuilder org.python.pydev.pythonNature

.pydevproject

<pydev_pathproperty name="org.python.pydev.PROJECT_SOURCE_PATH">
    <path>/${PROJECT_DIR_NAME}</path>
</pydev_pathproperty>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python interpreter</pydev_property>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>

</pydev_project>

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/olivierkes/manuskript/issues/324#issuecomment-623173090, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVW6FWXUQAVGDIOP7HB7ADRPXFMJANCNFSM4EUPK3XQ .

-- Ceterum censeo, delenda est Trumpo

-- Ceterum censeo, delenda est Trumpo

[siliconserf]

Having more fun digging into the code than I've had with software in some time. I tested my first impressions of the code by copying a manuskript project into the Sample directory. Lo and behold, the program listed "The-Book-of-Acts" as well as my dummy project and opened that project when clicked. Not a huge discovery, but it is nice to confirm some impressions as to what some of the code is doing, as well as knowing how to set up "template" projects. Thanks again for helping me get to this point.

Regards,

Charlie H.

On Tue, May 5, 2020 at 12:03 AM Charlie H. [email protected] wrote:

Got it!!!! Found the problem was a missing install of lxml. Just ran manuskript without trouble. Thanks for your assistance and encouragement. Now I can experiment with the code and get a better understanding of the program (though if truth be told I really wish this were in C++ where I'm more comfortable.)

Regards,

Charlie H.

On Sun, May 3, 2020 at 9:17 PM Charlie H. [email protected] wrote:

Having an interesting time with this. Building manuskript gave an error reminding me to install pyQt5. Did that and now I get an error in version_0.py:

Line 13:from lxml import etree as ET

In the debugger: ModuleNotFoundError: No module named 'lxml'

I know that lxml showed up in a diagnostic over the file problem in the issue that kick off this adventure (thanks by the way for your indulging me with help on this.)

File "lxml.etree.pyx", line 2366, in lxml.etree._Attrib.getitem (src\lxml\lxml.etree.c:62258) KeyError: 'ID'

Since no file in the project answers to lxml I looked around and found lxml.py in a python directory. Is the IDE trying to tell me I have a configuration error?

On Sun, May 3, 2020 at 1:08 PM Curtis Gedak [email protected] wrote:

I suspect you are looking for more functionality than I have configured in eclipse. The steps I used to add the manuskript project to the eclipse workspace assume that you clone the manuskript repository into a directory to use as the eclipse workspace (mine is ~/workspace/).

1. In Eclipse, select File -> New -> PyDev Project.
2. In the dialog for the Project name enter the exact same name for the manuskript git repo that you have cloned into the workspace. For example: manuskript
3. Click Finish button

This creates the following two files in the manuskript directory:

.project

manuskript org.python.pydev.PyDevBuilder org.python.pydev.pythonNature

.pydevproject

<pydev_pathproperty name="org.python.pydev.PROJECT_SOURCE_PATH">
    <path>/${PROJECT_DIR_NAME}</path>
</pydev_pathproperty>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python interpreter</pydev_property>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>

</pydev_project>

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/olivierkes/manuskript/issues/324#issuecomment-623173090, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVW6FWXUQAVGDIOP7HB7ADRPXFMJANCNFSM4EUPK3XQ .

-- Ceterum censeo, delenda est Trumpo

-- Ceterum censeo, delenda est Trumpo

-- Ceterum censeo, delenda est Trumpo

[siliconserf]

OK. One of the reasons I set up to play with the code was to track down the copy bug that revolved around failing to install a new ID number in the new copy. Only when I went to do so I found version 0.11.0 doesn't exhibit the problem. Change log entries don't seem to reflect this. Did the problem get fixed in PyQt?

Regards,

Charlie H.

On Tue, May 5, 2020 at 10:39 PM Charlie H. [email protected] wrote:

Having more fun digging into the code than I've had with software in some time. I tested my first impressions of the code by copying a manuskript project into the Sample directory. Lo and behold, the program listed "The-Book-of-Acts" as well as my dummy project and opened that project when clicked. Not a huge discovery, but it is nice to confirm some impressions as to what some of the code is doing, as well as knowing how to set up "template" projects. Thanks again for helping me get to this point.

Regards,

Charlie H.

On Tue, May 5, 2020 at 12:03 AM Charlie H. [email protected] wrote:

Got it!!!! Found the problem was a missing install of lxml. Just ran manuskript without trouble. Thanks for your assistance and encouragement. Now I can experiment with the code and get a better understanding of the program (though if truth be told I really wish this were in C++ where I'm more comfortable.)

Regards,

Charlie H.

On Sun, May 3, 2020 at 9:17 PM Charlie H. [email protected] wrote:

Having an interesting time with this. Building manuskript gave an error reminding me to install pyQt5. Did that and now I get an error in version_0.py:

Line 13:from lxml import etree as ET

In the debugger: ModuleNotFoundError: No module named 'lxml'

I know that lxml showed up in a diagnostic over the file problem in the issue that kick off this adventure (thanks by the way for your indulging me with help on this.)

File "lxml.etree.pyx", line 2366, in lxml.etree._Attrib.getitem (src\lxml\lxml.etree.c:62258) KeyError: 'ID'

Since no file in the project answers to lxml I looked around and found lxml.py in a python directory. Is the IDE trying to tell me I have a configuration error?

On Sun, May 3, 2020 at 1:08 PM Curtis Gedak [email protected] wrote:

I suspect you are looking for more functionality than I have configured in eclipse. The steps I used to add the manuskript project to the eclipse workspace assume that you clone the manuskript repository into a directory to use as the eclipse workspace (mine is ~/workspace/).

1. In Eclipse, select File -> New -> PyDev Project.
2. In the dialog for the Project name enter the exact same name for the manuskript git repo that you have cloned into the workspace. For example: manuskript
3. Click Finish button

This creates the following two files in the manuskript directory:

.project

manuskript org.python.pydev.PyDevBuilder org.python.pydev.pythonNature

.pydevproject

<pydev_pathproperty name="org.python.pydev.PROJECT_SOURCE_PATH">
    <path>/${PROJECT_DIR_NAME}</path>
</pydev_pathproperty>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python interpreter</pydev_property>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>

</pydev_project>

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/olivierkes/manuskript/issues/324#issuecomment-623173090, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVW6FWXUQAVGDIOP7HB7ADRPXFMJANCNFSM4EUPK3XQ .

-- Ceterum censeo, delenda est Trumpo

-- Ceterum censeo, delenda est Trumpo

-- Ceterum censeo, delenda est Trumpo

-- Ceterum censeo, delenda est Trumpo

[gedakc]

What OS and versions of Python and PyQt did you test with?

I reran the test steps from post 2 on Kubuntu 20.04 and the copy chapter/scene problem still exists. Details follow:

OS & Versions

• Kubuntu 20.04 Focal Fossa LTS
• Python 3.8.2
• PyQt 5.14.1
• Qt 5.12.8
• Manuskript 0.11.0

Screenshot after Copy/Paste New Folder

Console Log After Restarting Manuskript and Loading Project

user@kubuntu2004:~$ manuskript 
/usr/share/manuskript/manuskript/main.py:104: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if platform.system() is not 'Windows':
Debug: Web rendering engine used: QWebView
Running manuskript version 0.11.0.
Preferred translation: ['en-CA', 'en'] (based on available ui languages)
No translation found or loaded. (manuskript_en_CA.qm)
No translation found or loaded. (manuskript_en.qm)
Using the builtin translation.
/usr/share/manuskript/manuskript/importer/opmlImporter.py:124: SyntaxWarning: "is" with a literal. Did you mean "=="?
  return len(s) is 0
/usr/share/manuskript/manuskript/exporter/pandoc/abstractPlainText.py:78: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if self.formats is "":
Icon theme "NumixMsk" not found.
Loading: /home/user/issue324-test.msk
Detected file format version: 1. Zip: True.
WARNING ! There are some items with same IDs: ['18', '19', '20', '18', '19', '20']
Project /home/user/issue324-test.msk loaded.
Project issue324-test.msk saved.
user@kubuntu2004:~$ 

Note that you can ignore the SyntaxWarning messages as these have been fixed in PR #762.

[obw]

I have programmed a little Proof-of-Concept in PHP, which cleans the dupe Id's.

Something like that, but in Python could be easily implemented and added too the Tool Menu or after closing the project, as long we could not solve this problem in the copy/paste code.

It was roughly two hour work for implementing this in PHP, but in Python I will need weeks:(

One thing I wanted too know is the speed for something like that:

This is only a prototype!
Make Backup before use!
(c) 2020 by Marcus Obwandner obwandner at googlemail.com
Licence: GNU General Public License https://www.gnu.org/licenses/gpl.html Version 3 or (at your option) any later version.
Found 3349 Files and 1358 files with a dupe ID


Do you have a Backup? Is the Projekt closed?
If for both yes! Press RETURN!

real    0m0,179s
user    0m0,059s
sys     0m0,082s

I have about 17MB text in this Project, outline, nothing else..

manuskript_id_clean_php.zip

No big magic done here, only one Parameter, path too the msk file.

First look if it's exists and proof if it's not a singlefile Manuskript-Project.

After that iterate over the Outline Dir where all the Texts exists and try to fetch all ID's out of them.

After that, when there are dupes wait for RETURN, else end.

Then iterate over the files with no or a dupe ID and give them a unique one. The first file with a ID which is used more then once is not changed.

It's a Prototype/Proof-of-Concept, so be aware that this thing can destroy your Project!

I have tested it with three different Projects and everything look perfect, but not in depth analyse is done!

[siliconserf]

I'm running Windows 10. Eclipse Java IDE 2 PyDev 7.5.0.202001101138

[image: image.png]

Python 3.8.2 (Opened News in the directory for Python and found the a text file that listed 3.8 as 3.8.2.

[image: image.png] I am certain that PyQt5 is on my computer, but having no luck finding it. Qt shows up in an MS cloud drive and others as Qt5. All of the development tools were installed two days ago. I don't think you can build and run Manuskript without those elements. If there is a way to interrogate Eclipse, or even my system for the information I'm game, even if we need to do a phone call and give you remote computer access.

From past experience I'm very familiar with the errors that get thrown up for the copy issue. If the source code hasn't been changed to address the copy issue, then something in Qt or Py has changed. I'm able to copy, save, exit, and restart Manuskript and re-load the test file without any problems.

One small detail: when I do a copy of a document, both the new and old md document files get their ID numbers embedded into the file names.

[image: image.png] Then

[image: image.png]

Copying two files, no matter to where gets the safe ID treatment. But copying a* folder* with documents still doesn't work properly. Files that get copied in the folders do not get new IDs and I get the error message. Interesting that just copying a file doesn't have a problem. Apparently the copy routine stops dealing with the ID issue when it encounters a folder.

Regards,

Charlie H. Everett, Wa., USA

On Fri, May 8, 2020 at 8:59 AM Curtis Gedak [email protected] wrote:

What OS and versions of Python and PyQt did you test with?

I reran the test steps from post 2 https://github.com/olivierkes/manuskript/issues/324#issuecomment-372386846 on Kubuntu 20.04 and the copy chapter/scene problem still exists. Details follow: OS & Versions

• Kubuntu 20.04 Focal Fossa LTS
• Python 3.8.2
• PyQt 5.14.1
• Qt 5.12.8

Screenshot after Copy/Paste New Folder

[image: Manuskript-issue324-kubuntu2004-outline] https://user-images.githubusercontent.com/10405019/81423959-69483480-9112-11ea-9889-014ae9b4f474.png Console Log After Restarting Manuskript and Loading Project

user@kubuntu2004:~$ manuskript /usr/share/manuskript/manuskript/main.py:104: SyntaxWarning: "is not" with a literal. Did you mean "!="? if platform.system() is not 'Windows': Debug: Web rendering engine used: QWebView Running manuskript version 0.11.0. Preferred translation: ['en-CA', 'en'] (based on available ui languages) No translation found or loaded. (manuskript_en_CA.qm) No translation found or loaded. (manuskript_en.qm) Using the builtin translation. /usr/share/manuskript/manuskript/importer/opmlImporter.py:124: SyntaxWarning: "is" with a literal. Did you mean "=="? return len(s) is 0 /usr/share/manuskript/manuskript/exporter/pandoc/abstractPlainText.py:78: SyntaxWarning: "is" with a literal. Did you mean "=="? if self.formats is "": Icon theme "NumixMsk" not found. Loading: /home/user/issue324-test.msk Detected file format version: 1. Zip: True. WARNING ! There are some items with same IDs: ['18', '19', '20', '18', '19', '20'] Project /home/user/issue324-test.msk loaded. Project issue324-test.msk saved. user@kubuntu2004:~$

Note that you can ignore the SyntaxWarning messages as these have been fixed in PR #762 https://github.com/olivierkes/manuskript/pull/762.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/olivierkes/manuskript/issues/324#issuecomment-625885493, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVW6FR73ZFSFINT4P4KV7LRQQT7NANCNFSM4EUPK3XQ .

-- Ceterum censeo, delenda est Trumpo

[siliconserf]

I'm game to try, but I've never worked with PHP. How do I launch this?

On Fri, May 8, 2020 at 12:23 PM obw [email protected] wrote:

I have programmed a little Proof-of-Concept in PHP, which cleans the dupe Id's.

Something like that, but in Python could be easily implemented and added too the Tool Menu or after closing the project, as long we could not solve this problem in the copy/paste code.

It was roughly two hour work for implementing this in PHP, but in Python I will need weeks:(

One thing I wanted too know is the speed for something like that:

This is only a prototype! Make Backup before use! (c) 2020 by Marcus Obwandner obwandner at googlemail.com Licence: GNU General Public License https://www.gnu.org/licenses/gpl.html Version 3 or (at your option) any later version. Found 3349 Files and 1358 files with a dupe ID

Do you have a Backup? Is the Projekt closed? If for both yes! Press RETURN!

real 0m0,179s user 0m0,059s sys 0m0,082s

I have about 17MB text in this Project, outline, nothing else..

manuskript_id_clean_php.zip https://github.com/olivierkes/manuskript/files/4601099/manuskript_id_clean_php.zip

No big magic done here, only one Parameter, path too the msk file.

First look if it's exists and proof if it's not a singlefile Manuskript-Project.

After that iterate over the Outline Dir where all the Texts exists and try to fetch all ID's out of them.

After that, when there are dupes wait for RETURN, else end.

Then iterate over the files with no or a dupe ID and give them a unique one. The first file with a ID which is used more then once is not changed.

It's a Prototype/Proof-of-Concept, so be aware that this thing can destroy your Project!

I have tested it with three different Projects and everything look perfect, but not in depth analyse is done!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/olivierkes/manuskript/issues/324#issuecomment-625976931, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVW6FW3Q63DUDV7GDHQOXLRQRL2HANCNFSM4EUPK3XQ .

-- Ceterum censeo, delenda est Trumpo

[gedakc]

@siliconserf it appears that the images you intended to include in the above post(s) are missing. Perhaps you emailed a response to github as opposed to writing a response in a web browser with this issue loaded?

The version details can be displayed when Manuskript is running using the Help -> About menu option.

EDIT:

I'm able to copy, save, exit, and restart Manuskript and re-load the test file without any problems.

I can also load the project after performing the copy, but the duplicate ID warning is displayed in the console log (not in manuskript, but in the terminal window / command prompt from which manuskript is invoked).

[obw]

@siliconserf

I'm game to try, but I've never worked with PHP. How do I launch this?

Well you need PHP on your machine first. If you don't have it by default, do not make the effort, it's more for developer.

If you want see how it's works just open it in a Text editor.

Line 66 to 80 iterates over the dirs and finds the dupes....

the loop in 103 make the cleanup...

function findNewId <- finds a unused id

function changeIdInFile <- change the datafile, also found a bug in this function, in line 134 is a return missing!

It as I said a proof of concept, with the hope that someone who is better in python implement it direct in Manuskript....

[siliconserf]

Yep. Forgot about that behavior. It tells me I have Python 3.4.4, PtQt 5.5.1, Qt 5.5.1

On Fri, May 8, 2020 at 1:34 PM Curtis Gedak [email protected] wrote:

@siliconserf https://github.com/siliconserf it appears that the images you intended to include in the above post(s) are missing. Perhaps you emailed a response to github as opposed to writing a response in a web browser with this issue loaded?

The version details can be displayed when Manuskript is running using the Help -> About menu option.

[image: Manuskript-issue324-kubuntu2004-about] https://user-images.githubusercontent.com/10405019/81446905-e71f3680-9138-11ea-81ab-626cbd52e3b4.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/olivierkes/manuskript/issues/324#issuecomment-626008450, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVW6FXQZNSRHMMLTCAV6PDRQRUFLANCNFSM4EUPK3XQ .

-- Ceterum censeo, delenda est Trumpo

[siliconserf]

And I'm down-rev on two of three. I will fix that shortly.

CH

On Fri, May 8, 2020 at 1:42 PM Charlie H. [email protected] wrote:

Yep. Forgot about that behavior. It tells me I have Python 3.4.4, PtQt 5.5.1, Qt 5.5.1

On Fri, May 8, 2020 at 1:34 PM Curtis Gedak [email protected] wrote:

@siliconserf https://github.com/siliconserf it appears that the images you intended to include in the above post(s) are missing. Perhaps you emailed a response to github as opposed to writing a response in a web browser with this issue loaded?

The version details can be displayed when Manuskript is running using the Help -> About menu option.

[image: Manuskript-issue324-kubuntu2004-about] https://user-images.githubusercontent.com/10405019/81446905-e71f3680-9138-11ea-81ab-626cbd52e3b4.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/olivierkes/manuskript/issues/324#issuecomment-626008450, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVW6FXQZNSRHMMLTCAV6PDRQRUFLANCNFSM4EUPK3XQ .

-- Ceterum censeo, delenda est Trumpo

-- Ceterum censeo, delenda est Trumpo

[gedakc]

And I'm down-rev on two of three. I will fix that shortly.

@siliconserf no need to change versions as those versions are still used in Kubuntu 16.04 LTS which is still a supported operating system.

The reason I asked about the versions is because we were trying to determine if a newer PyQt fixed the issue. From my testing it does not fix the problem. My thoughts are the the duplicate ID problem is an issue with the Manuskript source code and occurs with the copy and paste operation. That is ideally where it should be fixed.

I recommend ensuring that you can reproduce the bug before trying to fix the code.

[siliconserf]

Now that I know a folder copy is needed I think I can track it down.

On Fri, May 8, 2020 at 2:33 PM Curtis Gedak [email protected] wrote:

And I'm down-rev on two of three. I will fix that shortly.

@siliconserf https://github.com/siliconserf no need to change versions as those versions are still used in Kubuntu 16.04 LTS which is still a supported operating system.t

The reason I asked about the versions is because we were trying to determine if a newer PyQt fixed the issue. From testing it does not fix the problem. My thoughts are the the duplicate ID problem is an issue with the Manuskript source code and occurs with the copy and paste operation. That is ideally where it should be fixed.

I recommend ensuring that you can reproduce the bug before trying to fix the code.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/olivierkes/manuskript/issues/324#issuecomment-626030628, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVW6FRI6PRGTYFKBIBVGHDRQR3CNANCNFSM4EUPK3XQ .

-- Ceterum censeo, delenda est Trumpo

[gedakc]

Excellent. Happy bug hunting!

[siliconserf]

Interesting results. I find that sometimes I can copy a folder with files without an ID duplication occurring. And there is definitely a lag between copying and when ID numbers are rationalized. The copied file folders and documents initially have no IDs, then magically the program catches up and no problem. This bug is a bit more complex than I had imagined. At least more subtle. This is outside the IDE. I'm wondering if the ID code is thread safe. Or any of the code for that matter.

[siliconserf]

Here is my "stalking horse" project copied out of the Edit function using the tool Snipy.

The section Scenes was copied and pasted into Notes. No duplicated IDs. No problem reloading after shutting down. Previously I'd copied Chapter Two and then edited the document and folder names. No problem their either.

[gedakc]

I'm wondering if the ID code is thread safe. Or any of the code for that matter.

Previously PR #706 fixed a bug with multi-threading so at least that part of the code was previously not thread safe. It is possible that a similar issue resides in the ID code.

Note that I re-opened this issue (I'm guessing it was accidentally closed).

[siliconserf]

Looks as though I need to study mutexes in Python code. And spend more time just browsing through the code to understand it's operational structure.

On Sun, May 10, 2020 at 8:46 AM Curtis Gedak [email protected] wrote:

I'm wondering if the ID code is thread safe. Or any of the code for that matter.

Previously issue #706 https://github.com/olivierkes/manuskript/pull/706 fixed a bug with multi-threading so at least that part of the code was previously not thread safe. It is possible that a similar issue resides in the ID code.

Note that I re-opened this issue (I'm guessing it was accidentally closed).

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/olivierkes/manuskript/issues/324#issuecomment-626347513, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVW6FSMI2ZF5QO3KXIA743RQ3D4VANCNFSM4EUPK3XQ .

-- Ceterum censeo, delenda est Trumpo

[gedakc]

Re-opening this issue as it appears to have been accidentally closed, and the problem with the creation of duplicate ID's when copying and pasting still exists.

[siliconserf]

I set up a small project to use in testing the copy bug. What I am seeing is when a folder with three text children is copied, the ID's get stripped when pasting (which is correct). I tracked the process to the point where the code decides whether to install a unique ID. But the test for no ID being present always fails! See abstractItem line 139:

    if not child.ID():
        child.getUniqueID()

I think the code sees that the property is defined instead of testing to see if an ID has been assigned. To verify this I tried to change the test to one that explicitly checks the value of the ID, but my command of Python code is woefully short. Can you help with the correct code to test the ID? Should there be a two stage test, first for the property existing, and then for the value?

[gedakc]

Good work @siliconserf. From a quick look it appears that the copy only strips the folder ID, but the three child text ID's remain set. Hence when these are pasted they are not given a new ID.

EDIT: The problem might occur earlier with the copy operation not clearing all the children (and lower) ID's.

[siliconserf]

I'm not seeing quite that as I debug. When the children of the folder I copy are added to the database no ID is set. The string is empty. Doing a save immediately after the copy gives me md text files with no ID field.