timemanager
timemanager copied to clipboard
te-online
Add importer
It would be good for everybody switching to this to be able to import data from another app. Preferable from a csv or other table structure.
Hey there đ
Nice feature idea. There's a basic importer in the "Tools" section of the app. But it's lacking import of actual time entries, you can only import the structure around them so far...
If you'd like to enhance it, you're very welcome to do so. Just make sure to open a PR (even if you haven't changed anything yet! đ) as soon as you begin, so we can continue the discussion and I can help you along the way đ¤
Hey there
I wanted to develop a small python script for check all calendar of my Nextcloud users and create a CSV that I can import in TimeManager for create a link between calendar <> TimeManager... I'm pretty disappointed because I didn't understand at my first use of TimeManager that the import function is only usable for the "Table structure" ... and not for data with timestamp.
Right now, there is no alternative to "import" data with Timestamp ? :((( .. Maybe I will need to import this direct in the BDD but it seems to me that this is very very ugly. I'm not very confortable with development of Nextcloud plugin so I don't think I can help in the development of this feature.. But that make me sad :(
Is it so hard to add this feature @te-online ?
Hey đ
I understand your frustration with the basic feature of importing time entries missing.
It's not exactly about how hard it is to add something, but usually about how useful it would be to me personally. I've always made it very clear that I have very limited time for this project and no interest in adding something that I don't use myself. Of course contributions can contain features that I don't use personally. I'm sorry if this is disappointing, but I've decided to not dedicate more time if it's not absolutely required or I otherwise think it would be beneficial.
That being said, you may be able to import data using a python script, like you plan, and copying/adapting some of the JS the importer uses.
Have a look at this line (and the whole file in general): https://github.com/te-online/timemanager/blob/main/js/views/Import.svelte#L181
I'm posting the data read from the CSV file to TimeManager's REST API. The array for times is empty â this is where the time entries go.
You'll have to make sure to generate unique uuid v4s for each time entry and correctly link them to a task that either exists in the database or is part of your import.
If you're in doubt about the model, have a look at the entity here https://github.com/te-online/timemanager/blob/main/db/time.php.
Hey,
Thank you for your answer ! I totally understand that you don't want to spend more time on the plugin so, no problem for this.
I'm still not sure to understand everything but, if I understand well, there is a REST API usable which can allow me to add times entry ? Can you help me to find the URL of this API ? Is there any documentation of this API somewhere ?
Tell me if I'm wrong but for my purpose, if i want to add times entry, I'll need to send this kind of JSON data
{
lastCommit: "",
data: {
clients: {created: [],updated: [],deleted: []},
projects: {created: [],updated: [],deleted: []},
tasks: {created: [],updated: [],deleted: []},
times: { created: [
[
"changed" : "",
"commit" : "",
"created" : "",
"end" : "",
"note" : "",
"paymentStatus" : "",
"start" : "",
"task_uuid" : ""
"uuid" : "",
];
], updated: [], deleted: [] }
}
};
I'll study this later... But thanks for you answer again.
Your data structure looks good đ
Unfortunately, there's no documentation yet âšī¸
This should be the API endpoint:
https://example.org/apps/timemanager/api/sync-web
And you will have to be logged in with your user and send a Nextcloud request token along. Try something like this in your browser console :-)
fetch('https://example.org/apps/timemanager/api/sync-web', {
method: "POST",
headers: {
requesttoken: window.OC.requestToken,
"content-type": "application/json",
},
body: JSON.stringify(convertedImportData),
});
Oh thanks again !
I'm trying to create a request that work but I always got this error
"message": "CSRF check failed"
For this command :
curl -X POST -k -H 'requesttoken: -----------------------------' -H 'Content-Type: application/json' -H 'OCS-APIRequest: true' -i 'https://-------------------------------/index.php/apps/timemanager/api/sync-web'
Don't know what I'm doing wrong :( .. Perhaps you will have an intuition?
Yes! I gave you the URL for use in the browser. To protect against CSRF, Nextcloud uses a CSRF token in the header for requests from the browser. If you're using CURL, try this URL instead, it should work without CSRF check:
'https://example.org/apps/timemanager/api/updateObjects'
However, it will not work with the OC.requestToken, you need to use your user credentials in a Authorization: Basic ${base64(username:password/token)} (this is pseudo code) header instead.
You basically have two options:
- Execute the request in the browser with
requesttoken: window.OC:requestTokenusingfetch, with URLapi/sync-web - Execute the request outside the browser with
Authorization: Basic ...header using your credentials, with URLapi/updateObjects
Both API endpoints call the same code internally and accept the same shape of data.
Hope that helps.
Hii !
I still try to add Times via curl request. I'm using this command with this data but I got a 404 error, I don't know why. Same question that my last message, do you have a quick intuition ?
{
lastCommit: "",
data: {
clients: {created: [],updated: [],deleted: []},
projects: {created: [],updated: [],deleted: []},
tasks: {created: [],updated: [],deleted: []},
times: { created: [
[
"changed" : "2023-01-26 10:00:00",
"commit" : "8d955d75-4f75-4c5b-96ce-518ae140d9e5",
"created" : "2023-01-26 10:00:00",
"end" : "2023-01-26 13:50:00",
"note" : "",
"paymentStatus" : "NULL",
"start" : "2023-01-26 10:50:00",
"task_uuid" : "6c11269a-ed10-4430-96da-002317cb405f",
"uuid" : "729fb8d3-949d-44d7-8be3-fbc29fd4b528",
]], updated: [], deleted: [] }
}
}
all the uuid was generated manually by https://www.uuidgenerator.net/version4
The command is...
curl -X POST -k -H 'Content-Type: application/json' -H 'Authorization: Basic xxxxxxxxxxxxxxx' -i 'https://xxxxxxxxxxxxx/index.php/apps/timemanager/api/updateObjects' --data 'DATA BELOW'
And the server return me nothing, just a nice 404 :D
Thanks again for your help.
Hi @Kayoku â I think the reason it's not working is that your JSON is malformed. It's strange Nextcloud returns a 404 in that case, maybe something I could fix đ
I tried this request and it worked:
curl --silent --location --request POST 'http://localhost:8000/index.php/apps/timemanager/api/updateObjects' \
--header 'Authorization: Basic XXX' \
--header 'Content-Type: application/json' \
--data-raw '{
"data": {
"clients": {
"created": [],
"updated": [],
"deleted": []
},
"projects": {
"created": [],
"updated": [],
"deleted": []
},
"tasks": {
"created": [],
"updated": [],
"deleted": []
},
"times": {
"created": [
{
"changed": "2023-01-26 10:00:00",
"commit": "8d955d75-4f75-4c5b-96ce-518ae140d9e5",
"created": "2023-01-26 10:00:00",
"end": "2023-01-26 13:50:00",
"note": "",
"paymentStatus": "NULL",
"start": "2023-01-26 10:50:00",
"task_uuid": "6c11269a-ed10-4430-96da-002317cb405f",
"uuid": "729fb8d3-949d-44d7-8be3-fbc29fd4b528"
}
],
"updated": [],
"deleted": []
}
},
"lastCommit": ""
}'
An exporter accompanying the importer would also be very helpful to me. I'm migrating to a different Nextcloud installation and don't see any way to transfer the data from timemanager over there.
The importer is meant to import the basics from another program or a manual spreadsheet. No time entries, yet, though.
A separate backup & restore feature would be really nice, but isn't on the roadmap right now đĸ
Do you have access to the databases of your old and new instances? Then you can copy over the tables and you should remain all information.
Otherwise your only option is to use the REST API (as mentioned in some of the comments above).
Hey !
I'm coming back :D Just want to know if there is a way to know if an object already exist with the field "name" Now, I can create Client/Projects/Task but I need to know if they already exists with the API, and I don't know how to do that. I didn't find any "getObjectByName" or similar stuff in the API, but maybe I'm wrong.
Can you help for this ?
Hey there â I don't think there's a way. You can download the entire dataset, by "pretending" to synchronize for the first time (I believe this is done by sending an empty string as commit). Then you can filter it client-side.
Generally, the datastructure doesn't require the name field to be unique. Each object has a unique uid v4. That's what helps you distinguish objects from each other, not their name.
Hope it makes sense đ
Mmh ok, I got many question now...
- The request return me all the "deleted" object, how can I delete them forever ? (without going to the database with a MySQL request)
- If I got 100 Clients, 1000 Projects and 10000 times, every data will be return after each POST request I do ? It seems a bit overextend.
- Also, even when I make a request without commit / uuid, to the "updateObject" URL, it create the object on the database :(
Well, this is really a challenge to try to automated a process for import data when the API of the plugin doesn't really respect CRUD but OK, I like challenge :-)
You are 100% right, the API is not designed in typical CRUD fashion. It was designed for synchronization with mobile apps or 3rd party systems.
- You send your data with no
commit? -> the API sends you everything - You send your data mentioning the last synchronized
commit? -> the API sends you the changes (including deletions) from that point on
A commit is just a marker in the dataset to determine what changes have been made after a certain point in time, without relying on clocks to be synchronized between server and client.
It's only natural that this API is not very useful for importing data. I did use it for the basic importer, but that was not ideal.
To your questions:
The request return me all the "deleted" object, how can I delete them forever ? (without going to the database with a MySQL request)
Unfortunately, there is currently no way to do that. Mobile sync had highest priority in the early version of this app (and it may become a focus again), that's why I made the tradeoff to do soft-deletion. Otherwise you would never know what is safe to delete on the mobile client. There's an issue #66 here where I documented the problem and I'm going to address this flaw eventually.
If I got 100 Clients, 1000 Projects and 10000 times, every data will be return after each POST request I do ? It seems a bit overextend.
Provide the commit the API gave you for the last commit. If you do that, it will only give you the changes that have been made after this marker had been created.
Also, even when I make a request without commit / uuid, to the "updateObject" URL, it create the object on the database :(
Yes, objects are only identified by their uuid. If you don't provide it, they are treated as "new". This is by design and unlikely to change.
Can you explain again what your use-case is? At some point I'd like to get started with a proper import/export feature and I'd like to make sure to cover this case, if possible đ
Thanks for you fast answer.
Would it be more logical to have two different request :
- HTTP POST for add/update data, returning just success or error
- HTTP GET for synchronizing data (get specific data or all data), with the commit UUID in the parameter of the URL ?
Probably that I don't understand all the problems raised by mobile synchronization.
About my use case :
I got some CSV file exported from each Calendar in a Nextcloud instance, with this information (I made you a small example, but there is ~~30 CSV files, with 100 lines by file) :
| Title | Start date | Start time | End date | End time | Categories | Location | Description |
|---|---|---|---|---|---|---|---|
| Meeting XX | 2023-05-15 | 10:00:00 | 2023-05-15 | 12:00:00 | Important Meeting | Office | Structure |
| Coffee | 2023-05-16 | 08:00:00 | 2023-05-15 | 08:30:00 | Coffee | ||
| Urgent report | 2023-05-17 | 14:00:00 | 2023-05-17 | 17:30:00 | Urgent | Office | BestClient |
this information determines the links between project / client / tasks...
-> Description column from CSV = Client in TimeManager -> Categories column from CSV = Project in TimeManager -> Title column from CSV = Tasks in TimeManager
The main issue is that I need to check each time I send new data, if Client/Project/ already exists, because I don't want to recreate it.
It would be possible to check directly in the DB, or to store all the UUID of all the client/project I need but, this means complicating the script with a kind of local database, not my favorite way.
Hope it's clear ^^
I understand your frustration with the API that is available.
However, what you say is true: If you want to make this more efficient, you will have to sync with a local database. Then you can first sync your calendar to the database and then send all new or updated items to the TimeManager API.
The other option is to fetch everything every single time you synchronize. Then you can check all you want in memory and send the composed set of changes as "created" or "updated" to the API. This would consume as much memory as you have stored, but would probably work most reliably.
If you have a calendar system to track time, I wonder why it would be necessary to use TimeManager at all?
To answer your question, our customer use the Nextcloud calendar, but it's not really pratical to watch all the stats about one project / one client. TimeManager was nice for this : fast data vizualisation, with different filter by project / client etc...
And it's a Nextcloud Plugin that allow us to have everything in Nextcloud, instead of use CSV exported from Nextcloud to local to process data.
Hey,
I'm still working on this. I'm using some MySQL request to check if client/project already exist, it's finally the simple way for me.
But I'm facing an issue, when I create a new Project/Client by HTTP Request, the "status" of the row is never set to "0" (even If I put it in the JSON send by HTTP). Do you know how the status is set ? I found nothing in the plugin code about it.
With this in my DB (client/project without status define), browsing Projects in Nextcloud interface freeze, and raise an error in the backend log, I don't really know what is wrong...
[Mon Sep 11 15:33:49.258616 2023] [proxy_fcgi:error] [pid 1983098:tid 139670042793728] [remote 178.20.64.30:37006] AH01071: Got error 'PHP message: {"reqId":"EvCToNcKVzzSxNvtRorb","level":1,"time":"2023-09-11T13:33:48+00:00","remoteAddr":"178.20.64.30","user":"xxxx","app":"richdocuments","method":"GET","url":"/index.php/apps/timemanager/projects","message":"Fetched capabilities endpoint from https://xxxxxxxxxxxx/hosting/capabilities in 0.061 seconds","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","version":"25.0.10.1","data":{"app":"richdocuments"}}PHP message:
{"reqId":"EvCToNcKVzzSxNvtRorb","level":3,"time":"2023-09-11T13:33:49+00:00","remoteAddr":"178.20.64.30","user":"xxxxx","app":"PHP","method":"GET","url":"/index.php/apps/timemanager/projects","message":"Undefined property: OCA\\\\TimeManager\\\\Db\\\\Project::$client at /var/virtual_www/crealead.com/nextcloud.crealead.com/htdocs/apps/timemanager/templates/projects.php#78","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","version":"25.0.10.1","exception":{"Exception":"Error","Message":"Undefined property: OCA\\\\TimeManager\\\\Db\\\\Project::$client at /xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/htdocs/apps/timemanager/templates/projects.php#78","Code":0,"Trace":[{"file":"/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/htdocs/apps/timemanager/templates/projects.php","line":78,"function":"onError","class":"OC\\\\Log\\\\ErrorHandler","type":"::"},{"file":"/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/htdocs/lib/private/Template/Base.php","line":180...PHP message:
{"reqId":"EvCToNcKVzzSxNvtRorb","level":3,"time":"2023-09-11T13:33:49+00:00","remoteAddr":"178.20.64.30","user":"xxxxx","app":"index","method":"GET","url":"/index.php/apps/timemanager/projects","message":"Call to a member function getName() on null","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","version":"25.0.10.1","exception":{"Exception":"Error","Message":"Call to a member function getName() on null","Code":0,"Trace":[{"file":"/xxxxxxxxxxxxxxxxxxxxxxxxxxxxx/htdocs/lib/private/Template/Base.php","line":180,"function":"include"},{"file":"/xxxxxxxxxxxxxxxxxxxxxxxxxxx/htdocs/lib/private/Template/Base.php","line":150,"function":"load","class":"OC\\\\Template\\\\Base","type":"->"},{"file":"/xxxxxxxxxxxxxxxxxxxxxxx/htdocs/lib/private/legacy/OC_Template.php","line":181,"function":"fetchPage","class":"OC\\\\Template\\\\Base","type":"->"},{"file":"/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx..PHP message:
{"reqId":"EvCToNcKVzzSxNvtRorb","level":3,"time":"2023-09-11T13:33:49+00:00","remoteAddr":"178.20.64.30","user":"xxxxx","app":"PHP","method":"GET","url":"/index.php/apps/timemanager/projects","message":"chmod(): Operation not permitted at /xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/htdocs/lib/private/Log/File.php#86","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","version":"25.0.10.1","data":{"app":"PHP"}}'
Maybe you'll have another intuition what I'm doing wrong ? Thanks :)
The first error looks to me like maybe you deleted a client that was the parent of a project that still exists? You need to cascade the delete to all child projects, tasks, times. Otherwise the frontend will break.
Generally, the app doesn't know yet how to delete records permanently. It uses soft-deletion instead (status=deleted).
I believe for available (aka not deleted) rows, the value for status should be null.
Ok, thanks for the first error.
When I'm trying to force JSON with status "Null", that doesn't change the result.
| id | user_id | uuid | changed | created | client_uuid | commit | color | name | note | billable | status |
| 15 | xxx | ca3b2293-c28f-4bba-8265-627180b801da | 2022-12-12 16:00:43 | 2022-12-12 16:00:43 | 17595d1e-2d08-44b4-a5bf-93d8d1917219 | 2cdbbe37-ed77-45e1-ab1d-a0af0577801f | NULL | Formation pro | NULL | 1 | 0 |
| 63 | xxx | 3500d2bf-6de7-4ef0-aaeb-1200db0cdf6a | 2023-09-12 09:33:23 | 2023-09-12 09:33:23 | 17595d1e-2d08-44b4-a5bf-93d8d1917219 | ec6fc149-83db-45d2-9e16-733c239c8fcc | NULL | Gestion cae | NULL | 1 |
As you can see, the first row have a "0" in status column, and the second (create by HTTP request from my script) have no value.
{
"lastCommit": "",
"data": {
"clients": {
"created": [],
"updated": [],
"deleted": []
},
"projects": {
"created": [
{
"changed": "",
"commit": "",
"created": "",
"name": "Gestion cae",
"client_uuid": "17595d1e-2d08-44b4-a5bf-93d8d1917219",
"note": null,
"status": null,
"uuid": ""
}
],
"updated": [],
"deleted": []
},
"tasks": {
"created": [],
"updated": [],
"deleted": []
},
"times": {
"created": [],
"updated": [],
"deleted": []
}
}
}
I try with null, 0 and "0" as value but it never works
Okay, I'm not sure I fully understand đ
- What are you trying to achieve by setting the
status? - Is the app breaking, because you're not setting the
statusto something specific?
Oh sorry.. Yes, I think the app is breaking because i'm not setting the status to 0 instead of nothing
Hm... the errors you sent, don't indicate that. As long as you don't set status='deleted', you should be able to set it to any value without breaking the app. I think that maybe something else is going on there. Do you get more errors than the ones you've already sent? đ
Hey, Thanks for you answer. I found the real issue... Finally, it wasn't related to the "status" column. It was because a client_uuid pointed to another user's because of one bad function on my script. Sorry for wasting your time :x
Your feature request is valid and I appreciate your input đ
However, in an effort to focus on maintaining existing features, I'm closing feature requests like this one for now.
For more context, please take a look at the "Update strategy" section of the Readme đ