clever-tools
clever-tools copied to clipboard
CleverCloud
Refactor `clever logs`
- Seems like we have a race conditions between old logs and live logs => investigate
- Now would be good time to remove bacon from this part and maybe move more logic to the clever client.
One issue I have with clever logs is that every log line starts with a timestamp.
While it's handy for unstructured logs, many logging libraries include a timestamp of their own. Having an option to remove the timestamp prefix would be good, even though for now this seems to do the trick:
$ clever logs | cut -c 27-
My logs are NDJSON, hence the need to drop non-JSON data before processing them.
@franky47 I have a similar issue with a personal projet with HapiJS. I'll remember this when we work on this issue ;-)
Another remark with logs: they often are displayed out of order.
Example output received when building a Node.js app with TypeScript in CC_POST_BUILD_HOOK:
2020-05-10T12:48:25.418Z: $ rm -rf ./dist ./coverage
2020-05-10T12:48:25.766Z: $ tsc
2020-05-10T12:48:25.167Z: yarn run v1.21.1
2020-05-10T12:48:25.167Z: $ run-s clean build:ts
2020-05-10T12:48:24.267Z: Running CC_POST_BUILD_HOOK: yarn build
2020-05-10T12:48:34.605Z: Done in 9.38s.
Expected output (ordered by increasing timestamp):
2020-05-10T12:48:24.267Z: Running CC_POST_BUILD_HOOK: yarn build
2020-05-10T12:48:25.167Z: $ run-s clean build:ts
2020-05-10T12:48:25.167Z: yarn run v1.21.1
2020-05-10T12:48:25.418Z: $ rm -rf ./dist ./coverage
2020-05-10T12:48:25.766Z: $ tsc
2020-05-10T12:48:34.605Z: Done in 9.38s.
@franky47 thanks for the report. We are already working on it, we will soon release a fix for that.
Do you have an idea on when it will be released ?
I find it very hard to use clever logs since it only fetch 1k lines of logs (and it's not in the documentation :roll_eyes: ) and since it hangs at the end (#328 ) it's not easy to script.
I feel that it's difficult to use them for my simple workflow that is to grep for a pattern and check logs around this time.
@antoine-de I cannot comment on an ETA, maybe @KannarFr knows more about this?
We are currently hard working on better logs retriever solutions, but there is currently no ETA.
The pain was strong enough that we implemented an Elixir task to fetch the logs.
It is inefficient (we didn't parallelise the queries on purpose at this point), requires de-duping the lines & double-checking that we didn't miss anything due to pagination.
While I understand that we could setup a drain etc, I find it very surprising that this feature is not prioritised, because I believe it is quite essential for a lot of uses...
I believe this alone is a compelling reason not to use CleverCloud, I really hope you'll manage to implement something later so we can ditch that code!
Good luck.
We are working on it yes!
Le jeu. 8 juil. 2021 à 16:40, Thibaut Barrère @.***> a écrit :
Hi @KannarFr https://github.com/KannarFr, is this feature still planned? Thanks!
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CleverCloud/clever-tools/issues/403#issuecomment-876497183, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZLQJMM2V366ORTYWNI4I3TWW2GPANCNFSM4LZETYAQ .
I'm closing this issue since we made good progress:
- performance was enhanced
- there is no more limit to 1000 lines of logs
- added support for
--sinceand--untiltime range bounds - added support to JSON output
- logs order is now respected
