Porting guide modernization

[abkro]

This PR contains the updated and extended UBports Porting Guide.

The guide has been reworked by Ari Börde Kröyer based on the current guide and the contents of the Halium Generic Build Tools scripts using AI assistance. Although the contents are by and large accurate, it is possible that some details need adjustment. The guide therefore needs to be examined by experienced developers/porters to confirm accuracy.

[doniks]

is this supposed to look like this?

[abkro]

No. That's what the mermaid module is for. It creates a flow chart.

[doniks]

ah. ugh. damn. probably the jenkins setup ... for some reason the jenkins builds always look a bit different :see_no_evil: we really need to fix this :unamused: sorry

[doniks]

So. I'm trying to read this, but my first impression to be honest is I get lost all the time. I never know where I end up after I click on a suggested step and end up going back to the start page and trying all over.

It seems to me the new organisation is making a lot of accomodations for an "expert" and tries to anticipate what that expert needs, while still writing all the content. And I have the feeling the result is so full of "hey maybe you want to go here or there"-pointers and "this is for experts"-disclaimers that it just becomes hard to read for ... I think for everyone. I'm wondering whether it's worth it to invest so much in this non-linear structure for some "expert" ... I'd say if they are experts then they will have no problem scanning and skipping to get to what they need. And if they are not able to scan and skip because they don't understand the content and the relevance ... well, then they are probably not actually experts, just impatient.

[doniks]

ok. I'm slowly getting somewhere. I start from fundamentals. Always ignore any "maybe you want to" parts. Read top to bottom. At the bottom click "next"

I really like the content! it gives much clearer overview of what's the purpose of which part. I love the summaries on the top! with the navigation I still feel a bit challenged

[abkro]

I could probably tone down the "expert/novice" pathways a bit without too much trouble, but I do think it wise to let the different sections start with the main steps in brief, whatever one wishes to call it. "Quick start" or "Steps in brief" or just "In brief" or any better suggestion you might have?

But maybe we can try it out this way first while completing what is still missing and then adapt it?

Still missing:

• The section on finalizing the port. I need either reliable sources for this, or a reliable source (person) to discuss it with.
• The section on transitioning a xenial port to focal. Ideally written as generically as possible (if possible), to allow for subsequent releases,
• And less crucial: Adding relevant links/sources to the additional resources section at the end.

[abkro]

Let me know if there's anything in particular that you're expecting from me. Otherwise, I will just try to address a couple of the points I mentioned above, when I get the chance. I will also check if there is anyone who can help me with the section on finalizing the port. But the things I mention here can be added in an iteration, so they're not really necessary for a first release, in my opinion.

The mermaid visuals need to build and display properly, though.

[doniks]

yeah, my feeling is the disclaimers and pointers could be thinned out a bit.

I agree to keep the short preambles on top of pages to give a summary. That is super helpful.

Wrt missing parts - I would say, ideally we're not removing anything that's already there and is not meant to go away. So, if finalizing is missing in the sense that the existing content hasn't been moved into the new structure, then I'd say that should be fixed before publishing a new version. Any other parts that we don't have today shouldn't block a publication. And in fact going in iterations is often easier than in a big bang.

Wrt mermaid, I'm leaning towards kicking that can down the road. I can see them when I build locally, and I expect that they would build correctly once they are merged. I forgot the details, but I think the CI takes some parts from main branch, or from some vanilla uncustomized setup or something. So, I think that is the reason why the CI build of the PR doesn't look right, but I expect this would not affect the actual build of the site once merged. Now, of course it would be great if we could improve our CI build, it actually has a couple of rendering problems (links, etc), but again, that's a separate problem and shouldn't block a merge of this porting modernization.

What I think does need to happen is a full review. There should be at least four eyes that agree with the new structure and content. While I'm happy to review it in detail, it will take me some time to finish it, since it is quite big. If someone else wants to step in and do a review I'd certainly welcome it!

Maybe just one more thing that I noticed: In the current version the navigation looks like this:

ie a section "Porting" with a couple of chapters

The PR looks like this:

ie a section "Porting" with a single chapter called "Ubuntu Touch Porting Guide". The "Ubuntu Touch" seems redundant, but moreover everything is collapsed into a single chapter. I like it better if the hierarchy is that one level flatter and we could pull the content chapters up by one level. But I can't really suggest a useful set of top level chapters because I haven't managed to understand and digest the newly proposed structure

[abkro]

I'm busy on Saturday, but I'll try to follow this up on Sunday. By the way, I agree fully with the need for more eyes on the content mainly, but also the structure. However, my posts in the porting group, and the mention in the last Q&A have not triggered any interest.

I wonder, is it because I am the one doing this, and people have seen the many dumb questions I have asked in this group, so they assume the result is just garbage? Or maybe it's down to a general lack of knowledge about AI, so they think this is pure fiction? Or is it simply that they're too busy with other stuff? My point is that if you know how to enlist someone with the necessary knowledge to go through this, please help to make it happen!

In the meantime, I will be working on addressing the points you mentioned.

[doniks]

Ah. No. Please don't take it personal! Anyway there is no such thing as a dumb question. From where I sit its just that there is not much contributions to the porting docs, or even the docs overall. And yeah it's probably a mix of lack of knowledge or lack of courage or lack of time. To me you're just one of the few unsung heroes keeping the porting docs up to date. Much appreciated! I wish I could recruit more contributors or reviewers, but it is what it is. Which probably also means that you are stuck with me as a reviewer:) I sense you would wish for someone with more porting know how than me, but well that's also just how it is. And of course it's all volunteer work and people should scratch their itches and otherwise everybody has a life outside so noone to blame, we just make do with what we have 💪

[doniks]

Ah one other small feedback: I think you can remove the "this is work in progress" disclaimer now. That's understood

[abkro]

I have now addressed all points you mentioned above. As a result a couple (but not all) of the mermaid visuals are now gone. Don't worry that I'm taking things personally! I'm just seeking to understand and find a way to get help checking the re-written guide. And I greatly appreciate your feedback! The end result will turn out to be a better product thanks to it!

I am sure many improvements are still possible, but I am also sincerely convinced that the guide, when the current PR gets published, now vastly surpasses what it previously had to offer.

I think that maybe actually publishing this would trigger the feedback that is needed.

[peat-psuwit]

@abkro pinged me on Telegram to figure out the graph issue. TL;DR, it's only a problem when viewing directly from Jenkins artifacts.

See, Jenkins has a so-called Content Security Policy which, amongst other things, prevents scripts not hosted on its server to load. And it just to happens that the way mermaid module works is by fetching a JS script from a CDN and then render the graph locally. You can try downloading the whole build archive (go to the artifact page and click "Download All") and then open index.html locally. When merged and deployed on docs.ubports.com, it should work.

However @abkro it would be nice if we can avoid a potential privacy concern and make it use a copy of Mermaid script locally hosted. I see something in the module's config section; it would be nice if you can figure it out.

And finally, next time, consider pinging me by typing my handle (like this: @peat-psuwit). This will let GitHub send an E-mail to me (easier to manage personally) and also it keeps all relevant contexts within this PR itself.

[peat-psuwit]

Or, actually, a better thing to do is to pre-generate the graph itself instead of generating it by the client. Although from the doc I think it will complicate the CI setup a bit.

[abkro]

@peat-psuwit Thanks for the pointers! About using the handle: I first clicked on the link next to your name on this github page (the one designed for requesting a review). I assumed it would result in you receiving an email from Github. Afterwards, I contacted you to point out the problem with the visual.

I have now added a commit to host the javascript dependencies of the mermaid extension locally. If I understand correctly, this should fix the security concern.

[abkro]

I see it still does not render in the CI-build artifacts. I didn't modify the requirements.txt file. Maybe I need to add the version of the mermaid module there as well? Or maybe it will not render here in any case, but will work as expected once merged into the the build?

I will await further instructions before doing anything else.

[peat-psuwit]

Hmm Ok I misread Jenkins' CSP policy [1]; apparently they block all Javascript from running at all (which makes sense, actually; the build artifact is essentially user-controlled). So unless you pre-render the diagram, it will never show up when you preview the doc on CI. However when merged it will work properly. The requirements.txt plays no role in this issue.

While we're still at it, could you please also serve D3 Javascript file locally as well?

[1]:

content-security-policy: sandbox allow-same-origin; default-src 'none'; img-src 'self'; style-src 'self';

[abkro]

While we're still at it, could you please also serve D3 Javascript file locally as well?

[1]:

content-security-policy: sandbox allow-same-origin; default-src 'none'; img-src 'self'; style-src 'self';

Did this now.

I also completed and added a first version of the bibliography section.

[abkro]

I had some time on my hands, so I improved and extended several of the configuration sections.

[abkro]

@peat-psuwit General info about my limited coding background/experience and how I have worked.

Myself:

• I hold degrees in marine biology and educational management
• I have used Linux for 20+ years. Needed to constantly recompile the kernel during the early years in order to enable my specific hardware.
• I have followed basic programming courses long ago, and learned some C and C++ on my own, but don't have extensive knowledge.

How I reworked the guide:

• Used Claude.ai (by Anthropic)
• Defined a project in Claude.ai. Permits the user to upload a knowledge base for the AI.
• Uploaded the complete original porting chapter as well as the complete set of Halium Generic Build Tools scripts used for the standalone kernel build.
• Supplied other information relating to what Ubuntu Touch is in order for the AI to refer to information in it's own knowledge base.
• The AI's knowledge cut-off date is April 2024. It does not have direct access to internet, but has an enormous knowledge base from this date.
• Claude.ai is from my own and colleagues' experience way more potent than other AIs we have tried when it comes to coding.

There is no guarantee that what it produces will be correct. Specially not for something like Ubuntu Touch, where an untold amount of what needs to be understood may not be accessible online. However, what is well documented online will generally come out accurately.

The handiwork with AIs, i.e. what makes the difference between a good and a poor result, is largely a question of how you prompt it. You can make it double check and evaluate it's own produced content against specified criteria.

I have done some of this, but probably not enough, seeing as I am working in my spare time, and I generally believe that the lack of sound online documentation specifically targeting Halium and Ubuntu Touch is the main source of errors in the text I have produced.

Conclusion:

• I hope that understanding my background and method will be of use to you when evaluating the contents and that you can help me make this worthy of publishing.
• In my porting efforts, I have frequently needed significantly more assistance and wished for a better understanding than the current guide has been able to offer.
• My goal is to produce something closer to what I have been needing and thereby give the porting effort a serious boost.

[abkro]

I am not sure how to deal with your mermaid - related comment. It is unclear to me whether to leave things as they are, or seek a different solution altogether for displaying this visual? The important thing is for the end users to see it, but without building this locally, of course.

Would removing mermaid-layout-elk.esm.min.mjs solve the issue?

[peat-psuwit]

See, in this situation the ideal solution is to generate the graph image ~locally~ at build time using mermaid_output_format config, which is the only way that will work with on-CI preview. But that means fiddling with Jenkinsfile to make sure mermaid-cli is installed, preferably without disturbing the rest of the system.

So if that's too hard to you. The next option is to revert to just let the generated HTML pull the JS files from CDN, which is the situation before I made the first comment. On-CI preview will continue to be broken, but it'll work if you pull the preview from CI and preview locally, and also when it's merged. (This is because apparently support for *_use_local is broken in sphinxcontrib-mermaid, and so probably requires some ugly hack for it to work.)

[abkro]

See, in this situation the ideal solution is to generate the graph image ~locally~ at build time using mermaid_output_format config, which is the only way that will work with on-CI preview. But that means fiddling with Jenkinsfile to make sure mermaid-cli is installed, preferably without disturbing the rest of the system.

Did it this way and confirmed locally that this works. I included the Chrome dependencies for good measure, since they turned out to be necessary when I built locally. I placed everything in an adjacent Jenkinsfile.setup-mermaid which I called from the main Jenkinsfile. I hope this looks satisfactory now, @peat-psuwit ?

[abkro]

Well, I can see that this did not work, but this time I don't see any error message detailing the problem, so I'll try to add some debug messages and see what happens.

[abkro]

Finally got this to build properly, so I guess this is ready for you to proceed with the review, @peat-psuwit ?

[abkro]

@peat-psuwit I have addressed the specific points you mentioned, but the remaining one, i.e. your request that I proof read the whole thing better, is something that I really can't do much about, since my knowledge is the limiting factor here. Also, I cannot see any place I can mark this last requested change as addressed, so I guess that's up to you?

I will read through it all, but please understand that beyond things that I have actually tried out myself and properly understood, I don't actually have the necessary knowledge to be able to point out where the text contains factual mistakes.

What I can and will do is to attempt addressing this point as well with AI assistance and see if anything turns up. But this carries the risk of "going in circles", as I am sure you understand.

However, if you can point out additional relevant source material (code) that I can get the AI to analyze in order to better grasp the inner workings of the Ubuntu Touch system, that could be useful.

Lacking something like this, there is no other recourse than for someone who actually has the necessary knowledge to proof read this.

[abkro]

Actually, I have an idea for how to check and adjust the contents. I will close the PR for a bit while I do a some more research and adjustments. I'll reopen it once I have something I think warrants a closer look

[peat-psuwit]

Actually it's better to leave the PR open but mark as draft. See https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/changing-the-stage-of-a-pull-request#converting-a-pull-request-to-a-draft

[abkro]

Ok. Will fix when I get the chance.

[grenudi]

Wow! this is amazing rework ! I can't wait to see it on the web site, as current state of the porting guide is very outdated..