Syntax highlighting for PTX code

[Ellipse0934]

Removes :snake: pygments dependency and adds highlighting similar to base.

We can create a different highlighting scheme which should be easy to do but I don't have any ideas about that.

We have a simple lexer which processes one line at a time.

In general a PTX file has

• pre-processor directives: We assume these will not be present
• Comments: We assume there will never be any multiline comments
• Statements: Which are either be a directive or an instruction
• Directive: e.g. .reg .b32 r1, r2
• Instructions: e.g. label: mov.b32 r1, %tid.x
• Labels, Instruction name (mov, st, ld), data types (b32, f64),
• Identifiers and Numbers (NVIDIA PTX docs)

We can use regex to distinguish between the various tokens and do a more complex theme like: But I would personally not prefer that.

[codecov[bot]]

Codecov Report

Merging #275 (64f6123) into master (09f8b5a) will decrease coverage by 0.55%. The diff coverage is 2.43%.

@@            Coverage Diff             @@
##           master     #275      +/-   ##
==========================================
- Coverage   86.75%   86.19%   -0.56%     
==========================================
  Files          22       22              
  Lines        2023     2246     +223     
==========================================
+ Hits         1755     1936     +181     
- Misses        268      310      +42     

Impacted Files	Coverage Δ
src/reflection.jl	58.86% <2.43%> (-19.40%)	:arrow_down:
src/execution.jl	92.00% <0.00%> (-8.00%)	:arrow_down:
src/ptx.jl	93.67% <0.00%> (-2.65%)	:arrow_down:
src/error.jl	28.00% <0.00%> (-2.44%)	:arrow_down:
src/jlgen.jl	80.58% <0.00%> (-1.73%)	:arrow_down:
src/gcn.jl	84.40% <0.00%> (-1.04%)	:arrow_down:
src/irgen.jl	94.05% <0.00%> (-0.96%)	:arrow_down:
src/mcgen.jl	97.36% <0.00%> (+0.07%)	:arrow_up:
src/validation.jl	96.92% <0.00%> (+0.12%)	:arrow_up:
src/debug.jl	97.22% <0.00%> (+0.16%)	:arrow_up:
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 09f8b5a...64f6123. Read the comment docs.

[maleadt]

Thanks! Pushed some small changes.

I think we'll have to improve the tokenizer before this can work though, simple things like ret; or __test = {1, 2} are mismatched into resp. ret;, {1, 2} tokens. I wonder how hard it would be to just use a proper parser generator like RBNF.jl. although I couldn't find a simple BNF grammar for PTX. Here's one I found for ANTLR: https://gist.github.com/maleadt/44966c96fa389b857910ccd3a4286bf6

[Ellipse0934]

Yes, I missed the tokeniser failures. I have added some additional regex to identify the type of token. Also, some colour changes have been reverted but we will fix that before merge. I wanted to check if the current strategy is fine. It seems easy to add support for functions and other features.

I wonder how hard it would be to just use a proper parser generator

Is there any use for this besides code highlighting ? I don't think a BNF for this should be particularly difficult but then I don't have any experience in this area. There do appear to be some ANTLR to BNF converters out there. Maybe it's worth looking into one of those.

Remaining TODOs:

• [ ] Add missing instructions/declarations (e.g call/bra.uni , %p<0-..>)
• [ ] Tests
• [ ] Final colour scheme

[maleadt]

s there any use for this besides code highlighting ? I don't think a BNF for this should be particularly difficult but then I don't have any experience in this area. There do appear to be some ANTLR to BNF converters out there. Maybe it's worth looking into one of those.

Currently not, no. If we can get reasonable regex-based highlighting, it's probably overkill to consider a proper grammar (which would also incur some latency when getting compiled), but if we end up having to write down individual regular expressions for every token type it's a small step to a real grammer (where expressing those might be more compact even).

But I'm just spitballing here, I don't have experience with parser generators in Julia either.

[Ellipse0934]

I wrote some implementations and got confused as to whether we want to be minimalist here or do something closer to an actual parser with proper tokens. The Julia replacement for Pygments is Highlights.jl and they have an issue on this. The best course of action imo is to implement this there and use that here. I am going to implement a lexer there first. If that one is unsatisfactory I can resume this PR otherwise close this PR.

[maleadt]

I didn't know about Highlights.jl -- that looks to be the best place to implement PTX support, indeed. Note that you can also find some regular expression-like things in the PTX ISA manual, e.g. https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#identifiers, that may make it easier to implement a lexer.