LADSoft
Adding #embed
This does NOT include features such as the prefix and postfix or limit, which are needed for standards compliance.
This draft PR is KNOWN to not build, half of this PR's purpose is allowing me to show a proposal for the design of the internal hook mechanism of the compiler so that we're not wasting cycles making the compiler recompute known values.
The other half is that it seems like the compiler chokes on #809 and this needs to be investigated separately, for the immediate future it seems like I can just do a boolean check instead of comparing against nullptr.
Furthermore, I added a commit that temporarily alleviate #809 ( 30673702e59b78222fef71d4986d2bdab260cf44 ) however that has a different access violation with a nullptr'd std::list.
It's been awhile so I will try debugging these but I'm dropping this here as I have no idea the source for the latter and the former should 100% be fixed, I don't know what I did with this file but it just breaks things.
yeah hooking the preprocessor that way is what is intended...
im glad you are working on this, I've been thinking maybe you are right, and the language specifications for C23 and C++17 are low-hanging fruit that should be addressed sooner rather than later...
i rewrote the VS projects last night so you might have to redo a little work...
yeah hooking the preprocessor that way is what is intended...
Moreso I'm interested in the hook between the preprocessor back to the declaration engine, since in order to make this fast we need to actually be able to insert the values into the declaration during compile, otherwise we're doing a lot of extra tokenization for little-to-no value.
In terms of effort this amounts of a minor rewrite of the declaration engine in its entirety as far as I'm aware to deal with this fact.
you wouldn't have to modify the entire declaration sequence, just the initialization aggregrator.
I would be tempted to use some kind of trick, like when intialize finds a structure/array to initialize it saves the current symbol somewhere, then when #embed is encountered it calls some callback that fills in the data for the variable (and maybe sets its size if it is an array????) and finally when it is all done a 'special' token is put into the input stream which tells initialize_aggregate_type not to do any processing... that would also scale well if you have a structure where you want to use #embed for one part of the structure but want to fill in things in code for another part of it... but i guess there would have to be a check somewhere if you are initializing a structure or sized array that would that you don't overrun the end...
I would be tempted to use some kind of trick, like when intialize finds a structure/array to initialize it saves the current symbol somewhere,
My thoughts were that during the declaration/initialization sequence after the declarator is found we set the embed_function callback I made with the current information required from the lexer captured as part of it (since std::function can allow for that) for the declaration then continue during the initialization sequence we just append new bytes as needed, so that constructs such as:
char thingy[] = {
0, 1, 2, 3,
#embed <file> limit(12) prefix(4, 5, 6) postfix(19, 20, 21)
, 22, 23, 24 // (note the first comma)
};
Work correctly. I may not actually do this portion on this PR but I will definitely be thinking about it during it...
that would work; you might further be able to further speed it up by making some class 'Embedded Data' which has an accessor to return the next byte of data... then make a new lexeme type that would take a pointer to that class as data. Then in expression_primary when you run into it you get the next byte of data and make an 'expression', 'type' pair for it which you return as normal.
That way the rest of the code for parsing initializers remains completely unchanged. You wouldn't call getsym() until you ran out of data so it would keep processing the same lexeme token over and over until it was done... All that would get around having to build a ton of lexeme structures to reflecting the incoming data...
@chuggafan, I finally read the docs for this, and I was thinking, we might have an extension that changes word size and endianness... that could be added later though. Anyway I'm mostly through with the C23 stuff although haven't pushed the latest yet... that leaves this as pretty much the only outstanding issue for C23. I was wondering what the status is? Will you have time to work at it sometime soon? I can work with you on the interface to the compiler if that is worrying you...
I was working on the interface change, but between vacation + other things (and the fact that the change is just so large) I have been on a semi-pause, I have uploaded my current work so you can see what I'm doing, if you want me to pursue that method still instead of going after another in a bit, I'll continue working on it in the background, here's the current work, it doesn't compile at all (in fact just compiling for the preprocessor gives me 302 errors), but it still requires some extraneous work (e.g. substituting the static IsSymbolChar in main() for the proper one, giving PreProcessor a default (which will happen last), etc.)
i took a quick look and what you are doing seems reasonable. But we are possibly going to have to update the compiler for those function-pointer-as-template argument things you've got going, I don't know what it will do with them.
We aren't in any hurry on finishing up the 'embed' tag I guess, so you can keep going along these lines if you want.
I'm currently not done with the work on this draft pr, as it's SIGNIFICANTLY ballooned in scope (the change from cpp files to .h files for a number of things including kw enums for stuff like the preprocessor basically means I have to namespace large sections of code in order to ensure that it's correct).
I am also debugging some random MS issues with my code that aren't exactly clear in their error reports, so I am also trying to diagnose that (what fun).
In the meantime, this PR may die for quite some time and I might even switch to trying to send patches another way because github is requiring 2fa via some phone or secret app for my account, and because this account is tied with my "public" identity rather than my "private" (offline) identity I am not putting a phone number within lightyears of this account. So this may languish for quite some time if I am no longer able to do things such as make commits as of tomorrow. I will try to upload all of my current work tonight, however.
my interpretation is if I didn't do the 2fa by the deadline, I would lose access to my github account going forward.... after the deadline there is no longer a way to log into the account if you don't have 2fa... so that was something to think about...
i ended up using 1password for 2fa, as I could put it on my computer and it didn't bother the login process terribly, but that is something you buy. Don't know if the other alternative I was given was any better; at that point I didn't much care as I was very confused by the whole process anyway and just wanted it over with lol...
I didn't care much if they had my phone number so I got the github app as a backup...
hope to see you around going forward :smile:
I've did the two factor with aegis 2fa, no private linking or mobile needed.
I'm not sure why one would need (or even use) a proprietary 2 factor application when not explicit needed. Aegis works really well on Android https://getaegis.app/ and if running Windows 11 you can also use it via the Windows Subsystem for Android. On Linux Waydroid works quite fine, too.
So @chuggafan Are you still there?
Apparently Github is doing a discord style "We will bother you once a week until you comply", which means I can continue to converse, which is nice.
In other news: I have begun on namespacing the entirety of ocpp in order to help with one of the issues I pointed out in my most recent commit's infodump.
@GitMensch
yeah i might have done something different if I had any clue what I was doing lol... but it is worth noting that I tried to browse to the aegis web site the other day after you brought it up, and my antivirus warned me that the site was blacklisted...
Hi @chuggafan , wondering if you are still around after the 2FA problems or if I should tackle this one myself?
I haven't been working on this PR (and the 2FA problems actually increased) due to increasing amounts of work on my end, my likely restart date for finishing the rest of this is in mid-december as that's when I have less time on my hand, if you want to finish off what I started here that's no issue on my part, I've just been buried under an unusually heavy amount of work.
On 11/25/23, David Lindauer @.***> wrote:
Hi @chuggafan , wondering if you are still around after the 2FA problems or if I should tackle this one myself?
-- Reply to this email directly or view it on GitHub: https://github.com/LADSoft/OrangeC/pull/810#issuecomment-1826387427 You are receiving this because you were mentioned.
Message ID: @.***>
Oop, I mean I have more time on my hand mid-december.
Also as you can see I'm communicating over email, it seems that github really did eventually lock my account out until 2fa was used, and my lazyness on investigating alternative routes is really high coupled with an unusually high workload is going to be cutting off whatever chances I have to work on side-projects. Sadly, this means I've basically been ignoring this for now.
On 11/25/23, chugga fan @.***> wrote:
I haven't been working on this PR (and the 2FA problems actually increased) due to increasing amounts of work on my end, my likely restart date for finishing the rest of this is in mid-december as that's when I have less time on my hand, if you want to finish off what I started here that's no issue on my part, I've just been buried under an unusually heavy amount of work.
On 11/25/23, David Lindauer @.***> wrote:
Hi @chuggafan , wondering if you are still around after the 2FA problems or if I should tackle this one myself?
-- Reply to this email directly or view it on GitHub: https://github.com/LADSoft/OrangeC/pull/810#issuecomment-1826387427 You are receiving this because you were mentioned.
Message ID: @.***>
So I've decided I'm probably going to scrap the rewrite of the parser
engine, and do a temp-roll-out of my own state-machine-based parser
for #embed next time I have enough free time to work on this, I've
spent wayyyy too much time on trying to solve newer and newer issues
each time I come back to this and I've frankly grown tired of running
into issues that require I go back through every line to fix
everything.
The most recent happening is that I tried rewriting the Preprocessor because when I swapped out the enum system the fact that not every enum has an Assign parameter came up, and the way the code was written just effectively blocked me and required a large rewrite so that everything was being dealt with in preprocessor for the actual template structure creation for the Preprocessor type, just to reduce the amount of code being written.
Because it's taking too long, I'm going to investigate when I have
free time just reverting to my pre-Preprocessor changes where I only
touched #embed and just statemachining out everything I was
originally intending to parse, it'll be faster and cleaner for now and
whenever we want to change up the preprocessor again so that we can
have multiple of them in one project we can cross that rubicon when we
get there.
On 11/25/23, chugga fan @.***> wrote:
Oop, I mean I have more time on my hand mid-december.
Also as you can see I'm communicating over email, it seems that github really did eventually lock my account out until 2fa was used, and my lazyness on investigating alternative routes is really high coupled with an unusually high workload is going to be cutting off whatever chances I have to work on side-projects. Sadly, this means I've basically been ignoring this for now.
On 11/25/23, chugga fan @.***> wrote:
I haven't been working on this PR (and the 2FA problems actually increased) due to increasing amounts of work on my end, my likely restart date for finishing the rest of this is in mid-december as that's when I have less time on my hand, if you want to finish off what I started here that's no issue on my part, I've just been buried under an unusually heavy amount of work.
On 11/25/23, David Lindauer @.***> wrote:
Hi @chuggafan , wondering if you are still around after the 2FA problems or if I should tackle this one myself?
-- Reply to this email directly or view it on GitHub: https://github.com/LADSoft/OrangeC/pull/810#issuecomment-1826387427 You are receiving this because you were mentioned.
Message ID: @.***>
Sorry the 2fa is a mess. I really wish we didn't have to do it... but maybe it will give me an excuse to start actually using a password manager... I've got it now but all I use it for is githum... lol...
I was working on getting the libc++ tests to compile again... then I got into working on fixing the constexpr support, apparently some stuff got added in C++17. That is kinda where I am now... rewriting the constexpr module to make it better lol...
Whenever I'm done with that I'm going to try a couple of things I thought of to speed up compiles and then it would be time for a release. It would be nice if we could also wrap up the #embed and get c2x finished up as well. Realistically we are several weeks out from a release...
I can try, I had a break which allowed me to do some things, but I've realized I would have to rewrite a new parser system anyways to notice, but not necessarily throw around warnings and errors during the parse, during said break as I was re-reading a newer variant of the standard. https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3220.pdf
Part of it is the method of implementation necessary for __has_embed and
#embed requires that we can identify and detect unknown keywords and
allow for people to fall back onto the standard as an alternative to the
extensions that some compilers might have, so I have to parse and identify
the non-existent keywords required for this, I did successfully revert back
to my original implementation and begin a quick state-machine based
alternative system however, which I plan on using more of this weekend.
An excellent example of some of the things I am considering and trying to figure out how to handle include:
#if __has_embed(__FILE__ ext::token(0xB055))
#define DESCRIPTION "Supports extended token embed parameter"
#else
#define DESCRIPTION "Does not support extended token embed parameter"
#endif
This needs to be able to be processed using the same processing as #embed
without any of the errors cropping up, while also allowing the regular
#embed to error, and so I'm going to be working on the __has_embed
aspect first as it's more intricate and complex than I first thought nearly
a year ago when beginning this before I stopped having time.
On Sun, Mar 17, 2024 at 11:26 AM David Lindauer @.***> wrote:
Sorry the 2fa is a mess. I really wish we didn't have to do it... but maybe it will give me an excuse to start actually using a password manager... I've got it now but all I use it for is githum... lol...
I was working on getting the libc++ tests to compile again... then I got into working on fixing the constexpr support, apparently some stuff got added in C++17. That is kinda where I am now... rewriting the constexpr module to make it better lol...
Whenever I'm done with that I'm going to try a couple of things I thought of to speed up compiles and then it would be time for a release. It would be nice if we could also wrap up the #embed and get c2x finished up as well. Realistically we are several weeks out from a release...
— Reply to this email directly, view it on GitHub https://github.com/LADSoft/OrangeC/pull/810#issuecomment-2002508670, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSPBTTOTXV2TRDZUBE6B43YYWY2HAVCNFSM6AAAAAAZZEIDAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBSGUYDQNRXGA . You are receiving this because you were mentioned.Message ID: @.***>
being able to live with arbitrary sequences while still parsing it properly sounds like a difficult prospect. It would be nice to put this to bed though..l
Have gotten back to this, now that my time is a bit freer. I made some commits and made a test push to see what was going on with the entire system, turns out that I am being stupid because I forgot that AsmToken.cpp and AsmToken.h:
- exist
- Matter and need to be brought under the Nu-Tokenizer fold
Thankfully, I did the light redesign for actually heavier duty purposes, so updates to this should be much faster than my original attempt at doing this modification to the ENTIRE preprocessor, likely blockers however will include using keyword values directly, instead of having designated certain values to certain meanings (e.g. an UNKNOWN keyword being -1, always)
No ETA on finishing, but I should be able to go and dedicate way more time in my life to this soonish, at this point this is all me trying to get to the point that I'm comfortable enough with everything compiling with the new Tokenizer, then (ab)using that Tokenizer for the new embedded token processing so that I can parse out generic sequences and ignore them if need be.
Should be fun, but challenging.
On Sun, Mar 17, 2024 at 7:19 PM David Lindauer @.***> wrote:
being able to live with arbitrary sequences while still parsing it properly sounds like a difficult prospect. It would be nice to put this to bed though..l
— Reply to this email directly, view it on GitHub https://github.com/LADSoft/OrangeC/pull/810#issuecomment-2002653799, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSPBTTNXFQ64OEZSWSADB3YYYQH5AVCNFSM6AAAAAAZZEIDAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBSGY2TGNZZHE . You are receiving this because you were mentioned.Message ID: @.***>
Ok, now that I'm returning to working on this even more, I've noticed that the MSVC build system HATES WITH A BURNING PASSION the changes to .lib and .exe, I'm trying to debug stuff and it will just throw errors at me. When I attempt to fix this, I get major issues where the output just decides to completely die, do you recommend I just modify all the projects to no longer use TargetName, just use their actual name, or something else? This is causing issues such as not properly rebuilding for debugging when I want it to in MSVC.... I've attached the output here. I'll probably just try to rename everything on my side, but I want to work on a compromise solution since this will touch every project file again no matter the fix, and both of us are independently working on changes that affect a lot of things, directly or indirectly.
On Thu, Apr 18, 2024 at 8:55 PM David Lindauer @.***> wrote:
im glad you are back to it....
— Reply to this email directly, view it on GitHub https://github.com/LADSoft/OrangeC/pull/810#issuecomment-2065557904, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSPBTQ6D4KUAONKSQ7MT73Y6BTPZAVCNFSM6AAAAAAZZEIDAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRVGU2TOOJQGQ . You are receiving this because you were mentioned.Message ID: @.***>
1>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(1554,5): warning MSB8012: TargetPath(C:\OrangeC\src\Debug\ocpp.lib.lib) does not match the Library's OutputFile property value (C:\OrangeC\src\Debug\ocpp.lib). This may cause your project to build incorrectly. To correct this, please make sure that $(OutDir), $(TargetName) and $(TargetExt) property values match the value specified in %(Lib.OutputFile). 1>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(1556,5): warning MSB8012: TargetName(ocpp.lib) does not match the Library's OutputFile property value (ocpp). This may cause your project to build incorrectly. To correct this, please make sure that $(OutDir), $(TargetName) and $(TargetExt) property values match the value specified in %(Lib.OutputFile). 1>ocpp.lib.vcxproj -> C:\OrangeC\src\Debug\ocpp.lib.lib 1>Done building project "ocpp.lib.vcxproj". 2>------ Build started: Project: ocpp.exe, Configuration: Debug Win32 ------ 2>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(531,5): warning MSB8028: The intermediate directory (Debug) contains files shared from another project (ocpp.lib.vcxproj). This can lead to incorrect clean and rebuild behavior. 2>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(1392,5): warning MSB8012: TargetPath(C:\OrangeC\src\Debug\ocpp.exe.exe) does not match the Linker's OutputFile property value (C:\OrangeC\src\Debug\ocpp.exe). This may cause your project to build incorrectly. To correct this, please make sure that $(OutDir), $(TargetName) and $(TargetExt) property values match the value specified in %(Link.OutputFile). 2>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(1394,5): warning MSB8012: TargetName(ocpp.exe) does not match the Linker's OutputFile property value (ocpp). This may cause your project to build incorrectly. To correct this, please make sure that $(OutDir), $(TargetName) and $(TargetExt) property values match the value specified in %(Link.OutputFile).
Good news is I have no intention of touching the VS build process any time soon, I'm just reorganizing existing source files without adding or removing any... so you have freedom to work on this...
maybe putting two projects in the same directory is the first questionable thing I did... the second one being misuse of the TargetName.
For the first issue we probably ought to make separate directories to put the project files for the executables in. We need to leave the sources where we are though when we do this, so that command line builds won't be affected.. That itself might be enough. But if it isn't or you want to go further with it I'm ok with taking the extensions off the project names and restoring the proper relationships between all those variables. But you might have to rename the library files as you can't have two projects with the same name....
Just out of curiousity, do you know how to edit the project files by hand? Sometimes using an editor can be easier than using the IDE dialogs...
I use VSCode as my primary editor these days, but I use the IDE dialogs specifically to stay entirely within the process occasionally... I will probably be experimenting with this off and on again over the next week/week and a half, but I agree with your assessment that the two projects in the same directory is likely the (main) source of this, the secondary source is renaming the target without relabeling the corresponding linker outputs and inputs. Both combined are the deadly combination here. I think if it were possible to make it so that the "static library" and the "exe" were linked intrinsically in properties that would be good, but that sounds like a pain to even think of accomplishing.
On Wed, May 8, 2024 at 10:20 PM David Lindauer @.***> wrote:
Good news is I have no intention of touching the VS build process any time soon, I'm just reorganizing existing source files without adding or removing any... so you have freedom to work on this...
maybe putting two projects in the same directory is the first questionable thing I did... the second one being misuse of the TargetName.
For the first issue we probably ought to make separate directories to put the project files for the executables in. We need to leave the sources where we are though when we do this, so that command line builds won't be affected.. That itself might be enough. But if it isn't or you want to go further with it I'm ok with taking the extensions off the project names and restoring the proper relationships between all those variables. But you might have to rename the library files as you can't have two projects with the same name....
Just out of curiousity, do you know how to edit the project files by hand? Sometimes using an editor can be easier than using the IDE dialogs...
— Reply to this email directly, view it on GitHub https://github.com/LADSoft/OrangeC/pull/810#issuecomment-2101809916, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSPBTS5XNHMMQJ5QDZUM53ZBLMOLAVCNFSM6AAAAAAZZEIDAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBRHAYDSOJRGY . You are receiving this because you were mentioned.Message ID: @.***>
I've been doing slow work, I've decided that I was going to scrap what I did and redo the tokenizer instead of with my own tokens, with the original kw tokens.
BTW: In the check sequence, is there a way I can get a preprocessed line? Or is there some trickery? I tried moving the embed check to the last of the checks but that didn't work in getting a fully pre-processed line with macros expanded.
On Mon, May 13, 2024 at 8:26 PM David Lindauer @.***> wrote:
so i was wondering if you are having any time to work on this?
— Reply to this email directly, view it on GitHub https://github.com/LADSoft/OrangeC/pull/810#issuecomment-2109050007, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSPBTUY47HLLO5YSZQOP3LZCFK4NAVCNFSM6AAAAAAZZEIDAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBZGA2TAMBQG4 . You are receiving this because you were mentioned.Message ID: @.***>
cool!
So anyway all the 'check functions' have an 'args' argument, I believe that is the full contents of the preprocessed line, without the initial token though. So I think for a line like this:
#define MY_DEF(x,z) ((x) * (z))
the args string passed into the check function for #defines woudl be:
MY_DEF(x,z) ((x) * (z))
and then the check function would parse the line of data to do what it needs to with it.
but yeah looking at it the name 'args' is a little misleading. You are welcome to rename it throughout if you want to... that is one of the things I've historically failed at, picking useful names for things... fwiw im trying to clean it up at least a little as I go through restructuring compiler sources.
The "Args" doesn't seem to be giving me pre-processed items, I've been running a test file attached and the line I've been receiving is as follows:
<Servos.png> limit(1024) __prefix__(1, 2, 3, 4) my_ext::ext(1, thing, thing + 2, 1, 1)
What I'm looking for here is thing, thing +2 to be compressed into their
actual value (hopefully), is that possible in this context or do I have to
pre-evaluate this somehow?
On Mon, May 13, 2024 at 9:18 PM David Lindauer @.***> wrote:
cool!
So anyway all the 'check functions' have an 'args' argument, I believe that is the full contents of the preprocessed line, without the initial token though. So I think for a line like this:
#define MY_DEF(x,z) ((x) * (z))
the args string passed into the check function for #defines woudl be:
MY_DEF(x,z) ((x) * (z))
and then the check function would parse the line of data to do what it needs to with it.
but yeah looking at it the name 'args' is a little misleading. You are welcome to rename it throughout if you want to... that is one of the things I've historically failed at, picking useful names for things... fwiw im trying to clean it up at least a little as I go through restructuring compiler sources.
— Reply to this email directly, view it on GitHub https://github.com/LADSoft/OrangeC/pull/810#issuecomment-2109092404, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSPBTTCW7CCIHAWBLS3YRLZCFQ77AVCNFSM6AAAAAAZZEIDAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBZGA4TENBQGQ . You are receiving this because you were mentioned.Message ID: @.***>
#include "stdio.h" #define thing 111 char vals[] { #embed <Servos.png> limit(1024) prefix(1, 2, 3, 4) my_ext::ext(1, thing, thing + 2, 1, 1) };