Need full working examples in the documentation

[esternin]

I am new to critcl. Trying to follow the manual, I find myself yearning for more simple examples. The moment I deviate from the one "simple example" on offer, and try to interpret the formal API descriptors, I get stuck.

The document Using CriTcl, "an introduction to critcl by way of a of examples", mentioned in the Developer's Guide (section "Playing with critcl") without a link, does not seem to exist anywhere. Perhaps it was replaced with the examples subdirectory, but I did not find these self-explanatory.

This is how I got stuck right away: the "simple example" is a C function that returns a numeric value. This works fine, but what if I needed to return a string, or a structure that may contain a variety of things?

Here's what I tried:

package require critcl

critcl::cproc IntFromC {int i} int { return i*2; }

critcl::cproc FloatFromC {int i} float { return (float)i*2.0; }

critcl::cproc StrFromC {int i} string { 
  #include <stdio.h>
  char str[60];
  sprintf(str,"%d",2*i);
  fprintf(stderr," in C: %d -> \"%s\"\n",i,str);
  return str;
  }

set n 99
set nn [IntFromC $n]
puts " in TCL: $n -> C as int -> $nn"

set ff [FloatFromC $n]
puts " in TCL: $n -> C as float -> $ff"

set response [StrFromC $n]
puts " in TCL: $n -> C as str -> \"$response\""

And here's what I get:

 in TCL: 99 -> C as int -> 198
 in TCL: 99 -> C as float -> 198.0
 in C: 99 -> "198"
 in TCL: 99 -> C as str -> ""

So the number gets into C, a string gets properly generated in C, but the very string that is OK for printf(" %s ") returns to tcl as empty. I tried a variety of tcl types, as per (but I am having a hard time understanding the differences between various return types):

char*
vstring

    A String Tcl_Obj holding a copy of the returned char* is set as the interpreter result. If the value is allocated then the function itself and the extension it is a part of are responsible for releasing the memory when the data is not in use any longer.
const char*

    Like char* above, except that the returned string is const-qualified.
string
dstring

    The returned char* is directly set as the interpreter result without making a copy. Therefore it must be dynamically allocated via Tcl_Alloc. Release happens automatically when the Interpreter finds that the value is not required any longer.

This is not an "issue" with the code, only with the user-facing documentation, but I am hoping a guru will offer some insight.

[andreas-kupries]

As quick answer, the Using Critcl document is at https://andreas-kupries.github.io/critcl/doc/files/critcl_usingit.html

It is titled FAQ / Examples on https://andreas-kupries.github.io/critcl/

The missing link from the dev guide is a bug ... Will look into that also.

Edit: And even the using critcl should be extended with a bit more sections on strings and Tcl_Obj before diving into the custom types. To be done.

[esternin]

As quick answer, the Using Critcl document is at https://andreas-kupries.github.io/critcl/doc/files/critcl_usingit.html

Yes, that is what I've been using, and that's the one that only has one "simple example".

(thanks for the link to additional examples, looking at them now)

Another minor bug: https://andreas-kupries.github.io/critcl/doc/files/critcl_sources.html has a broken link:

"...can be found directly at its download page."

[andreas-kupries]

Another minor bug: https://andreas-kupries.github.io/critcl/doc/files/critcl_sources.html has a broken link:

Fixed in the #114 PR. Also fixed is the cross-reference to the Using Critcl document. This is not up on the website of course. That said the directory embedded/www of a critcl checkout contains the updated HTML files.

Regarding clibraries

The way I interpreted the documentation was to use two calls ...

May I assume that you are talking about https://andreas-kupries.github.io/critcl/doc/files/critcl_usingit.html#subsection13 ?

[esternin]

Yes, exactly.

[esternin]

I have made some progress, guided by something I saw in one of your examples (using Tcl_Obj to pass strings - this definitely needs to be described).

I also spent a good portion of today on a stupid thing that is obvious in retrospect, but was not clear from the reading of the manual, namely how to deal with #includes and declarations. I kept repeating the same #includes in every cproc, and since the compiler errors were all over the place, I spent hours trying increasingly bizarre things to sort this out.

Somewhere the documentation needs to offer a basic skeleton:

# this is a skeleton tcl script 
# making use of C subroutines from an external library "xxxx.so"

package require critcl
critcl::cheaders -I/opt/xxxx/include
critcl::clibraries -L/opt/xxxx/lib/x86_64
critcl::clibraries -lxxxx

critcl::ccode {
  /* all C code will be compiled at once, so keep #include's in one place */
  #include <tcl.h>
  #include "xxxx.h"
  #define XXXX_VERSION 0.1
  }

# calls to individual C subroutines go here 

# this C call takes in nothing and returns an integer
critcl::cproc xxxxInt {} int {
  int i = xxxxCint();
  return i; 
  }

# this C call takes in an integer and returns a string
critcl::cproc xxxxStr {int i} object0 {
  char str[64];
  strcpy(str,xxxCstr(i));
  Tcl_Obj* ret = Tcl_NewStringObj (str, -1);
  return ret;
  }

# tcl code goes here
set n [xxxInt]
set s [xxxxStr $n]

I guess I should have used some malloc() calls instead of static sizes, and I am sure there are many improvements a guru could make to this, but something this trivial would have saved me - a noob - hours of frustration, so I think it might be useful to others as well.

I am now working on dealing with images and video data streams, so will be running into more problems with return types. It would be nice to have a series of such skeletons, for a variety of contexts, each building on the previous ones. One thing: they all need to be complete (if meaningless) examples, so that they could be copy-pasted and executed as is (IMHO).

You might disagree, and think that pushing the performance is the most important aspect of critcl, but to me the most important use of critcl will be in making use of external, optimized C libraries that will save me from porting things to tcl, and enable me to access interfaces that may only be available as manufacturer-supplied libraries, without needing to access the source code.

This critcl is so cool !

[andreas-kupries]

I have made some progress, guided by something I saw in one of your examples (using Tcl_Obj to pass strings - this definitely needs to be described)

Can you provide a more exact reference about example and code you used for guidance, i.e. file name, location, maybe a short quote, and then also the code you wrote based on that ? As is your text is unfortunately too vague for me to infer what helped you and how.

this is a skeleton tcl script

Skeleton - Makes sense ... Not sure if using one of the packages in the tarball I referenced at https://github.com/andreas-kupries/critcl/issues/115#issuecomment-1032341435 is too advanced already. ... Several skeletons are varying levels ...

Another recent package, and public is https://github.com/andreas-kupries/ankh . This one is a bit advanced, using Tcl for templating the C pieces.

You might disagree, and think that pushing the performance

I don't. While critcl can be used to write stand-alone C packages for use with Tcl, wrapping external libraries is a very important use case for it. Quite a lot of work like the ability to define custom argument and result types, and the various code generators were done in support of that use case, i.e. to make things simpler for the person writing the wrapper. Most of the examples/users at the top of https://andreas-kupries.github.io/critcl/ (**), AnKH, the examples referenced in https://github.com/andreas-kupries/critcl/issues/115#issuecomment-1032341435, are wrapping some external library. They also represent different phases of high-level support, i.e. the older ones use techniques which later got codified into generators and the like.

This critcl is so cool !

Thank you, and glad that you like it, despite your problems. Initial idea and coding by @jcw , later also @stevel6868 .

---

(**) CRIMP is the exception.

[andreas-kupries]

Some questions:

• Can you tell me which documents/web pages of the critcl documentation you read so far, and the order, at least roughly ? That would be interesting to know to see where I might need links to direct specific audiences. In your case a prospective user absolutely new to the system.

• For curiosity, where and how did you find Critcl ?

[esternin]

Can you provide a more exact reference about example and code you used for guidance, i.e. file name, location, maybe a short quote, and then also the code you wrote based on that ? As is your text is unfortunately too vague for me to infer what helped you and how.

Sure. In general, I found all examples, whether in #115 or in the examples and test directories too advanced (the latter were especially difficult because of a lack of descriptions of what they did), but from tzone: tzone-c.tcl lines 68-70, then from cdc-rabin: cdc.tcl, lines 38-40 were very important clues. My skeleton contains the lessons learned. The actual little test script I wrote is testASI.tcl (I had the C code return formatted tcl arrays as strings; the next step will be to learn to pass custom structures back and forth, maybe even to provide a full tcl binding to the C library). It produces the following sample output:

$ ./testASI.tcl 
Found 1 camera(s)
 --------- camera 0 info ------------------
cameraInfo(bitdepth)   = 12
cameraInfo(colour)     = no
cameraInfo(cooler)     = yes
cameraInfo(id)         = 0
cameraInfo(name)       = ZWO ASI294MM Pro
cameraInfo(pixelsize)  = 2.315nm
cameraInfo(resolution) = 8288x5644
cameraInfo(shutter)    = no
cameraInfo(usb3)       = yes
 --------- camera 0 controls --------------
cameraControls(AutoExpMaxExpMS)         = value 30000 min 1 max 60000 auto no
cameraControls(AutoExpMaxGain)          = value 285 min 0 max 570 auto no
cameraControls(AutoExpTargetBrightness) = value 100 min 50 max 160 auto no
cameraControls(BandWidth)               = value 80 min 40 max 100 auto yes
cameraControls(CoolPowerPerc)           = value 0 min 0 max 100 auto no
cameraControls(CoolerOn)                = value 0 min 0 max 1 auto no
cameraControls(Exposure)                = value 10000 min 32 max 2000000000 auto no
cameraControls(Flip)                    = value 0 min 0 max 3 auto no
cameraControls(Gain)                    = value 200 min 0 max 570 auto no
cameraControls(HighSpeedMode)           = value 0 min 0 max 1 auto no
cameraControls(Offset)                  = value 8 min 0 max 80 auto no
cameraControls(TargetTemp)              = value 0 min -40 max 30 auto no
cameraControls(Temperature)             = value 257 min -500 max 1000 auto no

*Can you tell me which documents/web pages of the critcl documentation you read so far, and the order, at least roughly ? That would be interesting to know to see where I might need links to direct specific audiences. In your case a prospective user absolutely new to the system.

After the initial installation trouble (see #114), I spent all my time alternating between these four: The Developer's Guide User Introduction FAQ / Examples Command Reference None of them seemed to have an entry point for a beginner (FAQ/Examples was the most helpful), I could never figure out how the material was divided among the four documents (esp. User Introduction, it was anything but), all of them seem to be written by, and for, (a) computer scientist(s), with much implied and expected. I will be returning to them all (and thanks for highlighting relevant ones), hopefulyl now I will be able to understand them better. Google searches for anything critcl were singularly unhelpful, an unusual case, but the need was great, so I persisted. I wonder, though, if a better introductory document/tutorial would improve the retention of a casual user stumbling upon critcl - and provide more google responses (right now, googling for critcl site:stackoverflow.com produces a grand total of four primary links).

For background, tcl/tk has been a good friend for close to 30 years now, but I am a physicist, not a professional coder, although I have written in Fortran, C, C++ and JavaScript, as well as various assemblers. We have a significant number of instruments that we support through tcl front-ends, sometimes eliminating, sometimes supplementing the manufacturer-supplied software, because the idea of throwing away good equipment due to software obsolescence offends me. In 2020 I assembled a box from a junk pile in my lab, with an ISA motherboard, and installed Win98 on it, just because this was the only way to support a proprietary instrument interface (a water leak destroyed its primary computer, born in 1994), and I promised myself this would be the last time.

*For curiosity, where and how did find Critcl ?

Googling for "calling C from tcl", probably through tcl.tk

[andreas-kupries]

all of them seem to be written by, and for, (a) computer scientist(s), with much implied and expected

Definitely by a computer scientist (i.e. me). ... And being well-versed in Critcl and its internals I may also be too near to the topic. Still, I will strive to be better.

User Introduction, it was anything but

Sad to hear ... Now, looking over it again I suspect that the Modes Of Operation, and System Architecture should be removed. The last should possibly go into the dev guide instead.

Thinking about the quadrants I suspect that I have the Reference nicely filled, yet everything else quite empty. The How-To contains only FAQ, Get Sources, and Installer's Guide for sure. Dev Guide currently does not fit anywhere.

This will have to be incremental work ...

tcl/tk has been a good friend for close to 30 years now

I came to Tcl/Tk in the later 90's. ... Side quest(ion): Were you aware of https://conf.tcl-lang.org/ ? (While the 2021 is done it might be interesting for you to present on your work with Tcl/Tk and Astro).

And greetings over to Canada ... Oh, near the Niagara's. Nice.

[esternin]

I came to Tcl/Tk in the later 90's. ... Side quest(ion): Were you aware of https://conf.tcl-lang.org/ ? (While the 2021 is done it might be interesting for you to present on your work with Tcl/Tk and Astro).

And greetings over to Canada ... Oh, near the Niagara's. Nice.

Never thought of participating in something like that, although I had used some of the slides from a European version of it, when I was sorting out BLT/RBC a few years back. We do have a few nice little things we could share, though not yet critcl-related, so... maybe. When is it in 2022? I really should upload them to my github, although they have been available for years from our server (e.g. PhysTks)

Yes, that's the rumble of the Falls you are hearing in the background... ;-)

In the meantime, another confusing bit in the documentation: interp. I grep-ed for interp alias, interp create etc. through the entire example set, and it just never gets called, yet a whole bunch of places use a pointer to an interpreter. How is it possible that a critcl::cproc gets a Tcl_Interp* on input, but I cannot find a calling invocation in tcl that creates this pointer in the first place? How does a cproc function set an equivalent of -code text message when returning a TCL_ERROR condition, back to the calling tcl process to catch? I can't seem to figure this out.

[andreas-kupries]

When is it in 2022?

No idea yet.

that a critcl::cproc gets a Tcl_Interp* on input

Look at the cdc package, cdc.tcl, line 48. It is a special argument which tells the translation layer to provide the function with a pointer to the current interpreter. This special argument does not appear as argument of the generated Tcl command. In the reference this is https://andreas-kupries.github.io/critcl/doc/files/critcl_pkg.html#7 and then scroll down to Attention: This is a special argument type..

It has nothing to do with passing in an interpreter of your own choice, and also nothing with the builtin interp command.

How does a cproc function set an equivalent of -code text message when returning a TCL_ERROR condition.

From correct.tcl, line 124:

if (wr < 0) { Tcl_SetErrorCode(ip, "AK", "IS", "CORRECT", "DECODE", NULL); Tcl_SetResult(ip, "unable to decode, too many losses/changes", TCL_STATIC); res = TCL_ERROR; goto done; }

This specific example uses the ok return type to simply return TCL__OK, TCL_ERROR, etc. and no other result. Similar to void.

It is the same for object0, object ... Return NULL to signal the error, and use standard Tcl_API calls to set error message and error code.

The simple type (int, ... and the string types) cannot return an error.

[esternin]

Thanks. I will take time to understand what you just said there, esp the correct.tcl code. I always thought that it is the responsibility of a subroutine to return an error condition and to set the error code, and the responsibility of the main to do something (or not) with those two. So I keep looking for a "set error condition to xxx" inside the critcl cproc, and "determine and interpret error condition" in tcl, and do not see that. I will need some time to get there, and I have to set this aside, at the moment the hardware does not seem to get or respond correctly to some of my cproc-wrapped commands.

[andreas-kupries]

I have started on the reorg and hopefully the new howto's will reach error signaling before you are getting back to working with critcl. :wink:

That said, would you be be willing to receive and review partial doc changes ? As a means of keeping me on track for a new user vs a knowledgable one.

And if yes, what method of communication would you prefer ? Mail ? Simply commit and push, and you work from a checkout ? Attachments to comments here ?

[esternin]

This "issue" is turning into more of a help-the-novice "forum" discussion. While terrifically useful to me personally, the instructional value to others may well be limited, but I guess once this process is finished, there will be a document that creates a less meandering path. Just thinking out loud...

I would be happy to discuss/contribute/review, since I think I am committed to make use of critcl (although at the moment, it's not working as expected with the hardware, but I do not know if this is related to critcl or the library I am using). As to the mechanism, I do not care, I can work through anything: email, or just keep plugging away here, or as comments on a push candidate, or whatever. If you want, you can even add me as a contributor, so I can push changes on a file - sometimes it's easier to do than to explain - which you could then reverse and I would not take offense - if this is more efficient for you. You could make this a subproject (critcl-tutorial, separate from revision/restructuring of the main docs) to keep me away from messing up anything else, and then fold it back in. Whatever you decide.

Before I forget: the compiler errors are really disorienting, I guess because what gets sent to the compiler is not necessarily what is written in the .tcl file. For example, I ran several times into typing a comma between the input variables on a cproc declaration (the mental switch to C was happening a bit early), and the resulting error is wildly far away, something about the syntax of, say strcpy() miles away, possibly even in a different cproc, so it was difficult to realize where one messed up.

[andreas-kupries]

Ok. In that case I believe I will simply push to a branch/PR you can then checkout. The readable docs will be under embedded/www ... I will mention again when I have stuff to review.

Compiler errors ... I guess you had code like foo { double x, double y}, or even foo { double x , double y } ... I would not see that as a compiler error, and more of an issue in critcl itself, i.e. not properly validating the argument names before placing them into the generated C code. And as that is generated C code it does not really have a reference to the location in the Tcl code.

A missing semicolon at the end of a statement in the command body on the other hand it should be able to assign the proper location to (It emits #lines directives by default).

Created #118

[esternin]

I have started on the reorg and hopefully the new howto's will reach error signaling before you are getting back to working with critcl. wink

Was that a challenge? ;-)

I tend to be very stubborn when the coolers are not cooling, so I will have the damn thing sorted before I go to sleep tonight. By now, I am pretty sure it's not a critcl issue, pun intended.

[andreas-kupries]

Was that a challenge? ;-)

Remember my qualifier of hopefully in the comment.

That said, #119 is the PR in which I work on the docs. Enough has been rewritten to show off the new style for introduction and how to documents, and review for suitability.

Details on how to look at the changed documentation is in the description of that PR.

And good night.

[esternin]

Well, the cooler is cooling just fine now...

Another point that I think deserves a section in the introductory documentation: what's the best way to pass large amounts of data (like a 50MB data block) to C and (my interest in particular) back to tcl?

Should one create a large data block in tcl and pass a pointer to it to C (is this critcl::cproc ... {char &data_block} {...} even possible?), or TCL_Alloc or malloc in C and then pass back through a large object0 - but then when does one release this memory, must be after return from cproc, so we must hold the pointer in tcl between calls to cproc? or the "not very useful" bytearray? [why? one could pass the size as a separate parameter?] or ... something else entirely?

What about a stream (like live video), can that be passed through C?

I also figured out what is not working with the libraries outside of the default LD_LIBRARY_PATH. The crit::clibraries does get used when compiling the code, but not when running it. The loader does not get passed this via, for example, the setting of LD_LIBRARY_PATH at run-time. I think this is a bug. The workaround is to only use libraries that are in default locations, like /usr/local/lib. This probably is a separate issue, I'll let you decide.

Should I be making these notes in #119?

[andreas-kupries]

I prefer them here, in the ticket.

[andreas-kupries]

what's the best way to pass large amounts of data (like a 50MB data block) to C and (my interest in particular) back to tcl?

I suspect it depends on what you intend to do with the data, i.e. how to process it, and maybe also on the target machine, i.e. available RAM.

Most of Tcl's datastructures have a 2G limit, i.e. string rep no large than 2G bytes, lists no more than 2G elements, etc.

If you look at the zstd wrapper among the examples you will see that these commands take and return byte arrays. Of course, if your data is text, taking and returning strings is ok.

The functions in the correct wrapper OTOH read from and write to channels. Could have used bytearrays for them as well.

[andreas-kupries]

"not very useful" bytearray? [why? one could pass the size as a separate parameter?]

Sure. Would look like a hack however. With a bytes type argument you invoke foo $data. With bytearray you would have to run foo $data [string length $data] and why do that if you have a type which does this for you internally ?

... LD_LIBRARY_PATH. The crit::clibraries does get used when compiling the code, but not when running it.

I thought about this over the weekend and I am still of two minds about it. Not sure how portable using -rpath would be. Not sure if I extending env(LD_LIBRARY_PATH) in Tcl before loading the package library will be taken into account by the dyn loader. ... Oh, also note, HP machines use SHLIB_PATH instead IIRC. No idea for Windows.

So, in the end, I suspect that it can be done. However do I want the entire portability stuff coming with it ?

[esternin]

... LD_LIBRARY_PATH. The crit::clibraries does get used when compiling the code, but not when running it.

I thought about this over the weekend and I am still of two minds about it. Not sure how portable using -rpath would be. Not sure if I extending env(LD_LIBRARY_PATH) in Tcl before loading the package library will be taken into account by the dyn loader. ... Oh, also note, HP machines use SHLIB_PATH instead IIRC. No idea for Windows.

So, in the end, I suspect that it can be done. However do I want the entire portability stuff coming with it ?

I think the issue is the inconsistency, because C is a compiled language, and tcl is interpreted, I am expecting that if the code did not raise a flag at compile time - it found the routines specified in crit::clibraries - then the code should run as well. But at run time, the same library is found in a different way, through the loader, and no longer resolves. So that's confusing.

If you decide to keep things as they are, I think it would be very useful to have a warning at compile time that the library specified in crit::clibraries is not in a standard location and may need to be either moved, or explicitly specified through the LD_LIBRARY_PATH or its equivalent.

Edit: even more importantly, what happens if there are two versions of the library (let's say you are a developer, and are working on a new version of something already installed on the system). When you specify crit::clibraries, you also expect that same version to be used at run time, not the one installed in /usr/lib, no?

[esternin]

what's the best way to pass large amounts of data (like a 50MB data block) to C and (my interest in particular) back to tcl?

I suspect it depends on what you intend to do with the data, i.e. how to process it, and maybe also on the target machine, i.e. available RAM.

Most of Tcl's datastructures have a 2G limit, i.e. string rep no large than 2G bytes, lists no more than 2G elements, etc.

OK, my definition of big is not in GB range, but tcl is slow to process images, so a C speedup would be a very welcome thing, I think - I envision C-based filters and deconvolutions, but interface in tcl. Upon reading, bytearray attracted my attention right away, and I was stopped by the comment of it being useless. For transparency, I would prefer to create an image structure in tcl, but do I want passing the whole 50MB through the stack? So I need to understand the Tcl_Obj stuff better. Again, it's a documentation thing...

I am looking at zstd and correct - thanks! - trying to figure out why they do what they do... I am slow.

[andreas-kupries]

Upon reading, bytearray attracted my attention right away, and I was stopped by the comment of it being useless

Ok. Greping through the docs it was only in the standard reference pages. Not in the how to use. That at least is good. The problem with binary data is that you cannot use the presence of \0 byte as terminator. You have to know the length. And the bytearray type simply does not provide that.

Tcl_Obj. The main structure used inside Tcl, and in the API to handle any kind of value. Passed around via pointers, reference counted, copy on writing, i.e. when a shared value (refcount > 1) needs to be changed. Maintains 2 representations, string and internal. The internal representation is type dependent, see the Tcl_ObjType structure for the set of function vectors needed to implement a new type. This is pretty much Tcl C API basics these days. I.e. not specific to Critcl. I will have to think how/where to provide links to external documentation - f.ex. https://wiki.tcl-lang.org/page/Tcl_Obj

Regardless, both string and the structures for complex internal reps are usually allocated on the heap, and stacks see only the Tcl_OBj*, i.e pointers.

As an aside, my old and stalled CRIMP image processing package implements a Tcl_ObjType to literally hold images a values and pass threm around.

A slightly newer attempt is https://chiselapp.com/user/andreas_kupries/repository/aktive/timeline

Both are stalled at the moment. Aktive might be easier to handle, as it does not do the heavy use of Tcl for templating as CRIMP does. I cannot guarantee that either is in a buildable state.

[esternin]

... LD_LIBRARY_PATH. The crit::clibraries does get used when compiling the code, but not when running it.

I thought about this over the weekend and I am still of two minds about it. Not sure how portable using -rpath would be. Not sure if I extending env(LD_LIBRARY_PATH) in Tcl before loading the package library will be taken into account by the dyn loader. ... Oh, also note, HP machines use SHLIB_PATH instead IIRC. No idea for Windows. So, in the end, I suspect that it can be done. However do I want the entire portability stuff coming with it ?

I think the issue is the inconsistency, because C is a compiled language, and tcl is interpreted, I am expecting that if the code did not raise a flag at compile time - it found the routines specified in crit::clibraries - then the code should run as well. But at run time, the same library is found in a different way, through the loader, and no longer resolves. So that's confusing.

My workaround is to make the tcl script executable, with

#!/bin/bash
# the next line restarts using wish\
LD_LIBRARY_PATH="/opt/ASI_linux_mac_SDK_V1.21/lib/x64:$LD_LIBRARY_PATH" exec wish "$0" "$@"
...
critcl::cheaders -I/opt/ASI_linux_mac_SDK_V1.21/include
critcl::clibraries -L/opt/ASI_linux_mac_SDK_V1.21/lib/x64
critcl::clibraries -lASICamera2

This solves the problem of running

• with a specific version of a library, e.g. when testing; or
• with a library that is not in the loader's search path, e.g. when the user has no root privileges to install.

Not sure if this particular solution deserves to be in the documentation, as this will not work on other platforms, but some mention of these two scenarios probably should be.

[andreas-kupries]

Ah, I forgot to respond to

I think the issue is the inconsistency, because C is a compiled language, and tcl is interpreted, I am expecting that if the code did not raise a flag at compile time - it found the routines specified in crit::clibraries - then the code should run as well.

earlier. ... This was actually a good argument for me to see it as a bug to be fixed. Less about the C/Tcl difference, and more that while I use critcl pretty much exclusively in package mode (i.e. ahead of time compilation via critcl -pkg ...) it does have a compile and run mode where you simply package require the critcl-based package and it compiles and loads the C code for you on the fly (if you have a working cc, with caching under ~/.critcl/<platform>). Without a fix this mode breaks on non-standard library locations.

Issue #121 created.

[andreas-kupries]

I have examples coming up for handling of external libraries and adjacent things (structures, enums, bitmaps).

[esternin]

Upon reading, bytearray attracted my attention right away, and I was stopped by the comment of it being useless

Ok. Greping through the docs it was only in the standard reference pages. Not in the how to use. That at least is good. The problem with binary data is that you cannot use the presence of \0 byte as terminator. You have to know the length. And the bytearray type simply does not provide that.

I must be dumb, but I am now confused by my apparent inability to control the Tcl_Obj type. This simply does not work for me (note Tcl_NewByteArrayObj)::

critcl::cproc ASIGetDataAfterExp {int cameraId int imgSize} object0 {
  void * imgBuf = ckalloc(imgSize);
  ASIGetDataAfterExp(cameraId, imgBuf, imgSize); 
  Tcl_Obj* ret = Tcl_NewByteArrayObj (imgBuf, imgSize);
  return ret;
  }
...
set imgData [image create photo -width $cameraInfo(width) -height $cameraInfo(height)]
# double the size if each pixel is 2-bytes deep
set imgSize [expr $cameraInfo(width) * $cameraInfo(height) * (1 + [expr {"$ASI_IMG_TYPE($imgType)"=="RAW16"}])]
$imgData put [ASIGetDataAfterExp $cameraId $imgSize]

Apparently the object type "shimmers" on return to tcl into a string, and so gets treated as pixel "colour", so the error is can't parse color... - and the whole thing hangs up, presumably because there is no \0 termination? Things do not improve if I follow this suggestion and use

$imgData put [binary format a* [ASIGetDataAfterExp $cameraId $imgSize]]

and so I have to resort to this to see the image (note Tcl_NewStringObj):

package require Img
package require img::raw
critcl::cproc ASIGetDataAfterExp {int cameraId int imgSize} object0 {
  void * imgBuf = ckalloc(imgSize);
  ASIGetDataAfterExp(cameraId, imgBuf, imgSize); 
  Tcl_Obj* ret = Tcl_NewStringObj (imgBuf, imgSize);
  return ret;
  }
...
set fp [open ".frame.raw" "w"]
puts $fp "Magic=RAW\nWidth=$cameraInfo(width)\nHeight=$cameraInfo(height)\nNumChan=1\nByteOrder=Intel\nScanOrder=TopDown\nPixelType=short"
fconfigure $fp -translation binary -encoding binary
puts -nonewline $fp [ASIGetDataAfterExp $cameraId $imgSize]
close $fp
$imgData read ".frame.raw" -format raw

In the native C implementation (from demo code that came with the library), imgBuf was allocated as

unsigned char* imgBuf = new unsigned char[imgSize];

[andreas-kupries]

$imgData put [ASIGetDataAfterExp $cameraId $imgSize]

Looking at the description of put I believe that it accepts binary data only if it can be recognized by one of the builtin or loaded formats. I am pretty sure that there is no builtin handler for raw unstructured pixel data without even width/height information.

Apparently the object type "shimmers" on return to tcl into a string, and so gets treated as pixel "colour"

That is what $imgData put does when it does not recognize the format of the input. See the manpage.

And in your second example you actually do explicitly load a handler for a raw format of some kind

package require img::raw

and then use that format with read, via -format raw.

Please try the same with the put example.

I would recommend to rewrite ASIGetDataAfterExp to not take the image size, but width, height, and the raw16 flag, and then return the entirety of the image, i.e. the magic prefix followed by the pixels. (Note that puts inserted a \n between prefix and pixels).

[esternin]

The magic prefix, as you call it, is the optional 7-line file header for the raw format. I was adding it just for compatibility, if something else wants to read the file and does not know its width and height (which I do, when I create the imgData ! ).

But it looks like put insists on there being such a header, not optional at all. I did try that before, but did not specify the -format raw, since I thought it was established as "known" by img::raw, and surely something without a header would be attempted as raw. Dumb, I know...

Thanks a bunch!!

This worked:

set tmpData [ASIGetDataAfterExp $cameraId $imgSize]
$imgData put "Magic=RAW\nWidth=$cameraInfo(width)\nHeight=$cameraInfo(height)\nNumChan=1\nByteOrder=Intel\nScanOrder=TopDown\nPixelType=short\n$tmpData" -format raw

and it worked identically whether I used Tcl_NewStringObj or Tcl_NewByteArrayObj which is another confusing thing. I guess since upon return to tcl we may get converted to string anyway, ByteArray does appear kind of redundant...

I decided to keep the tcl wrappers of hardware-specific calls to an absolute minimum, because eventually there might be other hardware in play, requiring different calls, so I want to handle as much as possible in tcl, to imitate the native C code sequence.