X-Sharp
Scripting - ExecScript
So far I detected 7 problems running a script via execscript:
1. parameters - only 3 parameters are recognized parameters a,b,c,d,e,f,g,h a,b,c //=> ok d,e,f,g,h //=> NIL return 0
2. IF CLAUSE - elseif does not work:
local x
x := DoSomething()
if x > 100
x := 100
else
x := 10
endif
=>ok
local x
x := DoSomething()
if x > 100
x := 100
elseif x > 50
x := 50
else
x := 10
endif
=> Macrocompiler: error XM0100: Expected 'END IF'
3. DO CASE - do case statement without otherwise does not work:
local x
x := DoSomething()
do case
case x > 100
x := 100
otherwise
x := 10
endcase
=> ok
local x
x := DoSomething()
do case
case x > 100
x := 100
endcase
=> Der Objektverweis wurde nicht auf eine Objectinstanz festgelegt
4. SWITCH - switch does not work at all => Macrocompiler: error XM0100: Expected 'EOS' BEGIN SWITCH is working...
5. FOR NEXT - not a bug, only a flaw if you type "next n":
for n := 1 upto 10
DoSomething()
next n
=> Macrocompiler: error XM0100: Expected 'EOS'
6. Default-function
parameters a
default(@a,5)
=> Macrocompiler: error XM0100: Expected 'type'
7. PROCNAME() - not a bug, only a flaw procname(1) gives the whole Script in capitals Maybe "[SCRIPT]" or something else might make more sense. Perhaps even a scriptname (in the first line of the script, e. g. "method testscript")
One more Issue:
locals can't be used in codeblocks, e. g. ascan:
local n as dword
local a as array
local nPos as dword
a := {1,2,3,4,5}
n := 3
nPos := ascan(a,{|x|x=n})
returns (in german): Variable existiert nicht
Karl
Karl, All of the issues you report are supported by the X# scripting engine. Maybe we should add an easier layer on top of this engine in the X# runtime?
Robert,
sorry I don't quite understand this. If i put my testcode into a string and call execscript I get the mentioned results:
LOCAL cScript AS STRING
text into cScript WRAP
LOCAL n AS DWORD
LOCAL a AS ARRAY
LOCAL nPos AS DWORD
a := {1,2,3,4,5}
n := 3
nPos := AScan(a,{|x|x=n})
RETURN 0
endtext
TRY
execscript(cScript)
CATCH e AS exception
TextBox{,"",e:Message}:Show()
END TRY
This pops up a textbox with "Variable existiert nicht". This refers to the line with ascan, n is not visible in the codeblock. If n were PRIVATE the script would run without problems...
LOCAL cScript AS STRING
text into cScript WRAP
LOCAL n AS DWORD
LOCAL i AS DWORD
n := 3
DO CASE
CASE n = 1
i := 2
CASE n= 3
i := 4
ENDCASE
RETURN 0
endtext
TRY
execscript(cScript)
CATCH e AS exception
TextBox{,"",e:Message}:Show()
END TRY
This pops up a textbox with "Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt." This does not happen if I insert OTHERWISE before ENDCASE.
Same with my other examples...
Karl
karl, I understand what you are saying. Execscript is coded in the macro compiler. Apparently we have not completely implemented the same functionality that the compiler has. If you whould have used the "Script compiler" (XSharp.Script.DLL) then the syntax to call this would have been a bit more complicated, but then the full compiler (inside XSharp.CodeAnalysis) is used to compile the script.
Robert,
to circumvent the problem with the 3 parameters I implemented my own function to execute scripts, code is of course mainly stolen from yours :)
INTERNAL GLOBAL XSCompiledScripts := XSScriptCache{200} AS XSScriptCache
FUNCTION XSExecScript(cScript AS STRING,uPars PARAMS USUAL[]) AS USUAL
LOCAL uResult AS USUAL
LOCAL cb AS CODEBLOCK
IF XSCompileScript(cScript,REF cb)
TRY
uResult := cb:Eval(uPars)
CATCH e AS exception
Message("",e:Message,0)
END TRY
ENDIF
RETURN uResult
FUNCTION XSCompileScript(cScript AS STRING,cb REF CODEBLOCK) AS LOGIC
LOCAL lSuccess AS LOGIC
IF XSCompiledScripts:ContainsKey(cScript)
cb := XSCompiledScripts[cScript]
lSuccess := TRUE
ELSE
ChkScriptCompiler()
IF RuntimeState.ScriptCompiler IS IMacroCompilerUsual VAR compiler
TRY
LOCAL oSB AS Stringbuilder
// 1. Zeile deaktivieren (Method(...)) und Parameter als locals übernehmen
oSB := Stringbuilder{slen(cScript)+100}
VAR nStart := at(CRLF,cScript)
IF nStart = 0
THROW exception{"kein gültiges Script"}
ENDIF
VAR cHdr := alltrim(left(cScript,nStart-1))
IF right(cHdr,1)==")"
VAR nPos := at("(",cHdr)
IF nPos > 0
cHdr := substr2(cHdr,nPos+1)
cHdr := left(cHdr,slen(cHdr)-1)
IF !empty(cHdr)
oSB:Append("LPARAMETERS "+cHdr)
ENDIF
ENDIF
ENDIF
oSB:Append(substr2(cScript,nStart))
cb := compiler:CompileCodeblock(oSB:ToString())
XSCompiledScripts:TryAdd(cScript,cb)
lSuccess := TRUE
CATCH e AS exception
Message("Compiler-Fehler",e:message,0)
END TRY
ELSE
Message("Compiler-Fehler","Der Script-Compiler wurde nicht gefunden",0)
ENDIF
ENDIF
RETURN lSuccess
STATIC FUNCTION ChkScriptCompiler() AS VOID
IF RuntimeState.ScriptCompiler == NULL_OBJECT
VAR oMacroAsm := AssemblyHelper.Load("XSharp.MacroCompiler")
IF oMacroAsm <> NULL_OBJECT
VAR oType := oMacroAsm:GetType("XSharp.Runtime.MacroCompiler")
IF oType <> NULL
VAR oMI := oType:GetMethod("GetScriptCompiler")
VAR oArg := <OBJECT>{RuntimeState.Dialect}
IF oMI <> NULL
VAR oMC := oMI:Invoke(NULL,oArg)
RuntimeState.ScriptCompiler := (IMacroCompiler)oMC
ENDIF
ENDIF
ENDIF
ENDIF
RETURN
INTERNAL CLASS XSScriptCache INHERIT System.Collections.Concurrent.ConcurrentDictionary<STRING,CODEBLOCK>
PRIVATE _keys AS System.Collections.Concurrent.ConcurrentQueue<STRING>
PRIVATE _maxItems AS INT
PUBLIC CONSTRUCTOR(maxItems AS INT)
_keys := System.Collections.Concurrent.ConcurrentQueue<STRING>{}
_maxItems := maxItems
RETURN
NEW PUBLIC METHOD TryAdd(key AS STRING,oCB AS CODEBLOCK) AS LOGIC
VAR lOk := SUPER:TryAdd(key, oCB)
IF lOk
_keys:Enqueue(key)
ENDIF
WHILE(_keys:Count > _maxItems)
LOCAL oldKey := NULL AS STRING
IF _keys:TryDequeue(OUT oldKey)
SUPER:TryRemove(oldKey, OUT VAR _)
ENDIF
END
RETURN lOk
END CLASS
message() is a function to pop up a textbox...
Karl
Robert,
do you have a short example how to implement the script compiler properly ? I'll make some tests tomorrow...
Karl
Robert,
here is my first attempt. I'm struggling with parameters and vo15 (treat missing types as usual)
-
options:SetOption(CompilerOption.Vo15,true) does not work, I always get "type expected" in the given script (local n, local n as dword works)
-
how can i declare the parameters in the script ? Do I have to write my own "preprocessor" which translates
function x(a,b,c)tovar a:=_SP_[1];var b:=_SP_[2];var c:=_SP_[3]
CLASS ScriptParams
PUBLIC _SP_ AS USUAL[]
PUBLIC _PrivatesLevel_ AS INT
END CLASS
FUNCTION TestScript() AS VOID
LOCAL cScript AS STRING
text into cScript WRAP
LOCAL n
LOCAL a AS ARRAY
LOCAL nPos AS DWORD
FOREACH p AS USUAL IN _SP_
Console.writeline(p)
NEXT
a := {1,2,4,3,5}
n := 3
nPos := AScan(a,{|x|x=n})
Console.Writeline(nPos:tostring())
DO CASE
CASE n = 3
Console.Writeline("Ok")
ENDCASE
RETURN n
endtext
StartScript(cScript,1,2,3,4,5)
Console.ReadKey()
RETURN
FUNCTION StartScript(cScript AS STRING,args PARAMS USUAL[]) AS USUAL
LOCAL uResult AS USUAL
TRY
LOCAL options AS XSharpSpecificCompilationOptions
options := XSharpMacro.GetOptions((INT) RuntimeState.Dialect)
options:SetOption(CompilerOption.Vo11, RuntimeState.CompilerOptionVO11)
options:SetOption(CompilerOption.Vo13, RuntimeState.CompilerOptionVO13)
options:SetOption(CompilerOption.Overflow, RuntimeState.CompilerOptionOVF)
// options:SetOption(CompilerOption.Vo15,true)
VAR references := System.AppDomain.CurrentDomain:GetAssemblies()
references:Where({a => !String.IsNullOrEmpty(a:Location)})
LOCAL scoptions AS ScriptOptions
scoptions := ScriptOptions.Default
scoptions:WithXSharpSpecificOptions(options)
scoptions:WithReferences(references)
scoptions:WithImports(<STRING>{"System","System.Text","System.Collections.Generic","System.Linq"})
VAR script := XSharpScript.Create(cScript,scoptions,TYPEOF(ScriptParams),NULL)
VAR errors := script:Compile()
FOREACH VAR e IN errors
Console.WriteLine("Compilerfehler: "+e:GetMessage())
NEXT
VAR ScriptArgs := ScriptParams{}
ScriptArgs:_SP_ := args
ScriptArgs:_PrivatesLevel_ := XSharp.RT.Functions.__MemVarInit()
TRY
uResult := script:RunAsync(ScriptArgs):Result:ReturnValue
CATCH e AS exception
THROW e
FINALLY
XSharp.RT.Functions.__MemVarRelease(ScriptArgs:_PrivatesLevel_)
END
CATCH e AS exception
Console.Writeline("Catch: "+e:message)
END TRY
RETURN uResult
Karl
Karl, I see that indeed setting vo15 does not work as expected. The work around is to include a pragma in the source. Unfortunately you cannot use the #pragma syntax inside a text .. endtext.
Try this:
text into cScript WRAP
~~PRAGMA~~
LOCAL n
LOCAL a AS ARRAY
...
endtext
cScript := cScript:Replace("~~PRAGMA~~","#pragma options(""vo15"",true)")
Passing in parameters is something else. The easiest way is to pass in a globals object like you're already doing. I'll discuss with Nikos how we can make this easier.
Maybe we should support the PARAMETERS and LPARAMETERS keywords in scripts to declare either private or local parameters?
I also see that declaring and using PRIVATE variables is a bit of a problem.
In the normal compiler we store the list of memvars, privates etc in the structure of the current method / function etc.
The scriptcompiler does not have that structure, since the script is a simple list of statements.
You can declare memvars with the PRIVATE keyword, but to access them the systax with the m-> prefix is needed:
PRIVATE M1, m2, m3
m->m1 := Date()
m->m2 := Time()
m->m3 := 42
? m->m1, m->m2, m->m3
And to be able to do this you need to add the pragma to allow memvars:
cScript := cScript:Replace("~~PRAGMA~~","#pragma options(""vo15"",true)"+CRLF+"#pragma options(""memvars"",true)")
Robert,
I'm not sure if that is constructive. My Scripts are running quite well with ExecScript, except my notes above. PARAMETERS and LPARAMETERS are also working with ExecScript already. For me it would suffice to implement the missing features. And ExecScript() is a documented feature of x#, it should work as expected...
Just to let you know how we are using scripting: We store scripts in a table, in MEMO-fields. The scripting-module is open to our customers. Every customer can write his own scripts to adapt the behaviour of our application or to make exports as csv, pdf, xml or whatever. The calling of some scripts is defined in the app, there we have predefined parameters. Other scripts can be defined freely with whatever parameter the customer needs, e. g. to return an expression to display data in a browser-cell or in a field on a window. Scripts can be called through a function (Runscript()). Runscript loads the script from the script-table and executes it: Runscript("testscript",kunden->kd_nr) We load the script "testscript", pass the value of the field KD_NR in the table KUNDEN as a parameter and execute the script. The script might return the name in a special formatted way which is then displayed.
pragma seems to work as you suggested, but I found many other problems, e. g.
`scoptions:WithReferences(references)`
gives "Der aufgerufene Member wird in einer dynamischen Assembly nicht unterstützt"
Calling functions or classes of our application does not seem to work either. Adding new lines like "pragma" also leads to wrong line-numbers in messages of the compiler.
So I'm quite lost if we switch to this "new" scripting.
However of course if you want to improve "new" scripting I'm ready to test everything with a little bit of your help.
Karl
The current ExecScript was added to support FoxPro scripting. For that several things that you have already listed, this is not working like in the full compiler. Supporting everything that the full compiler has, will be tricky, since we will have to maintain 2 sets of code. The advantage of the current ExecScript is that it creates dynamic methods, that can be unloaded when they are no longer used. The compiled scripts from the script compiler stay in memory in an in memory (dynamic) assembly. Some of the things you listed should not be too difficult to add / fix:
- parameters
- elseif
- otherwise
- comment on next line
- default function
Others require more work
- switch
- using (detached) locals inside codeblocks
Robert
The solution for the missing parameter is to declare your own ExecScript():
FUNCTION ExecScript( cExpression, eParameter1, eParameter2, eParameterN ) AS USUAL CLIPPER
RETURN ExecScriptFast(_Args() )
The solution for the missing parameter is to declare your own ExecScript():
FUNCTION ExecScript( cExpression, eParameter1, eParameter2, eParameterN ) AS USUAL CLIPPER RETURN ExecScriptFast(_Args() )
No problem, I did this already...
The current ExecScript was added to support FoxPro scripting. For that several things that you have already listed, this is not working like in the full compiler. Supporting everything that the full compiler has, will be tricky, since we will have to maintain 2 sets of code. The advantage of the current ExecScript is that it creates dynamic methods, that can be unloaded when they are no longer used. The compiled scripts from the script compiler stay in memory in an in memory (dynamic) assembly. Some of the things you listed should not be too difficult to add / fix:
* parameters * elseif * otherwise * comment on next line * default functionOthers require more work
* switch * using (detached) locals inside codeblocksRobert
Good to hear.
Switch should not be too difficult either, I found out that
BEGIN SWITCH
is working already: But it is also not so important as SWITCH wasn't present in VOSCRIPT, so no existing script uses it.
The codeblock-issue is however rather important as this is used in many existing (VO)Scripts of our customers.
I will also go on testing the script compiler within our application...
Karl
Robert,
I have the following easy script. In a standalone application it runs without problems.
#pragma options("vo15",true)
#pragma options("memvars",true)
local a as int
a := 1
return a
In my big application I get a compile error:
Do you know what is going on ?
Karl
I think you're missing a reference to the DLL that has the Immutable support (System.Collections.Immutable.dll). Btw I will run some tests here today with the ExecScript support in the macro compiler. I think I have fixed most or all of the issues that you reported. When that is successfull, then I'll upload a DLL here so you can test it.
The only thing that I am not sure of is the use of locals in codeblocks.
I do have System.Collections.Immutable.dll in my references. However it is Version 5.0.0.0. In my Test-application it is 8.0.0.0. If I Include 8.0.0.0 in the big app I get
warning XS1701: Assuming assembly reference 'System.Collections.Immutable, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' used by 'XSharp.Scripting' matches identity 'System.Collections.Immutable, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' of 'System.Collections.Immutable', you may need to supply runtime policy
and the app crashes right from the start
I think you're missing a reference to the DLL that has the Immutable support (System.Collections.Immutable.dll). Btw I will run some tests here today with the ExecScript support in the macro compiler. I think I have fixed most or all of the issues that you reported. When that is successfull, then I'll upload a DLL here so you can test it.
The only thing that I am not sure of is the use of locals in codeblocks.
I'll probably make the tests tomorrow then, the rest of today I will have to earn some money :)
BTW, If i reference System.Collections.Immutable.dll (from the x#-Redist folder) in the small app I also get the error "Typenititialisierer". So I have to find out, why 8.0.0.0 does not work in my big app and makes it crash...
Now it works also in the main app, I had a mismatch of the dll's in my exe (v5.0.0.0) and my scripting dll (v 8.0.0.0) The compiler-warning XS1701 however is still present...
What I also had to do to make it work is to add one line (IsDynamic) before XSharpScript.Create(....)
VAR references := System.AppDomain.CurrentDomain:GetAssemblies() ;
:Where({a => !a:IsDynamic}) ;
:Where({a => !String.IsNullOrEmpty(a:Location)})
Nikos, Can you look at the test case where a local is used inside a codeblock. I have added a test for this in the scripttests.prg It fails because inside the codeblock the local 'n' from the surrounding function is invisible. It works when 'n' is a private. When I add calls to __LocalPut() before evaluating the nested codeblock and __LocalsClear() after it, then it works. Maybe you can automatically do something like that in the code for the NestedDelegate(), when you detect that the codeblock uses a symbol that is not in its parameterlist?
Now it works also in the main app, I had a mismatch of the dll's in my exe (v5.0.0.0) and my scripting dll (v 8.0.0.0) The compiler-warning XS1701 however is still present...
The compiler warning is caused by an error in our build system. If you goto the folder "c:\Program Files (x86)\XSharp\MsBuild" and copy (and overwrite) the attached files, then the warning is gone.
Great, then we will have 2 options for scripting. For our customer-scripting I will certainly use ExecScript which replaces VOScript then. VOScript is about 10 times slower in x# than it was in VO. ExecScript gives us back the old speed. Maybe xSharpScript will offer us additional possibilities to make our app more flexible...
I'm looking foreward to the test-dll (although GAC will certainly make things difficult again...)
This is an updated macro compiler. Whats's not fixed in this compiler (since these are part of XSharp.RT) is no 1 and no 7 from your list. You also cannot use locals in a codeblock in a script. Privates work.
Please let me know if this works for you.
Robert,
your changes are all working for me. Because of a typo I noticed that in DO CASE as well as in SWITCH the compiler does not complain if you type OTHERWSIE (or anything else) instead of OTHERWISE. The branch is then simply ignored when executed. Not nice but also not important.
If Nikos can still implement the exported locals in codeblocks in the final version that would be perfect.
Karl
Procname() works too, I copied your changes from github and implemented my own procname(). Returns "SCRIPT" as expected.
Robert, Nikos,
any chance to get the __LocalPut() before evaluating the nested codeblock and __LocalsClear() after it in the final x# 2-Version ? Just tested 2.23 and it still does not work...
Karl