gdbsource Errors

[Njoselson]

trafficstars

Hello, I wrote this as a comment in a closed issue without knowing whether the developers could see it, so I thought I would create a new issue as well.

I am trying to debug an out of bounds error that keeps making R crash. The problem is, that when I use gdbsource("file") it only gives me an error: "Error in line:length(x) : argument of length 0 In addition: Warning message: running command 'R --vanilla < C:/Users/Nathaniel/Documents/MoultProb.R -d gdb --debugger-args="-x C:\Users\NATHAN~1\AppData\Local\Temp\RtmpE7k6FC\file163060ca5272 "' had status 255"

I have tried to run each line of the debugger separately, changed my User Path to include R and Rtools and Rstudio, and have tried to follow all the directions in the gdbsource document (having files of the same name; adding "-O0 -g" in the compilation step).

I have looked around this website but can't seem to find further documentation on debugging the debugger.

Thanks, Nathaniel

[kaskr]

Currently gdbsource does not work on the windows operating system. I will see if I can fix this. Otherwise it should be documented under ?gdbsource.

[Njoselson]

Thanks, No wonder. Will see what I can do without it.

Nathaniel

[Njoselson]

Do you know the equivalent system command in windows for R --vanilla < filepath.R -d gdb --debugger-args="-x tempfile "?

[kaskr]

gdbsource now also works on windows. One has to compile with compile(cppfile,"-O1 -g",DLLFLAGS="") I was unable to get the compilation working with lower optimization level on windows (required in order to print variables during a debug session).

[Njoselson]

This fix still hasn't worked for me. On the tmb example files, the same error still comes: "Error in line:length(x) : argument of length 0".

I am compiling according to your instructions, but for some reason, the function still doesn't work. Does it sound like there could be a problem with my gdb? I have the minGW gdb, Rstudio and a 64 bit windows.

[kaskr]

I forgot to say this is a fix on the master branch so you have to update your TMB installation:

remove.packages(TMB)
source("install_windows.R")

did you do that?

Also, make sure the *.cpp file name match the *.R file name.

If you cannot make it work in non-interactive mode

gdbsource("mymodel.R",FALSE) ## Non-interactive

what happens if you run it in interactive mode?

gdbsource("mymodel.R",TRUE)  ## Interactive

[Njoselson]

Yes, I did reinstall the package and yes the files are with the same name. When I compile it with interactive=FALSE, I get the length(x)=0 error, and when I use TRUE, it opens a command prompt where the gdb program opens, but then doesn't run with the error:

"Rterm: No such file or directory. No symbol table is loaded. Use the "file" command. Breakpoint 1 (abort) pending. Error in sources command file. No executable specified, use 'target exec'."

And then it goes to a blank gdb line (gdb) _

On Dec 29, 2014, at 4:18 PM, kaskr [email protected] wrote:

I forgot to say this is a fix on the master branch so you have to update your TMB installation:

remove.packages(TMB) source("install_windows.R") did you do that?

Also, make sure the *.cpp file name match the *.R file name.

If you cannot make it work in non-interactive mode

gdbsource("mymodel.R",FALSE) ## Non-interactive what happens if you run it in interactive mode?

gdbsource("mymodel.R",TRUE) ## Interactive — Reply to this email directly or view it on GitHub.

[kaskr]

Why is "Rterm" not found? Try this:

library(TMB)
dir(fileLocations()$rbin)  ## Should contain 'Rterm.exe'
shell("PATH")              ## Should contain fileLocations()$rbin
shell("start Rterm")       ## Should start Rterm in new window

[Njoselson]

Yes, this does start Rterm in a new window but gdbsource still doesn't work. Now it gives the message: "C:\Program Files\R\R-3.1.2\bin\x64/Rterm.exe": not in executable format: File format not recognized. No symbol table is loaded. Use the "file" command. Breakpoint 1 (abort) pending. Error in sources command file. No executable specified, use 'target exec'.

[Njoselson]

And running the gdbsource in the Rterm window gives the same error. Am I misunderstanding what you mean?

[kaskr]

This:

"C:\Program Files\R\R-3.1.2\bin\x64/Rterm.exe": not in executable format

is probably because of the space in the path. Do you have a space in

Sys.getenv("R_HOME")

If yes, does this get rid of the space:

shortPathName(Sys.getenv("R_HOME"))

If yes, could you try to modify 'install_windows.R' with an extra line:

fileLocations <- function(){
  rhome <- Sys.getenv("R_HOME")
  rbin <- paste0(rhome,"/bin/",.Platform$r_arch)
  rtools <- c("C:/Rtools",
              "~/Rtools",
              paste0(rhome,"/../../Rtools"),
              paste0(rhome,"/../Rtools") )
  gccbin <- paste0(rtools,"/gcc-4.6.3/bin")
  gdbbin <- paste0(rtools,"/gcc-4.6.3/bin64")
  ans <- list(rhome=rhome,rbin=rbin,rtools=rtools,gccbin=gccbin,gdbbin=gdbbin)
  ans <- lapply(ans,shortPathName)  ## <----- ADD THIS LINE
  lapply(ans,function(x)x[file.exists(x)][1])
}

Then reinstall TMB.

[Njoselson]

I don't have a space when I run Sys.getenv("R_HOME") The path there is: C:/PROGRA~1/R/R-31~1.2

I have also modified the system path for the bins in R as well from 'Program Files' to 'PROGRA~1', this didn't work either.

Should I still try to modify the 'install_windows.R'

Sorry to be so confused.

[kaskr]

No, my previous suggestion is not useful then. And I just discovered the R-documentation ?R.home says: " On Windows the values of ‘R.home()’ and ‘R_HOME’ are guaranteed not to contain spaces, switching to the 8.3 short form of path elements if required. " Could you instead post the output of

library(TMB)
shell("PATH")

I am still puzzled about the cause of the path spaces...

[Njoselson]

> library(TMB) > shell("PATH") PATH=C:\Program Files\R\R-3.1.2\bin\x64;c:\Rtools\bin;c:\Rtools\gcc-4.6.3\bin;c:\Rtools\gcc-4.6.3\bin32;C:\ProgramData\Oracle\Java\javapath;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0;C:\Program Files\Intel\WiFi\bin;C:\Program Files\Common Files\Intel\WirelessCommon;c:\Program Files (x86)\Common Files\Roxio Shared\OEM\DLLShared;c:\Program Files (x86)\Common Files\Roxio Shared\OEM\DLLShared;c:\Program Files (x86)\Common Files\Roxio Shared\OEM\12.0\DLLShared;c:\Program Files (x86)\Roxio\OEM\AudioCore;C:\Program Files\doxygen\bin;C:\PROGRA~1\R\R-3.1.2\bin;C:\PROGRA~1\R\R-3.1.2\bin\x64;C:\Program Files (x86)\Sophos\Sophos SSL VPN Client\bin;c:\Rtools\bin;c:\Rtools\gcc-4.6.3\bin;c:\Rtools\gcc-4.6.3\bin32;C:\ProgramData\Oracle\Java\javapath;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0;C:\PROGRA~1\Intel\WiFi\bin;C:\PROGRA~1\Common Files\Intel\WirelessCommon;c:\PROGRA~2\Common Files\Roxio Shared\OEM\DLLShared;c:\PROGRA~2\Common Files\Roxio Shared\OEM\DLLShared;c:\Program Files (x86)\Common Files\Roxio Shared\OEM\12.0\DLLShared;c:\Program Files (x86)\Roxio\OEM\AudioCore;C:\Program Files\doxygen\bin;C:\PROGRA~1\R\R-3.1.2\bin;C:\PROGRA~1\R\R-3.1.2\bin\x64;

[kaskr]

Well, that explains where the space comes from:

C:\Program Files\R\R-3.1.2\bin\x64;c:\Rtools\bin;c:\Rtools\gcc-4.6.3\bin;c:\Rtools\gcc-4.6.3\bin32

It seems to me this path is not auto-generated by TMB? (because of the order). When TMB is loaded on windows the function setPath() is called. Maybe that part failed? Could you post the output of

library(TMB)
setPath()
shell("PATH")

(or just say if anything changed)

[Njoselson]

PATH=C:/Rtools/bin;C:/Rtools/gcc-4.6.3/bin;C:/PROGRA~1/R/R-31~1.2/bin;C:\Program Files\R\R-3.1.2\bin\x64;c:\Rtools\bin;c:\Rtools\gcc-4.6.3\bin;c:\Rtools\gcc-4.6.3\bin32;C:\ProgramData\Oracle\Java\javapath;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0;C:\Program Files\Intel\WiFi\bin;C:\Program Files\Common Files\Intel\WirelessCommon;c:\Program Files (x86)\Common Files\Roxio Shared\OEM\DLLShared;c:\Program Files (x86)\Common Files\Roxio Shared\OEM\DLLShared;c:\Program Files (x86)\Common Files\Roxio Shared\OEM\12.0\DLLShared;c:\Program Files (x86)\Roxio\OEM\AudioCore;C:\Program Files\doxygen\bin;C:\PROGRA~1\R\R-3.1.2\bin;C:\PROGRA~1\R\R-3.1.2\bin\x64;C:\Program Files (x86)\Sophos\Sophos SSL VPN Client\bin;c:\Rtools\bin;c:\Rtools\gcc-4.6.3\bin;c:\Rtools\gcc-4.6.3\bin32;C:\ProgramData\Oracle\Java\javapath;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0;C:\PROGRA~1\Intel\WiFi\bin;C:\PROGRA~1\Common Files\Intel\WirelessCommon;c:\PROGRA~2\Common Files\Roxio Shared\OEM\DLLShared;c:\PROGRA~2\Common Files\Roxio Shared\OEM\DLLShared;c:\Program Files (x86)\Common Files\Roxio Shared\OEM\12.0\DLLShared;c:\Program Files (x86)\Roxio\OEM\AudioCore;C:\Program Files\doxygen\bin;C:\PROGRA~1\R\R-3.1.2\bin;C:\PROGRA~1\R\R-3.1.2\bin\x64;

Basically, only the first couple of lines have changed, and I still get the same error from gdb when I run gdbsource( ... , interactive=TRUE)

[kaskr]

setPath() correctly added

C:/Rtools/bin;C:/Rtools/gcc-4.6.3/bin;C:/PROGRA~1/R/R-31~1.2/bin;

but it did not add the 64bit gdb. Instead the 32 bit gdb from your (old?) Rtools installation is used to debug 64 bit Rterm.exe, which explains the error. Does

fileLocations()

have NA in the component gdbbin? If yes, it means that your current Rtools installation does not have the 64 bit gdb. Then the solution should be to remove your old Rtools installation and run the install_windows.R script again (this will install the correct version of Rtools).

[Njoselson]

The output is as follows:

fileLocations() $rhome [1] "C:/PROGRA~1/R/R-31~1.2"

$rbin [1] "C:/PROGRA~1/R/R-31~1.2/bin"

$rtools [1] "C:/Rtools"

$gccbin [1] "C:/Rtools/gcc-4.6.3/bin"

There is no component gdbbin.

[kaskr]

If your current source code of the install_windows.R script does not contain 'gdbbin', then the adcomp folder have not been correctly updated, and you need to download again and reinstall.

[Njoselson]

Perhaps I have been installing it wrong, because I downloaded it again and it still hasn't worked.

I first uninstall the package TMB, then run the following: devtools::install_github("kaskr/adcomp", subdir = "TMB")

[kaskr]

This approach only updates the subfolder 'TMB'. Please follow the install instructions stated in the README.

[Njoselson]

Using source("install_windows.R") with the newly cloned adcomp master folder, there is no change in the functionality of gdbsource. It still opens, gives the R.exe not an executable file message, and then opens a blank gdb line which doesn't take R documents as an input for the file command.

The only effect has been that the fileLocations() function now returns: $gdbbin [1] "C:/Rtools/gcc-4.6.3/bin64"

[kaskr]

Check that

shell("where gdb")
shell("where Rterm")

gives the matching (both 64 bit) versions. If not, fix the PATH.

Otherwise, remove your Rtools folder and repeat install_windows.R step (to make sure you get the right Rtools).

[Njoselson]

The problem is solved!

I uninstalled R, and then reinstalled it, making the location C:\PROGRA~1... Then, I reinstalled Rtools and manually added the bins C:\Rtools\gcc-4.6.3\i686-w64-mingw32\bin and C:\Rtools\gcc-4.6.3\bin64 to the path, during the install. Then I reinstalled TMB from source("install_windows.R") and the gdbsource() works perfectly.

Thanks so much for your patience through this! Great program!

[James-Thorson]

Hi all,

During a TMB training workshop, we built a function that can replace MakeADFun on the windows operating system, which calls the debugger and if it passes just compiles the original model, and if it fails gives the output from the debugger:

MakeADFun_windows_debug = function( cpp_name, data, parameters, random=NULL, dir=getwd(), overwrite=FALSE, recompile=TRUE, ... ){

  # Set local working directory
  orig_dir = getwd()
  setwd( dir )
  on.exit( setwd(orig_dir) )

  # Recompile
  if( recompile==TRUE ){
    unlink( dynlib(cpp_name) )
    compile( paste0(cpp_name,".cpp"), "-O1 -g", DLLFLAGS="")
    dyn.load( dynlib(cpp_name) )
  }

  # Get and save inputs
  Other_inputs = list(...)
  All_inputs = list( "data"=data, "parameters"=parameters, "random"=random, "Other_inputs"=Other_inputs )
  save( All_inputs, file="All_inputs.RData")

  # Write file to source
  if( (paste0(cpp_name,".R") %in% list.files()) & overwrite==FALSE ){
    message( "By default, we don't overwrite existing file ",cpp_name,".R")
    Return = "Didn't attempt running"
  }else{
    sink( paste0(cpp_name,".R") )
    cat("
      library( TMB )
      dyn.load( dynlib('",cpp_name,"') )
      setwd('",dir,"')
      load( 'All_inputs.RData' )
      Obj = MakeADFun(data=All_inputs[['data']], parameters=All_inputs[['parameters']], random=All_inputs[['random']], All_inputs[['Other_inputs']])
      save( Obj, file='Obj.RData')
    ",fill=FALSE,sep="")
    sink()

    # Try running
    Bdg_output = gdbsource(paste0(cpp_name,".R"))

    # Sort out outcomes
    if( length(grep("#0",Bdg_output))==0 ){
      Return = MakeADFun(data=All_inputs[['data']], parameters=All_inputs[['parameters']], random=All_inputs[['random']], All_inputs[['Other_inputs']])
      message("Compiled fine, and returning output from MakeADFun")
    }else{
      Return = Bdg_output
      message("Did not compile, and returning output from bdbsource")
      #print( Bdg_output )
    }
  }

  # Return
  return( Return )
}

We imagine that this function is probably not suitable for the TMB library (because its a bit gimmicky, and because it probably wouldn't pass CRAN standards). But perhaps we could put it in a new user-contributed library of helper functions, say, TMBtools, which could be downloaded from GitHub.

Kasper and others -- any thoughts or preferences for how to proceed?

cheers, jim

[kaskr]

This idea could easily be transformed into a useful package (purpose: make debugging easier and guarantee that the main R session never crashes).

Some comments to the code:

• Widen the scope to not being Windows specific.
• Don't save/load Obj. There is no time saving because the external pointers are recomputed in the main session anyway. Debugging must be robust and should not rely on saving/loading ADFun objects.
• Rather than renaming MakeADFun and calling compile I suggest you simply overload the two functions. Something like:

compile <- function(...){
    args <- list(...)
    if (.Platform$OS.type == "windows") {
        args$flags <- "-O1 -g"
        args$DLLFLAGS <- ""
    } else {
        args$flags <- "-O0 -g"
    }
    do.call(TMB::compile, args)
}
MakeADFun <- function(..., DLL = TMB:::getUserDLL()){
    ## Set local working directory
    orig_dir <- getwd()
    setwd( tempdir() )
    on.exit( setwd(orig_dir) )
    ## Save inputs
    All_inputs <- list(..., DLL=DLL)
    save( All_inputs, file="All_inputs.RData")
    ## Copy DLL to tempdir
    DLL <- All_inputs$DLL
    DLLfull <- paste0(orig_dir,"/",DLL)
    ## Write file to source
    txt <- c("library( TMB )",
             paste0("dyn.load(dynlib('",DLLfull,"'))"),
             "load( 'All_inputs.RData' )",
             "Obj <- do.call(TMB::MakeADFun, All_inputs)"
             )
    writeLines(txt, paste0(DLL,".R"))
    ## Try running
    Bdg_output <- gdbsource(paste0(DLL, ".R"))
    # Sort out outcomes
    if( length(grep("#0",Bdg_output)) > 0 ){
        message("Model has errors")
        print(Bdg_output)
        stop()
    }
    ## OK ==> Safe to run in main session:
    TMB::MakeADFun(...,DLL=DLL)
}

• You could name the package 'TMB-debug'. Now debugging becomes as simple as changing "library(TMB)" to "library(TMB-debug)".

[ghost]

Hello, I'm having about exactly the same problem as the topic starter (although more than a year later....) when running this in R:

compile( "model.cpp", "-O1 -g", DLLFLAGS="")
gdbsource( "model.R" )
"Error in line:length(x) : argument of length 0

So I read the whole conversation, did what kaskr suggested (obtaining basically the same results as njoselson), changed the path and completely uninstalled and reinstalled R, Rtools and TMB several times now, with different options and versions (TMB was installed used either adcomp or Rstudio install packages). I also deinstalled everything that might potentially interfere (ADMB, old Rtools packages). For the moment, I'm running 3.1.3 with Rtools32 to be sure to have a froozen Rtools, and to be closer to the version of the topic starter (but I also tried R3.2.3 with Rtools33). Could anyone please indicate what I'm doing wrong or what I might still try? I launched myself into learning TMB only recently, and it seems highly promissing. But without a debugger I stumble to get my own code running. Here are the outputs of what was previsously already asked:

> fileLocations()
$rhome
[1] "C:\\PROGRA~1\\R\\R-31~1.3"

$rbin
[1] "C:\\PROGRA~1\\R\\R-31~1.3\\bin\\x64"

$rtools
[1] "C:\\Rtools"

$gccbin
[1] "C:\\Rtools\\GCC-46~1.3\\bin"

$gdbbin
[1] "C:\\Rtools\\GCC-46~1.3\\bin64"

shell("PATH") PATH=C:\Program Files\R\R-3.1.3\bin\x64;c:\Rtools\bin;c:\Rtools\gcc-4.6.3\bin;C:\Rtools\gcc-4.6.3\i686-w64-mingw32\bin;C:\Rtools\gcc-4.6.3\bin64;C:\Program Files (x86)\git\bin;C:\Program Files (x86)\Windows Live\Shared; C:\Program Files\Common Files\Microsoft Shared\Windows Live;C:\Program Files (x86)\Common Files\Microsoft Shared\Windows Live;

> shell("where gdb")
C:\Rtools\gcc-4.6.3\bin64\gdb.exe
> shell("where Rterm")
C:\Program Files\R\R-3.1.3\bin\x64\Rterm.exe

gdbsource("model.R",TRUE) runs gdb, but without output, and Rterm can be found. It might also be of interest to know that in the beginning (when I had the newest R and Rtools installed), I had the error straight away, whereas with the somewhat older version of R and Rtools (well, I think this is the reason) it now takes a long time to get the error, and also forces me to restart Rstudio, and even internet. I hope I provided enough information, and that someone knows what to do! Thank you!

[kaskr]

@Wave9 Your locations of gdb and Rterm look correct - this should work.

• Did you remember to re-build model.dll after the installation changes? (You must update the time stamp of model.cpp to re-compile...)
• Are you sure your model.cpp has out-of-bounds errors? The output argument of length 0 indicates there is no interesting output from gdb.

[ghost]

Thank you very much for your answer. I rebuild it, but I suppose it is possible that there are out-of-bounds errors (sorry that the answer might be this simple). I was hoping/expecting that this debugger might give more detailed information than what I get from MakeADFun. Thanks again, now I know that the problem is really the model, rather than the installation!

[James-Thorson]

@kaskr I finally got around to following your suggestion above, and have added an R package that overloads compile and MakeADFun:

https://github.com/James-Thorson/TMB-debug

where I imagine it has to be loaded after TMB for it to take precedence in the namespace. I invite people to test it on windows machines. If it seems like its working, we could then transfer it to a "TMB_helper_functions" package maintained on some public site (rather than housing it in my personal repository).