How to write large dataframes to a CAS table

[stanwir]

trafficstars

Hi, I am trying to write a large set of cashflows to a cas table. I am using from_records. I create a SASDataframe and pass it to from_records. However, when the number of cashflows grows, the python process crashes. Is there a way to write a subset of cashflows at a time to the same cas table? What is the bet way to do this?

[bkemper24]

One possibility would be to use the addTable action.

You would need to create a data message handler for the SASDataFrame ( swat.cas.datamsghandlers.PandasDataFrame ). One of the parameters for PandasDataFrame is "nrecs". According to the doc, the records are uploaded in batches "nrecs" long, so you may be able to write the entire SASDataFrame with just one addTable by tweaking the nrecs value. The PandasDataFrame and addTable action would look something like this: dmh = datamsghandlers.PandasDataFrame(sdf,nrecs=50000) # where sdf is the name of your SASDataFrame tbl = s.addtable(table='myTable', **dmh.args.addtable)

If you can't get the entire SASDataFrame written to the cas table with a single addTable action, you can split the data up into subsets and issue multiple addTable actions with the append=True parameter specified. append=True will cause the data to be added to an existing table rather than creating a new table. You will need to create a new data message handler for each addTable. Note that if you want to do the data message hander / addTable in a loop, it's ok to specify append=True on the first addTable as well - if the table does not exist it will create the table, otherwise it will append to an existing table.

How are you creating the SASDataFrame ? The reason I ask, if you are uploading .csv files into a SASDataFrame, there is also a data message handler for CSV files ( swat.cas.datamsghandlers.CSV ), so you may be able to go straight from the csv file(s) into the addTable action.