Can CCIPCA be used to do partial_fit and transform incrementally in one iteration loop on a large dataset?

[jagandecapri]

Hi @kevinhughes27,

I have a dataset that has 2 million-plus samples. Can I do partial_fit and transform in the same iteration loop as (1) or the partial_fit needs to be called separately before transform as (2)?

(1)

for example in examples:
 ccipca.partial_fit(example)
 ccipca.transform(example)

(2)

for example in examples:
 ccipca.partial_fit(example)

for example in examples:
 ccipca.transform(example)

[kevinhughes27]

I think (and it has been a long time since I've worked on this) that both could be valid depending on what you want. If you call partial fit then transform you'll be getting transformations before the model is fully trained. For some online situations this is what you want however if you want to fully train the model you should use the two separate loops to fully train before transforming.