age icon indicating copy to clipboard operation
age copied to clipboard

Parrallel usage using the age wrapper

Open ishaan812 opened this issue 6 months ago • 10 comments

I was trying to parrallelise a sync script from postgres to an age graph to make it more efficient using go routines. The prepared statement started giving issues when I did it.

parrallelVerticeCreation := func(flow models.Flow) {
		defer wg.Done()
		tx, err := age.Begin()
		// TODO: see if flow aready exists before putting in
		if err != nil {
			fmt.Println("Error in creating tx", err)
		}
		err = s.repo.CreateVertice(context.TODO(), tx, &flow)
		if err != nil {
			fmt.Println("Error in creating vertice", err)
		}
		tx.Commit()
	}
	for _, flow := range flows {
		wg.Add(1)
		go parrallelVerticeCreation(flow)
	}
        wg.Wait()
	fmt.Println("All flows created")
	return nil

the create vertice just runs a create cypher query. This did not work. The error was:

SELECT * FROM age_prepare_cypher($1, $2); weber CREATE (n:Flow {id: '2', created_at: '2024-07-26 13:22:24.289636 +0530 IST', updated_at: '2024-07-26 13:22:24.289636 +0530 IST'})
Error in creating vertice pq: function age_prepare_cypher(unknown, unknown) does not exist
SELECT * FROM age_prepare_cypher($1, $2); weber CREATE (n:Flow {id: '1', created_at: '2024-07-26 13:18:19.30531 +0530 IST', updated_at: '2024-07-26 13:18:27.125151 +0530 IST'})
Error in creating vertice pq: function age_prepare_cypher(unknown, unknown) does not exist

Although if I tried with the simple sql statement this used to work to an extent (till db got overloaded and gave up if i made >500 requests or so) which is fine.

Wanted to understand what is different with age since it must be using the same sql connection that the rdb was using so wanted to u nderstand the only thing I felt was different was the prepared statement being used. If not wanted to know how I can do this better.

ishaan812 avatar Jul 29 '24 05:07 ishaan812