Bulk load in AWS Lambda silently fails

[neilgallagher]

trafficstars

The bulk load function doesn't seem to work within a .net Core 2.0 Lambda function. I'm using version 0.40 and Ive also tried with version 0.36. If the lambda function is run locally it works fine using the exact same database connection and table . Here is my simplified code with a simple table called TestBulk which has 2 varchar columns Col1 and Col2. count always returns 0

			using ( var connection = new MySqlConnection( connectionString ) )
			{
				var bulkLoader = new MySqlBulkLoader( connection )
				{
					Local = true,
					TableName = "TestBulk",
					FieldTerminator = ",",
					LineTerminator = "\r\n",
					FieldQuotationCharacter = '"',
					NumberOfLinesToSkip = 1
				};

				bulkLoader.Columns.Clear();
				bulkLoader.Columns.Add("Col1");
				bulkLoader.Columns.Add("Col2");

				var memStream = new MemoryStream() ;

				bulkLoader.SourceStream = memStream ;

				// Note that mem stream is kept open until bulk load has finished
				using ( var bulkFileWriter = new StreamWriter(memStream) )
				{
					bulkFileWriter.WriteLine("\"Col1\",\"Col2\"") ;
					bulkFileWriter.WriteLine("\"1\",\"2\"") ;
					bulkFileWriter.WriteLine("\"3\",\"4\"") ;

					bulkFileWriter.Flush() ;

					memStream.Position = 0 ;

					int count = bulkLoader.Load();

					Logger.LogInformation($"Loaded {count} items") ;
				}
			}

[bgrainger]

Are you able to enable logging in your lambda function and get the logs? There's not much logging around bulk loading, but they still may show if something's going wrong.

Use MySqlConnectorLogManager.Provider = new ConsoleLoggerProvider(); or one of the other logging providers documented here.

[neilgallagher]

Here are some logs below. Ive tried both SourceStream from a MemoryStream and a local temp file with same silent failure

Pool1 creating new connection pool for Server=db-core-xxxxxxxxxxxxxxxxxxxxxxx;Convert Zero Datetime=true Pool1 reaping connection pool Pool1 recovered no sessions Pool1 waiting for an available session Session1.1 created new session Pool1 no pooled session available; created new Session1.1 Session1.1 connecting to IP address 10.0.97.144 for host 'db-core-sandbox-006-dbcluster-19bztz6ay4do2.cluster-cgx7liuvaaxd.eu-west-2.rds.amazonaws.com' Session1.1 connected to IP address 10.0.97.144 for host 'db-core-sandbox-006-dbcluster-19bztz6ay4do2.cluster-cgx7liuvaaxd.eu-west-2.rds.amazonaws.com' with local port 35232 Session1.1 server sent auth_plugin_name 'mysql_native_password' Session1.1 made connection; ServerVersion=5.6.10; ConnectionId=43669; Flags: Attr Eof Ssl Session1.1 initializing TLS connection Session1.1 creating connection attributes Pool1 returning new Session1.1 to caller; m_leasedSessions.Count=1 Session1.1 Execute: LOAD DATA LOCAL INFILE ':STREAM:c572809ab42447658592fb0d0c146046' INTO TABLE TestBulk FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY ' ' IGNORE 1 LINES (Col1,Col2) Session1.1 entering FinishQuerying; state = Querying Session1.1 returning to Pool1 Pool1 receiving Session1.1 back

[bgrainger]

I'm not sure what's going wrong, and I don't have access to an AWS Lambda/RDS instance to try to reproduce the problem. (And even if I did, is there any way to debug a Lambda function to try to understand why it's behaving differently than running it locally?)

[neilgallagher]

Understand. Just to add that with the increasing popularity of aws lambda / serverless it might be worth setting up a test environment for this. It may well be an AWS bug but could be difficult to convince amazon devs. Within aws mysql is a really popular choice for Rds. I also see bulk loading becoming more used as lambda functions have a max 5 minute run time and getting late large scale data sets into rds becomes a real challenge.

[bgrainger]

This issue is 4.5 years old; multiple major versions of .NET, MySqlConnector, Aurora, etc. have been released since then. Closing as unlikely to be reproducible as written.