Repository navigation
Out Of Memory in Azure AKS deployment (Memory leak in FTPS) #710
Description
Computer OS: Debian based docker image. (buster-slim version)
FluentFTP Version: 33.0.3
Hi,
We have deployed a service in NetCore 3.1.407 that sends files to different outputs (Azure Storage, SFTP, FTPS, Sharepoint Online, etc.). In case the selected output is FTPS we are using a FluentFTP client. This service is deployed in an AKS cluster, inside a Debian docker image.
When the service is sending small files (~10kb) to an ouput that is not Ftps the RAM is around 200MB more or less, but when it is sending the same file to a Ftps output the RAM increases to 800MB/900MB and it is increased slowly in the next iterations until the pod restarts because of an Out Of Memory.
This behaviour is not reproduced when we run the service inside Visual Studio in our personal machines. The RAM increase up to ~220MB and then the garbage collector decrease the consumed RAM.
Logs :
We don't have exceptions. All the files are sent properly to the Ftps server until the pod restarts because of the out of memory.
Code:
public async Task SendFileAsync(
int executionId,
File file,
SecurityObjectUserPassword securityObject,
string certificateFingerprint)
{
var decryptedPassword = _encryptor.Decrypt(securityObject.Password);
using var ftpsClient = new FtpClient(
securityObject.Endpoint.Host,
securityObject.Endpoint.Port,
securityObject.User,
decryptedPassword);
await ConfigureAndConnectFtpClientAsync(ftpsClient, securityObject, certificateFingerprint);
await ChangeWorkingDirectoryAsync(ftpsClient, securityObject);
await UploadFileAsync(ftpsClient, file, executionId);
}
private async Task ConfigureAndConnectFtpClientAsync(
FtpClient ftpsClient,
SecurityObjectUserPassword securityObject,
string certificateFingerprint)
{
var timeoutInMilliseconds = _timeoutsOptions.FtpsConnectTimeoutInSeconds * 1000;
ftpsClient.OnLogEvent = OnLogEvent;
ftpsClient.EncryptionMode = FtpEncryptionMode.Explicit;
ftpsClient.SslProtocols = SslProtocols.Tls12;
ftpsClient.ValidateCertificate += new FtpSslValidation((ftpClient, e) => OnValidateCertificate(e, certificateFingerprint));
ftpsClient.ConnectTimeout = timeoutInMilliseconds;
ftpsClient.ReadTimeout = timeoutInMilliseconds;
ftpsClient.DataConnectionConnectTimeout = timeoutInMilliseconds;
ftpsClient.DataConnectionReadTimeout = timeoutInMilliseconds;
_logger.LogInformation("Connecting to Ftps {connection} with user {user}", securityObject.Endpoint.OriginalString, securityObject.User);
await ftpsClient.ConnectAsync();
_logger.LogInformation("Connected");
}
private async Task ChangeWorkingDirectoryAsync(FtpClient ftpsClient, SecurityObjectUserPassword securityObject)
{
var workingDirectory = securityObject.Endpoint.AbsolutePath.TrimStart('/');
if (!string.IsNullOrWhiteSpace(workingDirectory))
{
_logger.LogInformation("Changing directory from uri '{uri}' to '{workingDirectory}'",
securityObject.Endpoint.AbsolutePath,
workingDirectory);
await ftpsClient.SetWorkingDirectoryAsync(workingDirectory);
}
}
private async Task UploadFileAsync(FtpClient ftpsClient, File file, int executionId)
{
_logger.LogInformation("Uploading file {fileName}", file.FinalName);
var blobStream = await _blobStorageClient.GetBlobStreamAsync(file.InternalName);
var uploadResult = await ftpsClient.UploadAsync(
fileStream: blobStream,
remotePath: file.FinalName,
existsMode: FtpRemoteExists.Overwrite,
createRemoteDir: true);
if (uploadResult != FtpStatus.Success)
{
_logger.LogWarning(
"Execution {executionId}. Error uploading file {file} with code {Code}",
executionId,
file.InternalName,
uploadResult.ToString());
throw new FtpException($"Execution {executionId}. Error uploading file {file.InternalName} with code {uploadResult}");
}
_logger.LogInformation("File {fileName} uploaded", file.FinalName);
}
Activity
robinrodricks commentedon Apr 15, 2021
robinrodricks commentedon Apr 17, 2021
Try disabling SslBuffering.
ederpf commentedon Apr 19, 2021
Hi @robinrodricks
Thank you for your messages. We have tried disabling it without good results.
ftpsClient.SslBuffering = FtpsBuffering.Off;We have executed the tests three times sending in each 100 small files (7kb) to the same FTPS server. We can see in the capture that the CPU increases a little and it decreases when the test finishes. But the Memory only increases, a lot the first time and a little in each iteration of the tests.
Maybe the implementation of the FtpClient in the library can be improved, I am not sure..
Take please a look at these articles about HttpClient, maybe the key is there:
https://www.aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/
http://byterot.blogspot.com/2016/07/singleton-httpclient-dns.html
This is only happenning on images running on Linux so maybe a test in a docker image can be done as running the library in visual studio works perfetly
robinrodricks commentedon Apr 19, 2021
This is great info. Sadly we don't use HttpClient, we only use Socket internally (inside FtpSocketStream). FtpClient is a simple class and has support for all FTP methods but does not get involved in FTP internals. All those are handled by the stream classes. In the best case, there is a memory leak in FtpClient (unlikely since you said there is no issue when tested locally).
So more likely a weird OS specific behavior is occurring due to some handling inside the stream classes. Not sure if I can classify this as a bug, but its something I would want to improve for sure. I can almost guarantee its not something as simple as disposing (as all objects created are disposed), however, there is some wierdness that I observed during disposing - there are always uncaught errors due to various factors that I cannot control. Perhaps it is an error there, or, like I said, just some OS specific wierdness.
FtpSslStream might have been the cause of this leak, but its not used in .NET Core/.NET 5. Only .NET Framework versions have it compiling based on compiler constants, therefore I think you can rule it out.
Generally I would have debugged memory leaks with a memory profiler that lets me see the type of objects that are consuming RAM, thus allowing us to very quickly isolate the flaw, something like Redgate Memory Profiler, but not sure if you have a way to do that on AKS. This would, for example, allow you to see if its the Socket object that is causing the leak, or a user C# class, or something else inside .NET Core implementation. .NET Core on Unix is also known to have edge cases and people are still fixing issues as they come up as its not as mature as .NET Fx on windows.
[-]Out Of Memory in AKS deployment[/-][+]Out Of Memory in Azure AKS deployment[/+][-]Out Of Memory in Azure AKS deployment[/-][+]Out Of Memory in Azure AKS deployment (Memory leak in FTPS)[/+]ederpf commentedon Apr 21, 2021
Hi @robinrodricks ,
Thanks you for all the information on your message.
We have tried updating the FluentFtp version to the latest (33.1.6) and using another Linux distribution, for example "3.1.13-focal" that is Ubuntu. The Net. SDK version used is "3.1.407". But we are facing the same issues with the memory.
A workmate has a Linux machine and we are going to try to run the service there and do a memory profiling with a tool.
Other options we can try are:
ederpf commentedon Apr 26, 2021
Hi again,
We have solved the issue setting this property :c)
ftpsClient.ValidateCertificateRevocation = false;
robinrodricks commentedon Apr 27, 2021
This is fantastic!
robinrodricks commentedon Apr 27, 2021
I've changed the default to false so it does not cause anyone else the same headache it caused you!
robinrodricks commentedon Apr 27, 2021
Released.
https://www.nuget.org/packages/FluentFTP/33.1.8