使用 FluentFTP 从 FTP 并发下载多个文件，最大值

Download multiple files concurrently from FTP using FluentFTP with a maximum value(使用 FluentFTP 从 FTP 并发下载多个文件，最大值)

问题描述

我想从 FTP 目录递归下载多个下载文件，为此我使用 FluentFTP 库，我的代码是这样的:

I would like to download multiple download files recursively from a FTP Directory, to do this I'm using FluentFTP library and my code is this one:

private async Task downloadRecursively(string src, string dest, FtpClient ftp)
{

    foreach(var item in ftp.GetListing(src))
    {
        if (item.Type == FtpFileSystemObjectType.Directory)
        {
            if (item.Size != 0)
            {
                System.IO.Directory.CreateDirectory(Path.Combine(dest, item.Name));
                downloadRecursively(Path.Combine(src, item.Name), Path.Combine(dest, item.Name), ftp);
            }
        }
        else if (item.Type == FtpFileSystemObjectType.File)
        {
            await ftp.DownloadFileAsync(Path.Combine(dest, item.Name), Path.Combine(src, item.Name));
        }
    }
}

我知道你每次下载都需要一个 FtpClient，但是我怎样才能使用一定数量的连接作为最大值，我想这个想法是创建、连接、下载和关闭我找到的每个文件，但只是同时有 X 个下载文件.另外我不确定我是否应该使用异步、线程和我最大的问题来创建任务，如何实现所有这些.

I know you need one FtpClient per download you want, but how can I make to use a certain number of connections as maximum, I guess that the idea is to create, connect, download and close per every file I find but just having a X number of downloading files at the same time. Also I'm not sure if I should create Task with async, Threads and my biggest problem, how to implement all of this.

来自 @Bradley 的答案看起来不错，但问题确实读取了必须从外部下载的每个文件文件并且它没有最大并发下载值，所以我不确定如何应用这两个要求.

Answer from @Bradley here seems pretty good, but the question does read every file thas has to download from an external file and it doesn't have a maximum concurrent download value so I'm not sure how to apply these both requirements.

推荐答案

使用:

• ConcurrentBag 类 实现连接池；
• Parallel 类 并行化操作；
• ParallelOptions.MaxDegreeOfParallelism 限制并发线程数.

var clients = new ConcurrentBag<FtpClient>();

var opts = new ParallelOptions { MaxDegreeOfParallelism = maxConnections };
Parallel.ForEach(files, opts, file =>
{
    file = Path.GetFileName(file);

    string thread = $"Thread {Thread.CurrentThread.ManagedThreadId}";
    if (!clients.TryTake(out var client))
    {
        Console.WriteLine($"{thread} Opening connection...");
        client = new FtpClient(host, user, pass);
        client.Connect();
        Console.WriteLine($"{thread} Opened connection {client.GetHashCode()}.");
    }

    string remotePath = sourcePath + "/" + file;
    string localPath = Path.Combine(destPath, file);
    string desc =
        $"{thread}, Connection {client.GetHashCode()}, " +
        $"File {remotePath} => {localPath}";
    Console.WriteLine($"{desc} - Starting...");
    client.DownloadFile(localPath, remotePath);
    Console.WriteLine($"{desc} - Done.");

    clients.Add(client);
});

Console.WriteLine($"Closing {clients.Count} connections");
foreach (var client in clients)
{
    Console.WriteLine($"Closing connection {client.GetHashCode()}");
    client.Dispose();
}

另一种方法是启动固定数量的线程，每个线程都有一个连接，然后让它们从队列中挑选文件.

Another approach is to start a fixed number of threads with one connection for each and have them pick files from a queue.

有关实现的示例，请参阅我关于 WinSCP .NET 程序集的文章:
通过 SFTP/FTP 协议在并行连接中自动传输

For an example of an implementation, see my article for WinSCP .NET assembly:
Automating transfers in parallel connections over SFTP/FTP protocol

关于 SFTP 的类似问题:
使用 C# Parallel.ForEach 循环处理 SFTP 文件而不处理下载

这篇关于使用 FluentFTP 从 FTP 并发下载多个文件，最大值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持编程学习网！