Best Threading Practices Using SFTP in C #

Question

Good,

this is more than one of these "conceptual questions", but I hope that I have some pointers in the right direction. First, the desired scenario:

• I want to request an SFTP server for directory listings and files
• I want to upload or upload files at the same time.

Both things are pretty simple using the SFTP class provided by Tamir.SharpSsh, but if I use only one stream, it is slow. Especially recursion in subdirs is very "blocking the UI" because we are talking about 10,000 directories.

My basic approach is simple, create some kind of "pool" where I save 10 open SFTP connections. Then request the first worker for the dirs list. If this list has been received, send new free workers (for example, 1-10, the first one is also free again) to get detailed information about the subdirectory. As soon as the worker is free, send him for sub-sub. Etc...

I know ThreadPool, simple threads and did some tests. I am a little confused by the following: I basically need ...

• A list of topics I create, say 10
• Connect all streams to the server
• If the connection drops, create a new stream client / sftp
• If there is work, do the first free flow and process the work.

Currently, I'm not sure about the details of the implementation, especially about what needs to be done and about the sections “save the list of threads”.

This is a good idea:

• Inclusion of work in an object containing a job description (path) and callback
• Send threads in an infinite loop with a waiting time of 100 ms.
• If SFTP is dead, either upgrade it or destroy the entire stream and create a new one
• How to encapsulate this, I write my own "10ThreadsManager" or there are some of

Okay bye...

Btw, I could also use PRISM events and commands, but I think the problem is not related. Perhaps the EventModel is signaling that the "work package" has finished processing ...

Thanks for any ideas, critic .. Chris

Answer 1

Bouquet of small notes:

If you use some kind of .NET API that internally uses ThreadPool, then you cannot wait forever, since the OS owns threads from ThreadPool, these threads are intended to be used "briefly" and then returned back to the OS, that is, the most reliable behavior. Of course, the OS can grow the thread pool as needed if you finish them due to your lengthy processing, but the best design would be to avoid this behavior.

If you work in XP, you may also need to avoid ThreadPool (OS level and therefore .NET), as it was fixed / resigned in Vista and later, the XP version is considered less reliable.

If you use ThreadPool, you end up putting asynchronous work in it, since it is already waiting for work.

Writing your own ThreadManager is pretty easy to do, you can find many examples on this, but as always, these kinds of things should be kept as simple as possible.

For your third point, it is better to restore the SFTP connection than to kill the entire stream. If you kill a thread (if your ThreadManager can handle this, never kill threads from OS ThreadPool, of course), than first you need to return the raw task back to some place, it feels like too much work.

Answer 2

An alternative approach is to use FtpDlx from WeOnlyDo ( http://www.weonlydo.com/FtpDLX.NET/ftp.sftp.ftps.ssl.net.component.asp ). It is only $ 229 and is fully managed by the .Net 2.0 library. It includes directory recursive loading methods. It uses events to indicate progress and errors, allowing you to skip files, redirect them where you write them, and check and ignore errors. It is durable and works as advertised; we use it in multi-threaded production code.

You can run it in a separate thread and use events to update your user interface without blocking. You may even find that a single thread performing the download makes good use of your bandwidth, but will work fine with separate connections in separate threads. I would recommend you create your own threads instead of using ThreadPool, since you want to use callbacks, and the threads will usually be long anyway.

Answer 3

to use SFTP and .NET with tamir.ssh and multithead you can do this:

• Create the main instance that will start Thread.

Task.Factory.StartNew (() => new Main (). Run ());

• 1.1 Download files from SFT

• 1.2 Downloading files from SFTP

BASIC class

public class Main {Task [] _tasks = null;

public void run() { _tasks = new Task[2]; var task = new Task(() => Download(),TaskCreationOptions.LongRunning); task.Start(); _tasks[0] = task; var task = new Task(() => Download(),TaskCreationOptions.LongRunning); task.Start(); _tasks[1] = task; Task.WaitAny(_tasks); } private void Upload() { Sftp _Sftp = null; while (true) { filesToUpload = GetFiles(Sourcepath); Parallel.ForEach(filesToUpload, _fileData => { if(!_Sftp.Insconnected) { _Sftp = new Sftp(Host, User,Password); _Sftp.Connect(); } if(_Sftp.Connected) _Sftp.Put(_fileData.Path, DestinyPath); }); } } public void Download() { Sftp _Sftp = null; While(true) { if(!_Sftp.Connected) { _Sftp = new Sftp(Host, User,Password); _Sftp.Connect(); } if(_Sftp.Connected) 

{

ArrayList fileToDownload = _Sftp.GetFileList (_Instance.Sourcepath);

  Parallel.ForEach(fileToDownload, _fileData => { _Sftp.Get(Sourcepath + "/" + _fileData.Name,DestinyPath); }); } } }