How to reliably, but minimally distribute elements through a peer-to-peer system

If you have a peer-to-peer system that you can request, you need

  • reduce the total number of requests over the network (by distributing โ€œpopularโ€ items broadly and โ€œsimilarโ€ items together)
  • avoid excess storage on each node
  • ensure good accessibility even for moderately rare items in the face of customer downtime, equipment failure and user exit (possibly to detect rare items for archivists / historians).
  • avoid queries that are not found matches in the case of network partitions

Given these requirements:

  • Are there any standard approaches? If not, are there any reputable but experimental studies? I am familiar with some distribution schemes, but I have not seen anything really address learning for reliability.
  • I do not see any obvious criteria?
  • Anyone interested in working on this issue? (If so, I am happy with the open source code of a very powerful simulator that I put together this weekend and generally offer worthless advice).

@cdv: now I watched the video, and itโ€™s very good, and although I donโ€™t feel that it is well suited to a flexible distribution strategy, it is certainly 90% of the way. However, these questions point to useful differences with this approach, which touch on some of my further problems and give me some recommendations for further study. Thus, I temporarily accept your answer, although I am considering the question openly.

+4
source share
2 answers

If you have the time, itโ€™s worth checking out the Google technical conversation that Voila gave . They discuss the same problems they encountered when creating their peer-to-peer file system.

+2
source

There are many systems with various aspects of what you are looking for, and each of them makes different trade-offs, including but not limited to:

Amazon Dynamo: http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf

Kai: http://www.slideshare.net/takemaru/kai-an-open-source-implementation-of-amazons-dynamo-472179

Hadoop: http://hadoop.apache.org/core/docs/current/hdfs_design.html

Chord: http://pdos.csail.mit.edu/chord/

Beehive: http://www.cs.cornell.edu/People/egs/beehive/

and many others. After creating a user system on these lines, I also allowed some of the building blocks in an open source form: http://code.google.com/p/distributerl/ (this is not an entire system, but several libraries useful in building)

+2
source

Source: https://habr.com/ru/post/1276502/


All Articles