Skip to content

A Storage Primitive Designed for Live Migration

License

Notifications You must be signed in to change notification settings

loopholelabs/silo

Repository files navigation

Logo

License: AGPL 3.0 Discord Go Version Go Reference

Overview

Silo is a storage primitive designed to support live migration. One of the core functionalities within Silo is the ability to migrate/sync storage to various backends while it is still in use (without affecting performance).

Sources

All storage sources within Silo implement storage.StorageProvider. You can find some example sources at pkg/storage/sources.

Expose

If you wish to expose a Silo storage device to an external consumer, one way would be to use the NBD kernel driver. See pkg/expose/sources.

Block orders

When you wish to move storage from one place to another, you'll need to specify an order. This can be dynamically changing. For example, there is a volatility monitor which can be used to migrate storage from least volatile to most volatile. Also you may wish to prioritize certain blocks for example if the destination is trying to read them. pkg/storage/blocks.

Migration

Migration of storage is handled by a Migrator. For more information on this see pkg/storage/migrator.

Example of a basic migration. Here we have block number on the Y axis, and time on the X axis. We start out by performing random writes to regions of the storage. We start migration at 500ms and you can see that less volatile blocks are moved first (in blue). Once that is completed, dirty blocks are synced up (in red).

This example adds a device reading from the destination. The block order is by least volatile, but with a priority for the blocks needed for reading. You can also see on the graph that the average read latency drops as more of the storage is locally available at dest.

Same as above, but with concurrency set to 32. As long as the destination can do concurrent writes, everything will flow.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/loopholelabs/silo. For more contribution information check out the contribution guide.

License

The Polyglot project is available as open source under the terms of the AGPL, 3.0 License.

Code of Conduct

Everyone interacting in the Silo project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the CNCF Code of Conduct.

Project Managed By:

https://loopholelabs.io