Design Distributed and Parallel From The Start

Posted by Lydia Vasilyeva 2 years, 9 months ago 8 comments

The first part of every development effort is a proof of concept (POC) to demonstrate the concept and prove it possible. Since it’s built quickly and with a specific goal of showing the user, the back-end doesn’t get much love. It is usually minimal, just enough to get the POC to run.

If the concept gets approved, a slew of new requirements require the team to keep moving forward on enhancing the POC. Sometimes this works out fine; however sometimes it doesn’t. When it comes down to making systems that process data, building it on a distributed model with easy parallel processing is the key to scaling easy.

Trust me when I tell you; spending a few cycles on building in parallelism into your application before you start writing code to process your data is much easier than later retrofitting your code with it. Here are a few tips:

  1. Write your app so it is easy to deploy and manage on multiple hosts.
  2. Avoid state whenever possible; make it easy to load balance requests.
  3. Share data through network queues.
  4. Make sure each component can parallelize it’s work and scale independently.
  5. Don’t store data in files whenever possible. They require reading, writing, transfer and are local!
Current rating: 5