Optimizing Translated File Downloads

Many of our clients have the need to download a few translated files from our Smartling API every so often. But for one client in particular that continuously translated to 500 files into 10 different languages, downloading in one sitting became a bit of a time-waster.

In their case, downloading 5,000 files at once resulted in 1.5 hours of download time! This was way too long – so we decided to do a little investigating.

The Bottleneck

Potential reasons for long download times:

  • Server processing time of the translated file
  • Time to download the file over the network
  • Connection time over SSL connection

Our test case:

  • 5,000 files to download
  • Average file size is 12Kb

It was a bit of a surprise to us that establishing a connection for this client took the same amount of time as generating a translated file itself. Download time wasn’t an issue and server-side processing scales nicely with Smartling as we keep a lot of spare capacity on our API download servers. We also have the ability to scale horizontally by adding more hardware. But just imagine establishing that expensive connection 5,000 times – Ouch.

The Solution

There are a couple of benefits that are apparent right away when files are downloaded in parallel:

  1. Single file server processing stays constant – you can download many files in parallel in the same amount of time it takes to download a single file.
  2. Time to establish a connection is significantly reduced across multiple files – If we download 20 files at the same time, we are cutting the overall time it takes to make a connection 20 times.
  3. The network is utilized much more efficiently

The Results

Downloading files in parallel allowed us to reduce download time from 1.5 hours to 6 minutes.

Here’s the breakdown (ballpark):

  • 1.7 min to establish connections
  • 1.7 min for server-side processing
  • 2.6 minutes to download at roughly 3.1Mb/s

There are a number of ways to implement parallel downloads. We used gnu parallel. Download the sample bash script at our Support site and try it yourself!

To continue the conversation, tweet to Andrey at @chelya!