Sometimes we encounter odd application responses that seem to make no sense. One of these such issues is related to running virtual server instances (OS Containers not Para-Virtualized VMs) and attempting to back up their data to Amazon’s S3 cloud storage. For moderately sized virtual machines running MySQL databases or Python/PHP based websites and code repositories this can be an inexpensive, quickly provisioned, and easy way to provide disaster recovery backups in numerous geographic locations, since we generally want DR content to be located in a physically distant location. Nevertheless, we can encounter errors if using an S3 mount in a distance location from our server if the timezone/sync data is incorrect.
The commonly seen error is as follows – and it doesn’t give much information for troubleshooting and resolution.
WARNING: Upload failed:([Errno 32] Broken pipe) WARNING: Retrying on lower speed (throttle=0.00) WARNING: Waiting 3 sec...
The solution is seemingly unrelated to any network related or file-system settings on the virtual machine or the host server. It has to do with running S3 storage buckets in different time zones than your server and not having the system sync’d to NTP pools. So, the solution for Redhat/CentOS/Fedora/Scientific (for other Linuxes just replace the package management commands as needed):
First we have to enable the ability for the OpenVZ container to utilize NTP. Add the following line to your /etc/vz/conf/101.conf file (where 101 in this example is the ID of your own container, which you can find via the command “vzlist”).
CAPABILITY=" SYS_TIME:on"
Then restart the container(s) to get the setting to take and login to the container. You can either SSH or enter the container from the main host.
$ vzctl restart 101 $ vzctl enter 101
On the VM itself, install ntpdate package to be able to sync time data.
$ sudo yum install ntpdate
Sample ntp.conf file for NTP pool servers on CentOS 6.3. There are plenty of other configuration settings but these are the basics. This file goes on the VM server, not the host server.
$ sudo cat /etc/ntp.conf driftfile /var/lib/ntp/drift restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery restrict 127.0.0.1 restrict -6 ::1 server 0.centos.pool.ntp.org server 1.centos.pool.ntp.org server 2.centos.pool.ntp.org includefile /etc/ntp/crypto/pw keys /etc/ntp/keys
Restart the ntpdate service on the VM to sync to the pool.
$ sudo service ntpdate restart ntpdate: Synchronizing with time server: [ OK ]
Add a cron job to the VM (either in /etc/crontab or via “crontab -e”) for automatic ability to sync the time every day.
# sync date/time with ntp pool 05 01 * * * root /usr/sbin/ntpdate 2>&1 | /usr/bin/tee -a /var/log/messages
Now you can run S3 backups with throttling errors. Done and done. No more errors.