HOWTO:Local Rsync Mirror

If you have a couple of gentoo boxes on a LAN be a good gentoo netizen and setup a local rsync mirror. You can sync the server box across the net and then sync all your other machines faster across the LAN. This not only allows gentoo to grow faster by not abusing mirrors, but your LAN is way faster than your internet connection!!

I recommend you also set up a local package cache for downloaded package files, ebuild and HOWTO is here http://forums.gentoo.org/viewtopic.php?t=173226

Let me start by saying there is conflicting info in the forums and gentoo docs about how to setup a local rsync mirror. The howto athttp://www.gentoo.org/doc/en/rsync.xml is for OFFICIAL public mirrors! It can confuse a user wanting a LOCAL LAN mirror! There is an app-admin/gentoo-rsync-mirror ebuild that is currently outdated and possibly insecure!! If you think that ebuild has been updated and the problems fixed you should check the difference between it and this howto to see if it has been updated to include all fixes and features!

You really only need a config file to get up and running, but this version adds features and security.

Every gentoo box has rsync installed because it's part of the base system. Lets add the parts that make it a server.

1. /etc/rsyncd.conf 

Code:

uid = nobody gid = nobody use chroot = yes #limit access to private LAN's hosts allow=192.168.0.0/255.255.0.0 10.0.0.0/255.0.0.0 hosts deny=* max connections = 15 pid file = /var/run/rsyncd.pid motd file = /etc/rsync/rsyncd.motd #This will give you a separate log file #log file = /var/log/rsync.log #This will log every file transferred - up to 85,000+ per user, per sync #transfer logging = yes log format = %t %a %m %f %b syslog facility = local3 timeout = 300 #If you need this, UPGRADE portage please! #[gentoo-x86-portage] #this entry is for compatibility #path = /usr/portage #comment = Gentoo Linux Portage tree #exclude=distfiles/ [gentoo-portage] #modern versions of portage use this entry path = /usr/portage comment = Gentoo Linux Portage tree mirror exclude = distfiles/

2. /etc/init.d/rsyncd 

Code:

#!/sbin/runscript # Copyright 1999-2002 Gentoo Technologies, Inc. # Distributed under the terms of the GNU General Public License, v2 or later # $Header: /var/www/www.gentoo.org/raw_cvs/gentoo/xml/htdocs/doc/en/rsync.xml,v 1.26 2004/05/03 12:30:06 neysx Exp $ depend() { need net } # FYI: --sparce seems to cause problems. RSYNCOPTS="--daemon  --timeout=300" start() { ebegin "Starting rsync daemon" start-stop-daemon --start --quiet --pidfile /var/run/rsyncd.pid --nicelevel 15 --exec /usr/bin/rsync -- ${RSYNCOPTS} eend $? } stop() { ebegin "Stopping rsync daemon" start-stop-daemon --stop --quiet --pidfile /var/run/rsyncd.pid eend $? }

3. /etc/rsync/rsyncd.motd

Code:

Welcome to My Local Rsync Mirror!!

Then start rsyncd

Code:

# /etc/init.d/rsyncd start

rsyncd should be started at boot - add to default runlevel 

Code:

# rc-update add rsyncd default

Now you have a local server up and running. You must point your client boxes to this new server.

Edit /etc/make.conf on the client boxes and change the SYNC option to your new server.

The SYNC section should look like this : 

Code:

#SYNC="rsync://rsync.gentoo.org/gentoo-portage" SYNC="rsync://YourMirrorHere.com/gentoo-portage"

Replace YourMirrorHere.com with your server's Name or IP address.


Remember to open port 873 on the rsync server box or any firewall between your clients and the rsync server box.

Then kick back and admire your work, knowing your client boxes can sync faster over the LAN and your saving Gentoo/yourself alot of bandwidth!!

NOTE 1: My /etc/init.d/rsyncd above has --nicelevel set to 15. This purposely slows down the rsync server so you won't notice other boxes syncing - This is what you want if you spend time logged in on the server box. If you only use your rsync server as a server, set the nicelevel to 0 to make the rsync server run at normal speed (change the line with --nicelevel 15 to --nicelevel 0 in the /etc/init.d/rsyncd above. You could also set the nicelevel to max 19 to slow it down even more ).

NOTE 2: The default script installed with rsync is similiar but omits nicelevel, and timeout. You could use the default script and and add "--timeout=300" to /etc/conf.d/rsyncd , but that would still leave out setting the nicelevel which can only be done by editing the file. By replacing the script with my verson, I've kept all the changes in one file, which I find easier when merging future updates......

UPDATE* Latest versions of rsync support bandwidth limiting. Large shared installations might consider this option.
--bwlimit=KBPS limit I/O bandwidth; KBytes per second

If you need to setup this local rsync server on a non-gentoo box, this will sync your data with the official gentoo rsync servers: 

Code:

#!/bin/bash RSYNC="/usr/bin/rsync" OPTS="--quiet --recursive --links --perms --times --devices --delete --timeout=300" #Uncomment the following line only if you have been granted access to rsync1.us.gentoo.org #SRC="rsync://rsync1.us.gentoo.org/gentoo-portage" #If you are waiting for access to our master mirror, select one of our mirrors to mirror from: SRC="rsync://rsync2.de.gentoo.org/gentoo-portage" DST="/space/gentoo/rsync/" echo "Started update at" `date` >> $0.log 2>&1 logger -t rsync "re-rsyncing the gentoo-portage tree" ${RSYNC} ${OPTS} ${SRC} ${DST} >> $0.log 2>&1 echo "End: "`date` >> $0.log 2>&1

Let me say again, most of this info is taken from the Official Howto BUT modified for LOCAL use. The gentoo-rsync-mirror ebuild is designed for OFFICIAL PUBLIC mirrors only.

Version 1.8
--safe-links no longer needed or supported in daemon mode
added note about bwlimiting
Version 1.7
Added script note
Version 1.6
Updated rsync.conf location
added non-gentoo
Version 1.5
Added firewall port
Version 1.4
Cleaned up [gentoo-x86-portage]
section in case someone actually needs it
Version 1.3
Added trailing / to 'exclude distifles'
Version 1.2
Added nicelevel note
Version 1.1
added hosts allow/deny
Version 1.0
Initial Post