[H-SASIG] Move of Excalibur

[James Iseppi]

Hi Russell and other interested parties,

One issue I did notice early this morning after I arrived home at  
~3:00am (definitely not a good time to be looking at technical  
problems), was that one of the DNS servers [1] wasn't in sync with the  
DNS changes that you had made. This is because my server was still  
using the old excalibur's IP address as the master, and the zone on  
the old server had not been changed to reflect the changes, therefore  
creating two different views of the humbug.org.au domain in the DNS.  
This could have been why there were some issues with using  
humbug.org.au vs www.humbug.org.au because it would have been  
approximately a 2 in 5 [2] chance of getting the wrong data.

In my experience with migrating to new servers, the best thing to do  
is to:
1. Drop the TTL's of all records to a low value [3], to minimise any  
outage caused by IP address changes.
2. Get DNS working on the new server as the master without migrating  
any other services.
3. Fix up the glue records at the appropriate registry
4. Get all DNS servers to pull from the new master (including the old  
master).

This means that there is minimal chance of having different records  
being provided by the various name servers (including the old and new  
masters). It also means that if you can't re-configure one of the  
slaves immediately the changes will still make it to that slave  
through the old master.

Based on the current state of DNS I see several issues. We currently  
have 3 servers configured at the registry as name servers, those being  
ns2.zoneedit.com, ns16.zoneedit.com and cartman.pipegrep.com.au. While  
we have 3 servers configured in the zone on excalibur,  
ns2.zoneedit.com, ns16.zoneedit.com and excalibur.humbug.org.au [4].  
Also the old excalibur is not receiving any zone updates from the new  
excalibur which could cause issues if glue records aren't up to date  
in a cache somewhere, and you can't control the TTL [5] of glue  
records at the registry, so there is little you can do to prevent  
this. To fix this we need to decide which servers are going to be  
used, and ensure that the data is consistent at both the registry and  
in our own zone file, as differences like this can cause very  
unexpected behaviour.

I also note that the old excalibur is still providing information  
using an older revision of the humbug.org.au zone file, which could  
similarly cause issues based on any older caches out there, this is  
not likely to be an issue anymore as it has been more than 4 hours [5]  
since the migration occurred, but there are broken caches out there  
that will not expire records when they should [6]. My suggestion would  
be to get the old excalibur acting as a slave for the new server.

If I've missed anything obvious, or if you have any comments about the  
above, please feel free to let me know.

Thanks
James


[1] The one I'm providing, cartman.pipegrep.com.au
[2] Since there are 5 potential name servers involved depending on the  
state of the glue records and caches at the time of the query.
[3] I normally go with 300 seconds (5 minutes), but I note that 600  
seconds (10 minutes) appears to have been used in this migration  
unless this was the pre-existing value.
[4] Which are not all the same as the ones delegated to by the registry.
[5] It appears fixed at 14400 (4 Hours) for .au domains, not sure  
about others
[6] Not really our problem, but having the old server provide current  
DNS for at least a small while, would minimise this issue.

On 31/12/2009, at 8:44 PM, Russell Stuart wrote:

> Excalibur is on its new home.  Bar one annoying little problem, it  
> seems
> to be working.
>
> Could you all test it (log in, look at the wiki, send email, use DNS)
> and so on, and report back here.  Bonus points for spotting the  
> problem.
>
> _______________________________________________
> Sasig mailing list
> Sasig at lists.humbug.org.au
> http://lists.humbug.org.au/mailman/listinfo/sasig