Hi Slávek,
cc: Tim Bishop

Very helpful information here from Tim.

In short, could we change the mirror list on
  https://www.trinitydesktop.org/mirrorstatus.php
to replace his copernicus and kuiper with just
  http://www.mirrorservice.org/sites/trinitydesktop.org/trinity/
while our redirector could continue to use copernicus and kuiper?

Primary mirror would not be affected by this change.

Thanks,

--Mike


On Sun July 19 2020 03:25:35 Tim Bishop wrote:
> Hi Mike,
>
> The two servers sync independently. Whilst this is not an ideal
> situation, and can lead to them having slightly out of step content,
> it's the simplest approach and requires the least development and
> maintenance work. In theory they should sync within an hour of each
> other, so the likely reason for them to have different files for an
> extended period is that one of them is failing to sync. Obviously this
> is something that's quite easy to investigate at the time, but
> impossible to work out later. So if you see this please let us know at
> the time so we can see why it's happening.
>
> The load balancer doesn't take much action regarding this. It doesn't
> know which mirrors are up to date or not, just whether they're
> functioning. However, it does have "stickiness" enabled, which means a
> client from the same IP should repeatedly hit the same backend unless
> there's a failure. So they should at least get consistent content.
>
> My main gripe is about publishing the details of the backend systems
> directly. This leads to people using them not via the load balancer and
> then complaining when one is down. In your case you have a smart
> redirector in front, so presumably that'd deal with that problem without
> inconveniencing the users. I don't have any problem with you doing that
> but I'd be happier if the backend URLs weren't so obviously advertised.
>
> The period of downtime in early June was because we had a complete
> data centre outage for the first time in over 10 years. The UPS systems
> were being upgraded and a failure occurred which powered everything
> off - the systems supposed to save us were the ones that failed us :-(
> The Mirror Service sadly isn't a priority system for the University,
> compared to core teaching and research systems, so we had to wait until
> full power was restored before we could turn it back on.
>
> That issue also killed copernicus, or more specifically its RAID
> controller. I have now replaced that and spent the past couple of weeks
> rebuilding it and resyncing the data. This has proved quite time
> consuming but it is starting to come back online this weekend.
>
> Tim.
>
> On Sat, Jul 18, 2020 at 11:35:18AM -0700, Mike Bird wrote:
> > Hi Tim,
> >
> > Thank you for mirroring TDE!
> >
> > I don't maintain that web page so this is not a direct response
> > to your request but rather a request for further information.
> >
> > FYI https://www.trinitydesktop.org/mirrorstatus.php is not the
> > normal route whereby users retrieve files.  Users work instead
> > via a smart redirector http://mirror.ppa.trinitydesktop.org/
> >
> > The status page is more of a debugging tool.
> >
> > We monitor mirrors to determine when they have pulled updates.
> > There have been several extended periods when copernicus and
> > kuiper have held different files.  We use this information in
> > the smart redirector to only send user traffic to mirrors which
> > are known to have the file the user needs.
> >
> > I was wondering to what extent the UK Mirror Service load balancer
> > addresses this as the last time I saw copernicus rsync from us
> > was June 19th?  And earlier in June there was a period of a few days
> > when neither kuiper nor copernicus were updating?
> >
> > Thanks again!
> >
> > --Mike