--- On Sun, 5/20/12, Timothy Pearson <kb9vqf@...> wrote:

> From: Timothy Pearson <kb9vqf@...>
> Subject: Re: [trinity-devel] Que Pasa Bro -- What be the Story Tim?
> To: trinity-devel@...
> Date: Sunday, May 20, 2012, 1:31 PM
> > On 05/20/2012 01:05 AM, Timothy
> Pearson wrote:
> >> Nothing serious on this end fortunately. :-)
> >>
> >> The ISP that services the TDE server cluster had a
> localized equipment
> >> failure, but instead of trying to fix the outage
> they decided to blame
> >> the
> >> customer (me).  This prolonged the outage
> significantly, even though in
> >> the end it did turn out to be upstream ISP
> equipment failure.
> >>
> >> I do apologize for the outage; as you are probably
> aware the TDE
> >> services
> >> can only utilize a single carrier due to the
> long-term contractual
> >> expense
> >> of bringing out a redundant service provider to the
> server location.
> >>
> >> Tim
> >
> > Have we approached any of the high-bandwidth
> Universities or OpenSource
> > outfits
> > to see if they would be willing to provide mirror
> services?  I see so many
> > run-of-the-mill projects that have mirror services all
> over the world. It
> > sure
> > seems like a favorable project like TDE would be a
> prime candidate to be
> > accepted and helped by some of these institutions.
> 
> <snip>
> 
> We do mirror most of our static binary content on a
> worldwide mirror system.  However, most of the TDE services are dynamic
> (GIT, Wiki, mailing lists, searchable documentation, IRC, bugtracker, patch
> lists, etc.) and therefore cannot (easily/safely) be mirrored.
> 
> Good idea though!

I'm not concerned about the type of outage we experienced. Those things happen.

If the recent outage was a hosting plan squabble and you still had normal customer ISP access, then a personal email to somebody would have helped to pass the word as much as possible. If you lost personal services too, then perhaps a trip to the library or local coffee shop would have been in order to send an email or leave a message in IRC.

The recent outage leaves me concerned about what we can do should something serious happen long-term. Nasty weather and natural disasters could disrupt an ISP connection for days, weeks, or months. Serious illness could keep you out of touch too.

We need a basic contingency plan. Something so the project stays alive and the remaining participants are not left hanging.

When the outage occurred I had the latest (known) GIT sources. Worst case then the project could have move toward some kind of restoration with that, minus a few patches. Yet when I think about the bug tracker, etherpad, wiki, and mail list archives disappearing then I do not feel so comfortable.

Granted, part of what we do is dynamic, but not so much that a basic contingency plan can't restore.

Currently we have two domains: www.pearsoncomputing.net and www.trinitydesktop.org. If both are housed from the same location/ISP, then we become dependent upon those people not to get a bug up their backside. Perhaps we can move one domain elsewhere. Basically, diversify --- don't put all our eggs in one basket.

We do not need to mirror everything in the traditional sense. Some work-arounds would save the project.

For example, some of us can rsync our local GIT tree at least daily or more often. We don't need to "mirror" those sources if several of us do that. GIT is a distributed system so all we need are people keeping synced.

Losing the bug tracker would be a huge challenge. Is there a way some of us can rsync or spider the bug tracker to our personal systems? Likewise for the wiki, etherpad, mail list archives. Such efforts would be the equivalent of a backup system.

None of this need cost additional funds. We just need a way for several project members to rsync/spider files. Should disaster strike we then have a way to get us back on our feet.

I don't want to make mountains out of molehills. I'm just asking that as a group we consider basic stewardship. :-)

Darrell