WDY Enterprises, LLC Disaster Recovery Mode

We've made significant progress in resotoring the most critical functions, but things still aren't up to 100%..

On Tuesday, September 2, 2025, we experienced a critical failure of multiple components at our primary data center. As of the evening of Friday, September 5, 2025, we've restored the most essential functionality.

This has been an extremely trying time and, bluntly, my temper and nerves are a bit frayed after averaging 20-hour workdays for 4 consecutive days. We will still do a full write-up, but that may not be until next week. Human factors and my day job are becoming the determinants in how recovery work progresses.

We do not expect to make time estimates at the present time.

Being able to make up estimates would be nice, but doing so diverts our focus from the recovery work. We will note progress as the work continues.

This web page is the best place to get information about this outage.

If you call, text, or email to ask a specific question, (a) we're going to be slow to respond, and (b) it's taking us away from the work of recovery. We know you're concerned, and we're doing all in our power to restore things as quickly as possible.

What does this mean for you?

1. Your access to email, webmail, and user preference services is currently disrupted.
   • The most important services have been restored: As things currently stand, user email is being received, and users who rely on standalone email client programs such as Thunderbird on desktops/laptops, or phone/tablet apps for mail (but not webmail) have most of their services restored. Technically this means the following core protocols are working:
     1. (E)SMTP mail delivery
     2. POP3+TLS and POP3 mail retrieval
     3. IMAP+TLS and IMAP mail retrieval
     4. Submission (note that STARTTLS+AUTH are required)
     5. Webmail (functional, but missing imports of address books)
   • Some services are not yet restored:
     1. Webmail contact information has not been restored (backups are from 9/1/2025; conversion requires new code and logic)
     2. Website contents
     3. User preference controls (will also be a fresh implementation, because we changed too much to use the old code)
     4. Email Quarantines (fresh implementation)
     5. Spam Filtering (fresh implementation)
     6. Disk Quotas (fresh implementation)
     7. User Payment Engine and annual renewal invoicing
2. What specific impact can we discuss at this point in time?
   1. NEW: POP3/IMAP mail users (non-Webmail users) have restored services.
      • Some (all) messages may appear to be unread; that may fix itself, or not, depending on your mail client. Internal server message numbering was (necessarily) different, which means that you may need to review your messages again.
   2. We have been receiving email continuously at our secondary site, so not all mail was lost, but we believe some was.
   3. We moved primary email to our secondary site and it became functional roughtly 12:20am on 9/3/2025.
   4. Incoming mail has slowed down, suggesting that remote sites have also caught up.
   5. Some email has been lost, but we're unable to tell exactly how much.
      • There were no usable backups from the morning 2025-09-02 backup run (reflecging changes on 2025-09-01).
      • The last usable backup has been restored, from morning of 2025-09-01.
      • Mail received at the primary site after the 2025-09-01 backups ran (it finished before 6a eastern) is lost.
      • Logging data was also lost, so we don't have any idea what mail came in during that time.
   6. There have been a number of necessary software and hardware changes.
   • Webmail: Squirrelmail is beyond support; we'll use RoundCube mail
   • Mail Transfer Agent: replaced Sendmail with Postfix
   • Mail Services: upgraded Dovecot to current version
   • Web: replaced Apache HTTPD with Nginx
   • DB: replaced MySQL with MariaDB, rushing 80% ready upgrade plan into place
   • OS: moved to modern OS (Debian) rather than archaic Gentoo Linux
   • Quotas: discontinuing Linux aquota system; will transition to soft quotas with automated cleanup.
   • Anti-Spam: old version of SpamAssassin doesn't work; will have to upgrade

We ask that you remain patient, and check this page for updates.

This page will be updated as the situation evolves.

The highest recovery priorities are as follows:

1. 100% COMPLETE- Restore operation of email transmission protocols (ESMTP)
2. 100% COMPLETE - Restore email messages from backup
3. 100% COMPLETE - restore operation of email retrieval protocols (POP3, IMAP, SUBMIT)
4. 80% COMPLETE - restore web mail access applications
   1. GAP: we still need to figure out how to import Squirrelmail address books into RoundCube email.
5. 30% COMPLETE - restore normal website operations - deemed non-essential

1. restore user preference applications
2. implement new spam filters
3. disk quota control
4. payment engine

Approximate Timeline of events (all times US/Eastern) (decreasing order):

• 20250905-1910: work suspended until tomorrow morning: a full night's sleep, a glass of wine, and some time with my family are warranted.
• 20250905-1800: webmail services restored at https://wdyllc.com.
• 20250905-1520: work resumed
• 20250905-0800: work suspended - primary job
• 20250905-0630: work resumed
• 20250904-0152: work suspended - sleep required.
• 20250903-1950: restored Internet-facing access to email for users using mail programs such as Thunderbird (POP/IMAP client programs). Appears seamless, other than some messgaes being marked read/unread.
• 20250903-1923: user authentication is updated. The modification to user preferences (including passwords) that we could recover from the backups was performed at 2025-08-31 12:00:02 EDT. If you changed your password or made any other changes at https://wdyllc.com/preferences/, those changes were lost.
• 20250903-1608: all mail access protocols work some work is required to get updated passwords for users.
• 20250903-1550: most access modules functioning correctly; still debugging email submission.
• 20250903-1510: work resumes.
• 20250930-1100: work suspended - primary job
• 20250903-0725: authentication module reasonbly functional and secure; integration begins around scheduled work calls for primary job.
• 20250903-0520: work resumes.
• 20250903-2100: human factors - 2 hours sleep in almost 40 hours... will resume tomorrow AM.
• 20250903-1155: work suspended - primary job
• 20250903-1130: POP3, IMAP, and SUBMIT protocol answer, but I need to write a custom auth module to use them.
• 20250903-0729: resolution to ESMTP encryption issues
• 20250903-0700: restore-from-backup appear done: testing underway.
• 20250903-0120: began data restoration from email data backups
  • we've made some changes to the way disk storage works; what is goign on now is temporary (no disk quotas). Quotas will be implemented
• 20250903-0052: mail delivery functionality working without errors.
• 20250903-0020: email receipt integration appears complete.
• 20250902-2018: network encryption updates in place: shouldn't give more browser warnings on open.
• 20250902-1940: initial version of this webpage is created.
• 20250902-1820: confirmed hard failure due to worst possible kind of failure, storage controller.
  • All disk images are likely corrupted at the primary site.
  • Off-site data backup images are not corrupt; backups definitely ran on 9/1, may not be complete for 9/2.
  • rebuilding will likely go faster than trying to restore services at primary data server; outage likely to exceed 24 hours.
• 20250902-1725: second hard crash of data center services; this time, we got hardware diagnostic data.
• 20250902-1640: after on-site maintenance at the data center, we (only temporarily, it appears) restored services. Left site with external confirmation of normal functionality at 16:56.
• 20250902: between 11:00 and 15:00, the primary data center services went down. Departed home office for data center at 15:20.

We apologize for any inconvenience you may experience, and are working to restore services.