|
Note: You can also follow @bizgrok on Twitter for up-to-the-minute, out-of-band information.
 Tuesday, June 24 2014
da2 down for hardware maintenance between 0208 and 0642 today. May plan additional maintenance later this week to replace a drive in the RAID array.
Monday, February 24 2014
Our outbound email was blocked by our upstream provider today, which delayed the sending of some outbound customer email. The situation has been resolved. It is worth noting that this was due to a customer who allowed their email password to be compromised, which resulted in our server sending spam for the first time in Bizgrok's almost-14-year history. (We receive many bogus spam reports, but alas this one was real.) This is a good time to think about both choosing a strong password and being careful to use encryption when on unsecured networks. (Better yet: avoid insecure networks, like most public WiFi, entirely.)
Monday, July 22 2013
Not an outage, but FYI: Some customers are reporting bounced email this morning due to a listing in the CBL for one of our mail servers. We have initiated a delisting, and will continue to monitor the situation, but this listing appears to be in error. We cannot control the behavior of other mail servers, who may choose to accept or reject email from our server for various reasons, but there appears to be no legitimate reason for blocking email from this server at this time. Update: Note that some DNS servers may have cached the DNS result and still be rejecting email even though it the listing been removed for hours now. Again, this is not something we can control.
Sunday, January 13 2013
We found a temporary glitch that was keeping new web site log files from being processed. Our server and currently working through the backlog and web site statistics should be updated soon. No data was lost. [Update: All log files and statistics are now current.]
Monday, October 29 2012
At aproximately 1934 EDT Hurricane Sandy managed to take out both fiber networks connecting the NY data center where our primary servers are located. Connectivity was restored by 1727 EDT on October 30, and the data center is also now back on utility power. Our backup DNS server, and some other servers, are located elsewhere in the country to guard against a disaster like this one, and they remained reachable the entire time. No customer data was lost. Thank you for your patience. [Update: As of November 1, both fiber providers are now back to normal.]
Monday, November 07 2011
Brief denial of service issue at our upstream provider at about 1907, Restored at 1919.
Saturday, October 30 2011
Experiencing network degradation (high latency) at our upstream provider starting around 2245. Appears to be resolved at 0150.
Saturday, September 03 2011
A sustained and agressive attack on our customer mail server began at 1254. We have reported this issue to the attacker's network and escalated it upstream. There is no outage at this time, but service might be slightly degraded while we work to counter this threat. Update: As of 1330 we have blocked the traffic and server performance has returned to nominal. Unlike Virpus Networks, we and our co-lo provider work weekends...
Sunday, August 28 2011
"Is there any point to which you would wish to draw my attention?" "To the curious incident of the dog in the night-time." "The dog did nothing in the night-time." "That was the curious incident," remarked Sherlock Holmes.
Monday, August 22 2011
Outage of customer mail server at 2000. Service restored 2122. Expect a brief outage later this week to replace defective hardware.
Tuesday, June 21 2011
Emergency reboot of our customer mail server at 1312. Brief outage expected.
Saturday, March 12 2011
The failure of a power distribution unit at our colocation provider caused an outage of our primary customer web server between 0637 and 0711.
Sunday, September 05 2010
Routing issues at our upstream provider causing packet loss. Customers may experience intermittent problems accessing email and web sites.
Friday, June 25 2010
Reboot of customer web server at 0934.
Friday, April 16 2010
Our customer web and email servers were unreachable approximately one hour between 1234 and 1334 due to routing problems at our co-location provider.
Monday, January 18 2010
An issue with one of our name servers caused some delays in DNS resolution. May have resulted in temporary delays of mail. Fixed at 0925.
Friday, October 2 2009
SNET DSL customers are reporting problems reaching our servers, as well as some other web sites, this morning. This appears to be a problem with SNET's network, but we will continue to monitor the situation. Update: As of 1315, this problem appears to be resolved.
Monday, June 29 2009
Our content distribution partner lost statistics for requests between the morning of June 19 and this afternoon, June 29. There was no failure to serve requests, but web site site requests will be underreported for customers using the CDN.
Saturday, March 7 2009
Fixed a problem that was keeping some customer web site stats from updating. Your missing reports (if any) should be available by Monday.
Wednesday, December 17 2008
Two quick reboots of web server this morning.
Thursday, July 31 2008
Quick reboot of mail server at 1620. Back up at 1623.
Tuesday, June 24 2008
Quick reboot of mail server at 1400 to move to new power-distribution unit. Back up at 1405.
Thursday, June 12 2008
Web server performance sluggish around noon due to denial of service attacks from multiple networks. We continue to monitor and address the issue.
Thursday, June 5 2008
Our customer email server down from 1344 to 1347 to temporarily move to a new battery backup. Another short outage may occur later today to move it back.
Tuesday, June 3 2008
Our customer email server was rebooted at 0828 this morning; downtime was less than two minutes.
Friday, May 23 2008
Some SNET (aka AT&T/SBC) DSL customers are experiencing problems connecting to our mail server (and other, seemingly-random non-Bizgrok destinations) today. Please note that this appears to be an internal SNET routing issue, and Bizgrok's servers are not down. (We also note that our DSL service through an independent local ISP is working just fine and we are always happy to recommend an alternate provider for those who wish to switch from SNET to a more reliable service.)
Friday, April 11 2008
Some customers have experienced delays in delivery of email destined for users of Yahoo's email service. Please note that these delays are not due to any outage or misconfiguration of Bizgrok's email system (and similar delays are reported by operators of other mail servers). In general, we strongly advise our customers not to use Yahoo for their important email, both because of this issue and because Yahoo is a significant (and, we believe, irresponsible) source of spam.
Monday, March 31 2008
Update on the email server: we addressed some performance issues on our mail server that allow us to remove the workaround for Earthlink. (At the same time, we still think it would be better if Earthlink followed RFC 1123.)
Monday, March 24 2008
Some of our customers have been experiencing problems receiving legitimate email from correspondents who use Earthlink's network. After a few days of trying to find an actual human at Earthlink to deal with this problem, we gave up and implemented a workaround. This should solve the problem for those of you who want to receive email from Earthlink's network. (Unfortunately, it also means that the flood of spam from their network will likewise resume.)
Sunday, February 24 2008
A failure of the UPS at the data center took our customer web server temporarily offline between 1034 and 1208.
Thursday, January 24 2008
Our customer web server was temporarily unreachable between 1224 and 1238 because we did a dumb thing. We are very sad, because we were closing in on two years of uptime on that server and we spoiled it. Don't blame FreeBSD.
Saturday, June 30 2007
Our customer web server and mail server were intermittently unreachable between approximately 0300 and 0550 due to routing problems at our NY colo provider.
Monday, March 12 2007
Our customer web server was intermittently not responding to requests between 0135 and 0215 this morning when we encountered a bug while making routine software upgrades. The problem has been resolved.
Wednesday, February 28 2007
Our customer web server was briefly unreachable at about 0930 this morning because of a failure of the battery backup in the co-lo facility that caused the ethernet switch to reboot. Another brief (about 90 seconds) outage will occur when the defective unit is replaced. (The web server itself is now at 134 days uptime.)
Monday, September 11 2006
FYI, the issue with SNET's DNS noted below has not been resolved, depsite having been reported to SNET by multiple customers. Once again, this is not an issue with Bizgrok's servers. We have discussed this problem with SNET's technical support department and they have escalated the issue internally, but they have yet to address the issue. If you are a SNET DSL customer and your emails are marked "may be forged" or your emails are rejected by non-Bizgrok servers as spam, SNET's misconfigured and/or broken DNS servers may be the cause. We are happy to discuss the issue with you should you need clarification. We are also happy to recommend an alternative DSL provider should you wish to get service from a new company.
Saturday, August 19 2006
For over 48 hours now, some AT&T SNET customers have been experiencing intermittent problems connecting to our FTP server to update their web site. This is not an issue with Bizgrok's server. It is due to a misconfiguration or bug in DNS for adsl.snet.net, and possibly other domains served by ns2.snet.net. We reported this problem to AT&T over 48 hours ago, but have not received a response. Two SNET customers have attempted to resolve the issue with SNET by phone, but SNET's support technicians appear not to understand the nature of the problem. We will continue to attempt to educate SNET about this issue and welcome your calls if you experience this problem. In the meantime, we are implementing a temporary workaround.
Saturday, July 15 2006
01:30 (07/16/06) Delayed mail has been streaming in. It will take a while for the backlog to finish trickling in, but we look to be up and running again
00:50 (07/16/06) Old mail has been retrievable.
20:40 Dialup is new working
04:25 Server Outage affecting Mail & Dialup:
Our server outage began approximately 04:00 this morning. After initial attempts to revive the server, we have had to declare it dead. We are currently working on recovering and will post any further news or updates as to our Estimated Time of Recovery as soon as possible both here and on our home page. To the best of our knowledge, no customer email has been lost.
As our customers know, we pride ourselves on our uptime and quick resolutions to problems that do arise. You can expect that we will apply to same diligence to recovering from this server failure today. Thank you for your patience and understanding.
Thursday, April 27 2006
Problem with the RADIUS server this morning; folks couldn't get online from around 07:30, fixed at 08:40.
Monday, March 06 2006
A disk failure caused unreliability in our mail server and inability for some dialup customers to log in off and on over the weekend. The data has been moved to a new drive and service restored to normal; a planned migration to a new server has been accelerated and should be completed this week. One customer lost data and has been notified; we are not aware of any other permanent loss of data, though some email may have been delayed (resulting in automated notifications to the sender).
Thursday, December 09 2004
Due intermittent disk failures in our main customer mail server over the past 36 hours, all affected mail accounts have been migrated to our backup mail server as of 05:05 this morning. No loss of data occurred.
smtp.bizgrok.net was also re-pointed at our backup server starting last night. Please allow up to three hours for DNS changes to propagate to servers around the Internet. (If you're running MS Windows, rebooting may help.) If problems sending or accessing your email persist past 08:15 AM, please call us.
Wednesday, December 08 2004
Due to hardware failure, one of Bizgrok's mail servers is experiencing intermittent outages. Mail is being received and no mail is being lost, but delivery may be delayed. Maintenance is scheduled for later this evening. Problem should then be resolved.
Wednesday, July 21 2004
Bizgrok's customer web server was unreachable from 00:03 to 00:23 this morning due to another SNET outage.
SNET's outage managed to take down Bizgrok's SNET connection and our new alternate connection currently under testing.
Wednesday, July 14 2004
Early morning web server reboot needed Tuesday, July 13 and again, today due to failure of SNET router. Bizgrok's move of web server to alternate provider should be finalized soon.
Monday, June 29 2004
08:35 AM Email server attempting to take summer vacation. Was caught by Brindle and returned to service. Resolved 8:55 AM.
Monday,April 05 2004
Today's outage was due to the failure of some SNET equipment. It affected access to some customer web and FTP sites. Started aprox. 10:20 AM, completely resolved 2:03 PM. We apologize for the interruption in service.
Monday, March 15 2004
Between 20:35 Sunday night and 09:51 this morning, emails to many of our customers were rejected as spam when they should not have been due to a combination of an error in our configuration and the behavior of a particular anti-spam blacklist. The error has been corrected. We apologize for the interruption in service.
Friday, February 06, 2004
Outage affecting only the customer mail server (hustle) from approximately 1400 to 1600.
Monday, January 05, 2004
Outage on Bizgrok SNET connection from 0925 to 1006 today. Only services directly affected were customer outbound SMTP and mail for downcity.net.
Maintenance & Scheduled Service Interruptions
Thursday, August 1, 2013
Many web sites were transferred to a new, faster web server last night; many more are in progress today. We will be in touch to get your set up accessing your sites on their new server (and much, much more!). As always, feel free to call, though, if you need something done right now.
Wednesday, July 31, 2013
As scheduled, our old mail server (hustle) will be removed from service. All customers affected have received multiple notices. If your email mysteriously stops working around midnight (it's not missing, just not where you're looking), check your old unread email and then give us a call.
Sunday, July 28, 2013, 9 PM
da1 server will be taken down for a scheduled move to a new data center. Accounts hosted on that server will be unavailable for less than an hour. A backup mail server will be put in place to handle incoming mail smoothly during the downtime. Upgrade is complete. Mail transferred from backup.
Sunday, July 21, 2013, 7 PM
da1 server will be taken down for a scheduled hardware upgrade. Accounts hosted on that server will be unavailable for less than an hour. A backup mail server will be put in place to handle incoming mail smoothly during the downtime. Upgrade complete at 8:10 PM. Mail transferred from backup.
Friday, August 10, 2012, 12 PM
Scheduled reboot of hustle and charleston will interrupt some email and web services briefly.
Friday, April 23, 2010, 8 AM
WEB SERVER ONLY Customer web server will be down for less than five minutes due to maintenance at our co-location provider.
Thursday, January 08, 2004, 4 AM
DIALUP ONLY Upgrades being performed. May cause 5 to 15 minutes of downtime.
Thursday, December 09, 2004, 4 AM
MAIL SERVERS Emergency hard drive replacement performed. One mail server experiencing intermittent outages. No loss of data occurred.
| 
