2003
 Tuesday, December 02, 2003
Bizgrok mail server was down until 09:40 today. Only services directly affected were customer outbound SMTP and mail for downcity.net.
Sunday, November 02, 2003
Bizgrok & North East Digital Village Email and Bizgrok Web Access: There was a complete outage due to widespread problems with SNET around the state between aprox. 3:40 PM and 10:20 PM today. Some customers reported intermittent problems before that. This outage affected access of customer web sites and retrieval of customer email, but not other services.
No mail was lost. We have backup mail servers off-site that receive the mail while the main server is unreachable. The messages are in the process of being delivered to your mailbox and should be there before midnight.
We apologize for any inconvenience. We will continue to migrate legacy services to more reliable networks to avoid future problems due to SNET. (Our name servers and authentication servers as well as the backup mail severs are already located in New York and California.)
Thursday, August 14 & Friday, August 15, 2003
Dialup Access, Email, Bizgrok & North East Digital Village web access: As most of you are likely aware, at 4:15 PM EDT yesterday, a series of cascading grid failures caused a major power outage in the northeastern US and into the Canadian province of Ontario. This was a total and complete loss of power across over 9300 square miles stretching as far west as Detroit, Michigan. The consequenses of such a massive grid failure include loss of telecommunication services. However, domains and web sites hosted by Bizgrok were not affected as customer web servers, located in Connecticut and California, were maintained on battery backup during the brief loss of power experienced. Local power continued uninterrrupted throughout this major blackout.
The decentralization of telco transmission systems means that there is quite a large number of SONET hubs, muxes, repeaters and other fiber transmission gear deployed in environments lacking backup power sources. When power went down in the New York metro area, our upstream provider's lost all [redundant] links, effectively isolating their data center and affecting the ability of some dialup customers to authenticate (customers already online were not affected, only those who tried to connect while the RADIUS server was unreachable). In addition, backup power systems suffered partial failure [being investigated for cause/remedy].
Although power and then connectivity was restored by approximately 9 PM EDT, there was another outage between 12:30 AM EDT and 2:30 AM EDT. Service is now restored. Howver, sporadic outages are still possible over the next 24 hours as electric service is restored across the other outage areas.
Wednesday, August 6 2003, 3 AM - 6 AM
Dialup Access Only: Emergency maintenance will occur August 6, 2003 from 3AM-6AM EDT to correct security flaws reported by Microsoft and Cisco. Dialup customers may experience brief (up to two minutes) outages during this time.
Completed as scheduled.
Thursday, July 24, 2003, 05:33 AM
Dialup Access Only: 05:33 AM to 12:56 PM Loss of Routing may have been experienced due to major outage with our upstream provider. Althought first reported that replacement exquipment would not be installed until 7 PM EST, alternative means to resolve problem was found and routing to the Albany POP was restored at 12:56 PM. Users will now be able to authenticate and route without any problems. Thank you for your patience.
Tuesday, July 22, 2003
Dialup Access Only: At 7:00 AM EST emergency maintenance on our upstream provider's backbone circuits connecting Eastern and Central regions will be performed. This maintenance should not take longer than 15 minutes. Dialup customers may experience increased latency during the maintenance window. It is not expected that this maintenance will cause a loss of connectivity.
Monday, July 21, 2003
Dialup Access Only: Approximately 11:50 AM EDT - currently experiencing busies and operator intercepts due to yet-to-be diagnosed problem with telco.
13:35 Resolved. All service restored.
Sunday, June 22, 2003:
Dialup Access Only: Approximately 00:49 AM EDT - loss of routing due may have been experienced due to our provider's Emergency Maintenance/Upgrade to equipment.
01:11 Resolved
Wednesday, June 4 2003
Dialup Access Only: This afternoon at approximately 12:15 PM EDT, a pair of internal distribution breakers in our primary supplier's data center tripped, putting several racks on battery power. Alarms did not catch this condition quickly enough, and several batteries ran down before power was restored. This caused a loss of connectivity for some customers.
It appears that the pair of breakers in question have failed, so power was re-routed to the racks in question to an alternate power feed. In doing so, one of the aggregation switches went down causing a second brief loss of connectivity to some customers. Engineers will continue to work to determine the cause of this problem, and to take the appropriate steps to ensure that does not become a recurring issue. Please accept our apologies for any problems this may have caused, and thank you, as always, for your continued support.
Tuesday, May 27 2003
Dialup Access Only: Approximately 04:00 AM. Loss of Routing in Western MA & New Hampshire.
Resolved 06:30 AM
Wednesday, May 21 2003
Dialup Access Only: Power outage to regional colocation center may have caused intermitten problems with local dialups this evening beginning approximately 5:10 PM. Problem resolved: 8:05 PM.
Friday, May 16, 2003
Email Only: Mail stopped being transferred to primary mail server around 4:45 PM and the problem was corrected around 2 AM. No mail was lost; the backlog of mail is still in the process of being transferred but should finish this morning.
Sunday, March 30, 2003
Dialup Access/Email Only: Our upstream provider was attacked today by multiple computers on the Internet. During the attack, a large amount of traffic was generated causing serious congestion. While not a complete outage, end users may have experienced interruptions in service if their authentication packets were lost, and causing downstream customers to see traffic sourced from irregular IP addresses. On 03/31/03, emergency maintenance was performed from 4AM to 5AM EST with the installation of new intrusion detection system completed without interruption in service.
Thursday, January 30, 2003
Email Only: SNET routing problem caused outgoing email delivery to be delayed today. While no notice was received from SNET, the problem appears to be fixed. The outage time: 1022 to 1315. We have cleared the backlog of 500+ emails and customers have verified that mail is working now.
Saturday, January 25, 2003
General Alert: A new "worm" targeting a vulnerability in Microsoft SQL servers was unleashed on the Internet starting at approximately 12:30 AM eastern time on Saturday. The worm, dubbed the "SQL Slammer" attacks servers running Microsoft SQL server (version 7 and SQL 2000), infects these machines, and causes them to seek out other SQL servers to infect, generating huge amounts of erroneous network traffic in the process. While Bizgrok's upstream providers were affected, none of Bizgrok's machines were directly affected, since they run BSD UNIX rather than Windows.
From one of our providers: ". . . To add insult to injury, the effect of SQL Slammer on the global Internet was far worse, bringing down several major Internet backbones, including those to which we maintain connections. As such, while we had our local network running again by 5:15 AM, it was almost 9:30 AM before reliable connectivity in and out of the network was restored.
Microsoft has made patches available for SQL 7 and SQL 2000 that close the security hole exploited by SQL slammer. . . We urge all customers to be vigilant with regard to upgrades and patches. Security on the Net depends, to a great degree, on [Windows] system operators taking responsibility for running fully patched and upgraded systems."
Wednesday, January 22, 2003
Dialup Access Only: Some of our dial-up customers are experiencing intermittent issues with mail due to problem of our upstream provider. We are working on this issue. We currently have an ETR of 1100 EDT. Thank you for your patience.
2002
Friday, 20 December - Sunday, 22 December 2002
Dialup Access Only: Intermitten problems experienced by some users due to upstream provider router maintenance. All problems should be resolved by 1 PM E.S.T.
Tuesday, 17 December 2002
Dialup Access Only: This morning at approximately 9:30 AM eastern time, our dial-up customer/s may have experienced problems. Our upstream provider experienced trouble on their primary AT&T DS-3 transit link. It appears that AT&T was "black holing" them by dumping inbound and outbound packets in New York City.
Friday, 01 November 2002
Dialup Access Only: Working with the local Telco to build out Connecticut POPs that have been experiencing busy signals. Solution should be in place in the next thirty days.
Resolved as scheduled
Monday, 28 October 2002, 09:00-11:00 AM
Dialup Access/Email/Web: Unscheduled intermittent electrical power outages cause brief service interruption/server reboot.
Thursday, 24 October 2002
Email Only: If you continue to experience email problems, please reboot. If this does not solve the issue, for the quickest response, please email support@bizgrok.com.
Wednesday, 23 October 2002
Routing Problem: Bizgrok experienced a routing problem upstream that caused loss of connectivity today. The server has been moved to a new location and was up and running at approximately 19:30. It might take a couple more hours for local DNS servers to get the new IP. Sorry about the inconvenience.
Monday, 21 October 2002
Distributed Denial of Service Attack: A "Distributed Denial of Service (DDoS)" attack was detected by Matrix NetSystems monitoring locations in North America. This attack targeted the 13 root DNS servers that provide worldwide address translation for the entire DNS network. This attack began in earnest beginning at approximately 4:00 pm EDT. Several Internet Service Providers (ISPs) have seen an increase in latency coincident with this attack. These attacks resulted in periods of zero reachabilty for many of the root servers and an increase in packet loss approaching 10% from normal packet loss levels of less than 1%. At the peak of the attack, average reachability for the entire DNS network dropped below 94% from normal levels near 100%. This attack is clearly targeted at a crucial piece of the Internet infrastructure.
Monday, 14 October 2002
Switching problems interrupt service: 12:30 Effected by some instability in upline core switches that escalated into an large-scale hardware issue, and later software issue for upstream provider. Resolved by 15:30. Everything has been patched and is working at this point. If you notice that something is still not responding, please contact us at support@bizgrok.com.
Thursday, 3 October 2002
UUNET hit by outage
UUNET experienced a major service outage today interrupting Internet traffic on several parts of the WorldCom IP backbone network. IP traffic may have been constricted.
Monday, 23 September 2002
All Services: Lengthy electrical power failure in New London caused temporary lose of smtp server. Service restored 11 AM.
Monday, 16 September 2002
General Alert: 18:41:21: We are currently experiencing operator messages in Connecticut. Engineers currently investigating this issue. 22:25:47: Situation fully resolved.
| 
