NOTA BENE: IO is proud to be a member of Mayfirst which provides IO server space its website, emails and list servs!
Hurricane Sandy ravaged large parts of the northeast United States the end of October.
As everyone who attended our membership meeting knows, the hurricane’s arrival forced us to end our meeting early, as New York city residents rushed home before the subways closed.
Hundreds of thousands of people were affected by the storm. Some, over three weeks later, are still without electricity, heat or even their homes.
A crisis like this reveals both the shortcomings of our society as well as the strength of our communities. On Long Island, the electric company is being sued over gross negligence in their preparation and handling of the electric grid after the storm.
On the other hand, May First/People Link member InterOccupy, via the Occupy Sandy Web site, has demonstrated that the Occupy movement is not only still kicking, but is demonstrating a model volunteer relief organizing effort that is inspiration to all.
Getting high profile media attention, the site has been inundated with traffic as thousands of volunteers have used it to organize the most prominent relief effort of the region. Kudos to the MF/PL Support team members who worked day and night keeping the site running smoothly.
Although politically, we are a membership-based activist organization, we also provide a critical infrastructure to our membership. With servers based in New York City, we also were affected by the storm. Most of our servers are located in two colocation facilities in the Chelsea neighborhood of Manhattan: Telehouse (10th Avenue and 15th Street) and XO Communications (9th Avenue and 15th street).
At approximately 11:15 pm local time, Monday, October 29th, we lost contact with our XO Communications facility. We reported the outage immediately, but spent hours waiting for a confirmation on the problem. Finally, at 4:40 am, we received notice that the outage was due to flooding in the basement of the building housing our servers. The building, which is owned by Google, suffered building-wide electrical damage as a result of the flooding. Although XO Communications kept their generators running, the damage to the building’s electrical system prevented the electricity from reaching our servers, resulting in the servers being powered down.
At 8:15 am, power was restored and the storm had passed, however, our servers were still down due to our security policy.
The reason? In order to preserve our members’ privacy, our servers refuse to boot unless a disk-decryption passphrase is entered. We can restart servers on an individual bases remotely, by entering the passphrase via a running server. However, if all servers are turned off, a trusted support team member must go in person to boot strap the process. This additional protection prevents government or other agents from accessing our members data by removing a server from our cabinet.
With reports that no cars were being allowed over the bridges connecting Brooklyn (where most of our support team lives) and Manhattan (where the servers are located), one tech team member was dispatched by bicycle to enter Manhattan and properly start our servers, which fully came online by 11:00 am.
Our second location, Telehouse, suffered an outage as well from about 6:15 pm local time to 11:15 pm on Wednesday October 31. Thanks to the careful planning and preparations made by Telehouse, we did not lose power to our cabinet. Instead, the company that provides our Internet connection lost power in their facility. Therefore, as soon as their facility had power restored, our connection was immediately re-established without the need to dispatch a support team member.
We’re proud to have kept our services running as well as we did, but believe we could have done better. The support team is currently discussing a number of projects that will improve our uptime in the event of another storm, including battery-powered backups for our disk-decryption servers and multiple connections to the Internet in each cabinet in case one connection goes down.
Our hearts go out to those still suffering from the after match of Hurricane Sandy, not only in New York and New Jersey, but Cuba and other Caribbean countries. In addition, we stand with everyone who is suffering from the consequences of similar storms around the world, such as Katrina here in the United to the Niger Delta flooding from early October of this year.
As a political organization, we’re committed not only to maintain and strengthen our shared infrastructure in the face of more and bigger storms, but also to join in the global movement against human contributions to global warming and to help build a society better able to respond to communities ravaged by these storms.