Server Issues, Part 2

May 8, 2010 at ~6 PM
It turns I had not fixed the problems as I'd originally thought.  The database and server were down from around six o'clock Saturday evening through around mid-day Sunday afternoon as a result.

May 9, 2010 at ~2 PM
The system should be back online and functioning correctly this time (really).  I'm very sorry!

May 10, 2010 at 4:39 PM
Please let me know if you find that the system is down again.

Stats Being Tracked, but Reports Unavailable

May 7, 2010 at 7:12 PM
BlogPatrol is currently tracking stats for users, but an issue is preventing the stats reports from displaying properly.  We are looking into this issue and will post an update as soon as possible.

10:50 PM
BlogPatrol is functionally again.  I apologize for the inconvenience that this downtime caused everyone.

Users have reported stats issues as of early March 14

March 14, 2010 @ 11:06 PM
BlogPatrol users have reported that stats are not being counted and/or that discrepancies are being shown among their stats reports.  We are looking into this issue as of this blog post and will provide an update soon.

March 15, 2010 @ 9:18 AM
After much troubleshooting and problem solving, we have been able to resolve the issues affecting the system.  Everyone's BlogPatrol accounts should be functioning properly again.

March 15, 2010 @ 4:55 PM
We've received additional feedback that there are still problems happening with the stats reports.  We are investigating the issues further and will post additional information when it is available.

March 16, 2010 @ 4:36 PM
The system appears to be working correctly now.  The bad news is that:

• BlogPatrol was not able to record any stats from ~19:30 PM (UTC-4) on March 13 until ~01:15 AM (UTC-4) on March 15.
• Also, new accounts created during this time period will need to be recreated because the system was not functioning properly.

We are still monitoring the server closely to see if any other problems arise.

BlogPatrol Reconstruction Complete!

February 15, 2010: I'm very pleased to announce that our project is complete and 100% successful!  Not only are all the features of your BlogPatrol account functioning fully for you now, but we have released major upgrades to the service, including:

• A new architecture that allows us to do system maintenance with minimum downtime.
• The new architecture includes new hardware that provides a faster website for you.
• The ability for us to partition your historical data so that we no longer have to purge (delete) it from your account.  Using the Previous and Next links on your stats reports, you can scroll through your data and go back as early as December 27, 2009.  We anticipate being able to store at least twelve months of data using our current system design and hardware.
• The Detailed Analysis of the Last 50 Visitors report has been upgraded dramatically to the Detailed Visitor Logs.  The upgraded reports allow you to view the details of any visitor that we have recorded in our database.  You can scroll through all your detailed logs using the Previous Day and Next Day links.  The detailed logs are displayed using pages that list up to 50 visitors per page.  Pagination links allow you to scroll through the details of the day.

February 7, 2010: The BlogPatrol reconstruction project is nearly complete... We have successfully achieved our goals of reconstructing the server architecture for faster performance and, more importantly, partitioning historical stats.  This means that you now have a longer history of stats available to you and that you will be able to view your stats more quickly .  We are monitoring our system to see if any final tweaks are needed to further optimize performance.

January 30, 2010: Most major tasks have been completed in the reconfiguration of the server for speed and reliability.  We are starting to work on the part of the project that, hopefully, will allow us to provide historical stats.  (We currently have to purge stats forever after about 4 weeks. Otherwise, the database would grow too large and cause the system to operate too slowly.)  The partitioning project is very complex, and we cannot guarantee that it will be achieved.  We will, however, give it our best shot!

January 25, 2010: As of January 24, we have resumed the reconstruction project.  Please expect down times, both intermittent in frequency and extended in duration, until we are able to post an update on the next milestones completed.

January 19, 2010:  We are expecting the project to resume after January 21, when the schedule for our project team becomes available again.

January 18, 2010:  BlogPatrol is currently, intermittently back online.  Please expect slow site performance from time to time until the reconstruction project resumes.

January 17, 2010:  The site is currently down.

 January 8-21, 2010:  Due to scheduling constraints, there will be a break in the reconstruction project until after January 21.  At this time BlogPatrol is online, running a fast SSD drive, and benefiting from a database redesign that offers a single master for recording stats and a single slave for generating stats reports.  Until the reconstruction project is finalized, however, the site may still function slowly at times.

January 7, 2010:  The stats reports were unavailable this morning.  As of 2:30 PM Eastern, the system and stats are now back online.

January 5, 2010:  Completed upgrade to MySQL 5.1.  Completed some configurations needed for multi-master/slaves design.  Undergoing testing to make sure system changes are functioning properly.

January 4, 2010:  The reconstruction project is still in progress throughout this week.  The website and stats reports may at times perform more slowly than normal -- or be unavailable for extended periods -- while the project is being completed.  We will continue to post updates as project tasks are completed.

January 3, 2010:  Had problems with stats reports between Jan 1-3. To resolve the problems, backed up December 2009 data, purged the database, and resynced master/slave databases.

January 1, 2010:  Happy New Year 2010!  We have completed the reconstruction's first tasks: initial conversion to a virtual server environment and deployment of the first of multiple slave databases.  Slave databases are intended for the purpose of displaying your stats more quickly.  The second phase will occur next week, which will include the deployment of additional slave databases and an upgrade from MySQL 5.0 to 5.1.

December 31, 2009:  BlogPatrol currently is offline while we are reconstructing the server for better performance in 2010.

December 30, 2009:  We've started the reconstruction process for BlogPatrol.  We will be performing major upgrades on BlogPatrol.com that will allow for greatly improved speed and functionality in 2010.  We apologize for the down-time that will occur as a result of this work, but it's short-term pain for what will be long-term, worthwhile improvements.

Improvements:

• We will be upgrading our hard disks to include several solid-state drives, which operate faster than SATA, SAS, or SCSI drives.
• We will be going from a single instance database to a multi-master/slave database design.  Having a multi-master design means that it will be easier to keep BlogPatrol online and tracking your stats -- even when maintenance is being done on the system.  In a single instance database, maintenance usually required going offline completely.
• Having multiple slave instances of databases means that BlogPatrol means that you will be able to access your stats reports more quickly.
• We will be upgrading to MySQL 5.1 which will provide the basis for segmenting data into historical partitions.  If we were able to utilize this capability, then BlogPatrol would be able to retain your stats as archives that you can access, instead of having to purge (delete) them from the system.

Database Maintenance

November 9, 2009
BlogPatrol.com will be down and not recording any stats during this maintenance period.
2:50 PM UTC-4 (America/New York) - Performing maintenance to verify database integrity and optimize database performance. Purging records down to 14 days to help website and database perform faster for users.
4:59 PM - Doing a second round of purging to get data down to 8 days to help website and database perform faster for users.

5:25 PM - Completed database maintenance activities.  BlogPatrol is online and recording stats again.

Database issue preventing BlogPatrol logins

November 3, 2009
7:35 AM UTC-5 America/New York: We are aware of a database issue that is preventing users from being able to log into their BlogPatrol accounts. We are working to fix the problem and will post an update when the fix is complete.
12:48 PM: We have resolved the issue. BlogPatrol is online and functioning normally again.

Scheduled Hardware Upgrade

October 10, 2009
BlogPatrol.com will be offline and therefore not recording any stats during this upgrade.
11:37 AM UTC-4 (America/New York) - The BlogPatrol server will be offline later today in order to complete a hardware upgrade. The purpose of the upgrade is to replace a 15-month old 10,000 RPM, 300 GB SATA Hard Drive with a brand new 15,000 RPM, 300 GB Serial Attached SCSI (SAS) Hard Drive. The new SAS drive will help BlogPatrol to operate faster than the older SATA drive.
4:09 PM - The upgrade is presently underway. We will post an update when the activity is verified as complete and successful.
7:48 PM - The upgrade completed and successful. BlogPatrol is improved and back online.

Database Maintenance

October 8, 2009
BlogPatrol.com will be down and not recording any stats during this maintenance period.
11:50 AM UTC-4 (America/New York) - Performing maintenance to verify database integrity and optimize database performance.
12:35 PM - Initial database check completed. Purging records now down to 8 days to help website and database perform faster for users.
4:36 PM - Completed a partial database purge earlier this afternoon. Resuming purge now.
8:45 PM - Cleanup from the purge is still underway. I'm sorry this is taking so long.

October 9, 2009
12:33 AM - Finally the maintenance is done. This was a tough day with the maintenance taking so long. We are back online though, and I'm glad for that.

Database Maintenance

September 19, 2009
BlogPatrol.com will be down and not recording any stats during this maintenance period.
5:54 PM - Performing maintenance to ensure database integrity.
8:40 PM - Database required repair, which has been completed. BlogPatrol.com is online and counting stats again. We apologize for the inconvenience, and thank you for your patience.

Database Maintenance

August 31, 2009
BlogPatrol.com will be down and not recording any stats during this maintenance period.
8:50 AM - Performing database maintenance script to purge detailed stats down to 8 days.
2:00 PM - Completed database maintenance. BlogPatrol is now back online and tracking stats.

Bad Friday - Blackout, then Database crash

August 15, 2009 at 6:06 PM
The database is restored, and we're back online now. We did not track any stats from Friday evening (Aug. 14 circa 5 PM) through Saturday afternoon (Aug. 15, circa 6 PM).

I'm going to be making some changes so that I know what's going on right away when these disasters happen.

August 15, 2009 at 4:51 PM UTC-4 (America/New York)
I'm sorry folks. For a few hours on Friday evening, there was a blackout in the area where our server is located. The blackout caused a database crash which I only discovered this afternoon. I am currently working to restore the database.

Database Keeps Breaking

July 27, 2009 UTC-4 (America/New York)
3:42 AM - The database appears to be stable again. If you see otherwise when checking your stats, please let me know.

July 26, 2009
2:18 PM - Completed purge and database seems to be working for now. Will keep checking in.
08:46 AM - The database seems to keep breaking because the demo stats are saying "The information you have requested is not available at this time." every time I check on it the public BlogPatrol site . This is unacceptable. I'll try to trim the database again this morning. I'll also ask the technical team to find out what's going on.

July 25, 2009
BlogPatrol.com will be down and not recording any stats during this maintenance period.
01:05 PM - Database completed. Will be monitoring the site to see if shrinking the database size has helped the performance.
11:40 AM - Database has been performing erratically the last few days. Performing database maintenance script to purge detailed stats down to 7 days.

Completed - BlogPatrol Offline on July 24, 2009

July 24, 2009 at 6:24 AM UTC-4 (America/New York)
The aforementioned upgrades are complete, and BlogPatrol.com is back online.

July 23, 2009 at 2:24 PM
The Internet service provider for BlogPatrol.com will be upgrading its fiber network on Friday, July 24, 2009. The upgrade will require BlogPatrol.com to be offline between 12:01 AM and 6:00 AM EST on July 24.

The BlogPatrol website will be unavailable until the upgrade is completed and DNS changes to www.blogpatrol.com are propagated throughout the Internet.

Resolved - Annoying Pop Up/Under Ads

8:10 AM UTC-4 (America/New York) on July 19, 2009
I recently resolved the issue of the annoying pop up ads, which were appearing only on the login.php page. It seems that the rectangle ads placed under the user info are designed by one of my ad publishers to stay above the fold of the web page. When users are logged in, the information sometimes pushes that ad location beneath the fold. This layout caused a set of the rectangle ads to appear as if they were pop-up ads because they would forcibly appear above the fold.



I have replaced those types of ads from that publisher, and after a couple of weeks the problem seems to have been resolved. If you see otherwise, please let me know via comment on this post.

2:04 PM on May 22, 2009
I just received feedback from a BlogPatrol user that they are getting pop up/under ads when they log into BlogPatrol:

I'm getting a popup ad when I log in that is blocking my ability to see my stats without clicking the ad, when you click the X it goes to the ad itself rather than closing the window. Not appreciated....

I'm really sorry. As stated on our FAQ page and privacy policy, we do not condone pop up/under advertising. Therefore, I never give permission for such ads. Unfortunately, the third-party ad networks that I use sometimes publish them anyway. When this happens, I have to manually disable these specific ads (or sever ties with the ad network, which I have done on more than one occasion). I'm currently looking into which of our ad networks is publishing pop up/under ads so that I can disable those ads immediately.

I try to do my best for BlogPatrol users within my part-time means, and I am sorry that these ads have caused any discomfort or inconvenience. Thank you so much for your feedback, which helps to keep BlogPatrol friendly and useful for all users.

BlogPatrol Fixed

July 12, 2009
8:08 AM EST (New York): Database repair is complete. Historical and current stats are restored. BlogPatrol website and database are back online. User log-in capability is restored.
6:57 AM: Running database maintenance repair script because the manual reboot caused the database to crash.
3:45 AM: Manually power cycled the server to force a system reboot because remote access to server was unresponsive.

July 11, 2009
9:07 PM: There is a problem that is preventing BlogPatrol users from logging in, or staying logged in, to their BlogPatrol account(s).

Unfortunately, BlogPatrol also has not been tracking stats since about 4 PM Eastern today.

We are working to resolve this issue as soon as possible. We will provide updates as they become available.

Database Maintenance

July 9, 2009
8:23 AM - I'm noticing that the site is still sometimes unresponsive. At these times, the website displays the message "The Information you have requested is not available." Feel free to post a comment to let me know that you are still seeing this message a lot today?
8:08 AM - Maintenance is complete. BlogPatrol.com is back online and recording visits again. BlogPatrol did not record any stats from approximately 11:30 PM July 8 through about 8:00 AM July 9.
6:24 AM - Database maintenance is taking a long time. I'm still working on it...

July 8, 2009
BlogPatrol.com will be down and not recording any stats during this maintenance period.
11:36 PM - BlogPatrol users have not been able to log in to their accounts or view their stats. Performing database maintenance script to purge detailed stats down to 14 days.

Database Maintenance

May 2, 2009
BlogPatrol.com will be down and not recording any stats during this maintenance window.
1:21 PM - Received messages about BlogPatrol counters not working. Running database maintenance script.
1:49 PM - Maintenance is finished. BlogParol.com is back online.

May 1, 2009
BlogPatrol.com will be down and not recording any stats during this maintenance window.
11:57 AM - Performing database maintenance script.
12:42 PM - Running script to purge detailed stats down to 14 days.
03:42 PM - Maintenance is finished. BlogParol.com is back online.

BlogPatrol database maintenance

April 27, 2009
9:52 AM - Performing database maintenance to purge detailed stats down to 28 days.

3:07 PM - Database required re-indexing. May have to attempt the purge again later today.

BlogPatrol database maintenance

March 27, 2009
7:35 AM - The database is still responding slowly to user requests. Running another set of maintenance activities.

11:56 AM - The database seems to be loading much more quickly now. If anyone else is experiencing slow times for loading your stats, please let me know.

March 26, 2009
2:00 PM EST (UTC-4 America/New York) - BlogPatrol will be offline this afternoon so that we can conduct maintenance on our database. We will post an update once we are back online.

2:35 PM - Started database maintenance. BlogPatrol.com is offline and not tracking stats during this time.

9:25 PM - The database maintenance is complete. We have successfully purged records older than 21 days from the system. When the database gets too large, it makes the web site performs much slower. We have also run an integrity check on the database to make sure all is well with the system.

Verizon FIOS Internet Connection Down

March 4, 2009
10:00 PM EST (UTC-5 America/New York)
The Verizon Business FIOS service that provides BlogPatrol's connection to the Internet has been down since about 10 PM on Wednesday, March 4, 2009. Our website is currently offline, but we are in contact with Verizon representatives to resolve the issue. We will post updates as we obtain more information.

March 5, 2009
11:21 AM EST
No results from Verizon yet.

12:43 PM EST
Verizon has determined that the problem is a physical one occurring somewhere between Verizon's lines and our own network terminal. Verizon will be sending a technician to our physical location today before 5 PM local time in order to investigate the issue further.

4:33 PM EST
The Verizon technicians identified that the fiber optic terminal outside of our location died. They have replaced the ONT, and BlogPatrol.com is now back online.