Half a maple tree decided to break away on Diamond Ledge yesterday, 5/31, and caused a several hour outage for clients using the Diamond Ledge hub. Picture here.
After 3 months of being flat on its back, the Little Young access point was resurrected and permanently guyed. Thanks to Bob and Brad Streeter, Declan O’Connell, Jon Peaslee, John Enright, Ben Bullard and Cathy Graham for getting it up. Here’s a Windows Media Movie of the process.
The 2.4 GHz MIMO access point on Red Hill which serves about 20 clients disappeared in the early morning hours. Hiked up in snow like cupcake icing not knowing what to expect. Turned out the antenna mast had snapped off right above the top bracket and the whole upper assembly was hanging on by its ground wire. Repairs were made on that as well as the radome on the 5 GHz MIMO backhaul radio linking to Great Hill which had been smashed by falling ice and the 2.4 GHz panel antenna which had been blown 15 degrees out of alignment. It was a bad winter for ice and wind at that altitude. Here is a picture of before and after.
Three trips up Brown Hill to get the gremlins out. In case you were wondering what it’s like to maintain a wireless network in the winter, I put together a short video that pretty much sums it up.
The Wicked Wind of the West blew down the Little Young Mtn. tower this morning at 9:35 AM. The Rohn tower had taken everything Mother Nature threw at it for 7 years, but the freight train blasts out of the SW this morning finally did it in. Amazingly, the solar panels survived, but the baseplate was severely mangled. Pictures of the fallen tower are here, here and here. We will rebuild it, but this time of year it’s difficult, so I think it will be at least two weeks before it can be put back into service.
Brown Hill finally has a new and nicely functioning access point, thanks to Pete Hoag, Declan O’Connell and some clever semi-retro Norwegian engineering. The early January weather had been a big hold up, but we got a couple decent days and jumped on it. We actually ended up installing it twice because the cable had pulled out of the radio at the top of the mast the first time due to extreme wrestling with a floppy mast. 2” at a time with a come-a-long. What fun. As they say, “A good day in wireless is when nobody gets killed”. Here is a picture of the new mast and antenna in place. The old omni dating from 2003 is to the right. Here is a picture taken from the West side looking East. Hey- if the clients are happy, we’re happy and it it makes all these ordeals worthwhile.
After a frustrating 2 weeks for Diamond Ledge clients, the necessary repairs were done today by P.C. Hoag & Co. with their new MLE A93TDi Spider Lift. The pole mounted amplifier, which had been in continuous service, 24/7, since 2004 finally started dying. It was deemed unsafe to climb the tree to do the repair as that would have meant actually standing on the top of the tree with nothing to clip into other than the mast itself. Even Pete wasn’t up for that-to his credit. A picture of the operation is here. Click on the picture to enlarge it
You can never tell what you’ll find when you open up an antenna for repairs. Could be blackened electronics-could be...wasps. I only got stung once.
After a week and a half of daily trips up Little Young Mtn around 6:00PM to run the generator and charge the failing battery bank to get it through the night, 1500.00 of new AGM batteries were installed plus a new charge controller and some needed rewiring. The original gel cells had been in service since 2006.
Here is a sampling of equipment that was blown up in the latest round of lightning storms. It’s always a good idea to isolate any of your electronics from your house wiring before the storm hits.
Lightning strikes around Squam take out 4 radios, 2 switches, one power supply and one GFI outlet. Here are some pictures of the pine that was hit on Brown Point. This is what was left of the tree. The top 40’ was blown completely off. This is a picture of the trunk. Unplug your electronics before a storm hits.
Little Young goes down for unknown reasons. Cathy, Sophie and I ATV up part way with the help of tire chains and hike the rest with the help of carbide crampons. The cause of the outage was some unidentified rodent which had chewed through a power cable, disconnecting one of the batteries and blowing up the charge controller. His electrocuted carcass was nowhere to be found, but maybe the shock therapy cured him of his gnawing habits.
An online payment page was added to the website for your convenience.
Dennis, Declan and I hauled a new battery bank and backup generator up the Great Hill fire tower. 518 lbs. up, 520 lbs. down. Strong like bull, smart like tractor.
Neither rain nor snow nor sleet nor frozen ice pellets shall keep these messengers from their appointed rounds.
Climbed Red Hill to juice the batteries and de-ice the solar panels and antennas.
Here’s a picture of the Ghost in the Machine, likely one of your waypoints on the way to the world at large.
Here’s what the antennas and tower structure look like this time of year. I never cease to be amazed it works as well as it does.
Commercial radomes don’t exist for Ubiquity NanoBridge radios, and the one on Red Hill that makes the link to Great Hill turned out to be susceptible to icing and very low temperatures, so I had to make one from a livestock watering tub bottom. There was no snow on Red Hill today and temperatures were in the low 40’s, so it was an ideal winter day to install it and also swap out the XR9 900MHz AP radio.
Here is a picture of Declan, Dennis and me on a prior mission. We could have landed it.
Tracey Olafsen was on her way up to the tower as I was headed back down. She caught this great shot of the sunset after I left. Thanks Trace.
A new battery charge controller and backup generator was installed on Red Hill. Slushy snow on top of frozen leaves with freezing rain and sleet made this somewhat-well-challenging I guess you would say. I’m hoping that this will prevent any outages over the Christmas weekend, which is traditionally the peak of data usage on the network. I thought you might be interested in some photos I took today on my Droid camera. Here is one looking West over Squam from the firetower.
This one is looking Southeast over Winnipesaukee.
And this looking Southwest toward Meredith from the first grassy knoll(puff of smoke) a few hundred yards off the top.
Now that TamWireless and Cyberpine are joined at the hip and many Cyberpine clients are being routed to Great Hill and down to the Congregational Church in Tamworth and out newly installed fiber optic circuit, the Great Hill access point has become a critical hub in the network. On several occasions in the last two weeks, Great Hill suffered power outages which affected our network. We have installed a backup generator and new battery charge controller as well as replacing one of the two solar panel charge controllers we assumed must be malfunctioning. Hopefully this issue has been resolved. The coming dark days of winter will tell.
In case you missed the Cyberpine Annual Meeting(almost everyone did), here is a synopsis:
Declan O’Connell was elected to a 3 year term on the Cyberpine Board of Directors. Dale Mayer had served two consecutive terms on the Board, and I appreciate her contributions. She kept me posted through the years on national trends in the WISP world and forced me to pay attention to all that is Apple. My dim opinion of how Apple handles networking has not changed, but at least now I can troubleshoot MAC fanatic’s problems.
I have been working side by side with Declan for over two years now, and I can’t say enough about his newly learned abilities in administering very complicated wireless networks. The Board hounded me for years about tutoring someone to fill my shoes after I finally plunged to my death off an icy tower. I think Declan could actually do that now.
Many thanks to Pete Hoag, our reliable VP for being there in support at the drop of a hat. The most dangerous and complicated fixes invariably involve Pete. I really couldn’t do it without him.
And Cathy Graham, my partner and Cyberpine secretary/bookkeeeper. She keeps Cyberpine on an even keel and in the black as capably as she does the Town of Sandwich-even though it is a voluntary position with Cyberpine.
And many thanks to Bob Butcher for volunteering to be Cyberpine’s treasurer. Bob has already cast fresh eyes on the current state and future structure of Cyberpine’s billing system and I’m looking forward to his insight in the coming months.
Rant of the evening, provided by yours truly, concerned the continuing trend of very large corporations either basing their business models on the backs of independent ISPs or flagrantly unloading their traffic to the internet at large with absolutely no consideration of who is expected to carry that traffic for free. Netflix and AT&T immediately come to mind.
The main topic of the evening was how to equitably charge for bandwidth usage. As you probably know by now, 10% of our subscribers account for around 90% of the data consumed. My personal feeling is that this is neither democratic or fair. Light users essentially are subsidizing heavy users, which is to be expected with a flate rate all-you-can-eat pricing structure. Our competitors and peers in the WISP world have all moved toward tiered pricing plans with rather stiff overage charges. We cannot compete with wired internet networks on a price per megabyte basis, and even those networks are dropping flat rate plans. Wireless remains the only viable way to reach those on the wrong side of the digital divide, but the variables encountered in wireless networks are considerable. The technology of wireless networking equipment is improving exponentially, and we will continue to deploy the best and fastest as it becomes cost effective to do so. But bandwidth on wireless networks is always limited, as it is to a lesser extent on wired networks. So the problem becomes how best to apportion and charge for the available bandwidth. This is the problem we will be tackling in the coming year.
Lastly, thanks to all our loyal subscribers who stick with us through good times and bad. I’m sure you realize by now that everyone involved with Cyberpine is committed to providing the best service possible in a highly volatile industry and challenging geography.
While cleaning out the old Cyberpine office, I came across an article in the Meredith News from August, 2007. At that time, all Cyberpine clients were sharing a single T1 circuit at the Wentworth Library. It ran at 1.5mbps down x 1.5mbps up. Today almost all clients have that equivalent capacity for their own personal connection.
A point to point link was established between Red Hill and Great Hill with a 35mbps capacity
The fiber optic circuit went live at the Congregational Church in Tamworth. It will be shared access to the internets with Tamwort Wireless, our sister network. The first clients on the Little Young AP were successfully routed out the new circuit.
In a cooperative venture with Tamworth Wireless, a 3 year contract with Time Warner Cable was signed for a 25mbps x 25mbps fiber optic feed. This will satisfy the growing bandwidth demands on both our networks for the next several years. We will be trying to make the infrastructure improvements needed to make use of it in the next few months.
The workhorse of Red Hill, a MikroTik 532 with a daughterboard and 3 radios, was replaced with a faster 433ah. The old 532 was just plain worn out after years of continuous service and had started rebooting spontaneously in hot weather. The swap resulted in about 20 minutes of downtime and no reprogramming from the bench setup the day before needed to be done.
The Red Hill #2 MikroTik router was replaced with a new MikroTik 433 housing 2 5GHz radios. One makes the backhaul link to Meredith and the second provides service to the SSE in the direction of Moultonborough Neck. This model board has the handy feature of system input voltage, which reads the voltage of the battery bank and lets me know when it’s time to panic.
The first serious thunderstorm of the season on 4/11 knocks out 5 customers. Radios, power supplies, and ethernet ports on routers and switches bite the dust. Cyberpine absorbed all the labor costs and some of the equipment replacements, but remember-you own your own client equipment and it’s your responsibility to isolate it from your house wiring during thunderstorms.
Spring begins, but winter won’t let go. Trip up Little Young by Cathy and me, Declan on ground support, to switch traffic to Great Hill and Tamworth DSL. Success. The bridge radio had been up there since December, but never powered on, so it was a relief to find out it was working and talking to Great Hill. Changed the IP and Gateway of the AP radio and started routing client traffic over the link.
New DSL circuit is installed in Tamworth at the Congregational Church. It will be dedicated to Little Young Mtn. clients, breaking them off from Diamond Ledge.
Red Hill goes down for unknown causes. Trip up the following day reveals a melted charge controller. Replaced it, back online by noon.
Little Young gives up the ghost once again, automatically resets.
After a string of sub-zero nights, Little Young goes down, automatically resets in the morning sun
Eliza Jane and I hiked Red Hill today in 50 degree slush. We installed heavy duty battery interconnects and put things to sleep for the winter. Here’s hoping the new year is brighter than the last.
On trip #7 up Red Hill yesterday, the battery bank was replaced with 6- 105ah AGM batteries weighing in at 420 lbs. Jon Spence drove them up in his ‘53 Willys Jeep. Jon,Declan O’Connell and Ed Maheux helped this battered Norwegian carry them up the tower. Hopefully this will cure Red Hill’s power problems which has been plaguing me(and its clients) for almost 2 weeks. The cause of the failure is still unknown. The batteries that were taken out are at Ossipee Auto Parts and will be tested by their battery specialist tomorrow, although a test yesterday with their test meter showed them to be in fairly good shape.
It is worth noting here that Ed’s position as Towerman has been eliminated in Moultonborough’s 2011 budget. Ed has manned the tower from May to November for 23 consecutive years and aside from his valuable role in regional fire detection, he has been an indispensable steward in the maintenance of the tower and grounds. He’s always willing to share his knowledge of fire protection and tower history with the many hikers who climb Red Hill, as well as lending a much needed helping hand to those in distress-myself included.
Sunday morning, nasty weather outside-guess I’ll sit in front of the fire, read the news and have another cup of coffee-NOT! Phone rings at 8:00-Red Hill is offline. Pack up a can of gas and head up. Snow, sleet, freezing rain all the way up. The tower is encased in clear ice and I have to crawl up the last flight of steps. Up on the platform, windy and needle sharp sleet. Fortunately the generator starts up and people are back online. By the time I get back to the bottom I’m soaked to the bone.
What I’ll never be able to understand is why anyone would choose to hike Red Hill in weather like this. I met a man and two dogs on the way up and another man who was headed up after I left the tower. He was hiking to all the fire towers and picked today to see the Red Hill tower. Go figure.
I recently completed the switchover of the Library’s internet access from our original T1 line to two 7mbps down/1mbps up DSL circuits. We are retiring our T1 after 6 years of reliable service. It will be replaced with a 15mbps/1.1mbps dry loop DSL circuit.
Fairpoint finally began offering DSL out of the Sandwich CO a couple of weeks ago after much trouble with the switchover from Verizon and ensuing bankruptcy. I have recommended DSL service to anyone on our network it would benefit. Now we can shift our focus to those customers would forever be beyond the digital divide if it weren’t for Cyberpine.
Declan O’Connell and I have been busy getting the Tamworth Wireless Network off the ground. It will serve much the same function as Cyberpine does in Sandwich, and also provide a nearby source of additional bandwidth. They were fortunate to receive a grant from the Tamworth Foundation to cover most of the startup costs. There are now 12 clients running on that network with a waiting list of 40 or so more. They are currently running only 900MHz access points as trees and clear line of sight is even more of a problem in Tamworth than it is in Sandwich.
It’s been a long time since the last posting and many things have happened. The intense lightning storm of 7/21 knocked out 80% of the Diamond Ledge equipment-which had avoided damage for 6 years. It also knocked out 11 clients. A week later repairs are still ongoing and we’re not back to normal by any stretch of the imagination. I can repeat myself until I can’t stand it anymore about the importance of unplugging your equipment, but it seems to fall on deaf ears. 90% of our subscribers decided to stay up and running during that storm. Very bad idea. We have no choice in the case of access points-it would be impossible to shut them all down during a storm-but you subscribers do have a choice.
Data throughput on the network is also running 40% over a year ago. That is an amazing increase in traffic in one year. From now on any efforts on my part will be directed towards ensuring some measure of Quality of Service for our existing subscribers and searching for larger pipes to feed our network. Cyberpine is just a reflection of what is happening world-wide with bandwidth usage. Too much demand for too little supply. It will probably get worse before it gets better.
The Burleigh Farm AP radio and the first clients were upgraded from BreezeCom to new Ubiquity radios with extremely good results. More Black Magic in action.
Declan O’Connell and I climb Red Hill and using a custom antenna mount fabricated by Rod Teel manage to relocate the 900Mhz omni antenna in a position that is favorable to all the clients. Here’s a picture off the day’s efforts.
More problems crop up when Brown Hill clients start dropping off the network one by one until there are just 4 left connected-and not very well at that. A replacement radio was programmed and a new amplifier tested in case it was a hardware failure-but Cathy and I both suspected water intrusion was the culprit. Declan O’Connell volunteers to accompany me up the hill for an introduction to troubleshooting network problems. Hardware was swapped out, but it made no improvement. Took the coax N connector apart at the equipment box and found it was full of water-source of intrusion unknown. Solder (tricky in the wind) and crimp on a new connector and pick up a few more clients. That was all we could accomplish, so we headed back down and I made up a new length of coax with a new bulkhead connector and radio pigtail. Hiked back up and installed the new cable. All the clients came back to life. Met with Pete Hoag at the bottom and filled him in on the repairs. Our old original “Cyberpine” up there is showing its age since we installed the equipment in 2003 and may not be safe to climb anymore. We discussed alternate plans and future network strategies.
After losing the backhaul link from Ossipee to Diamond Ledge, the AP radio on Ossipee was put in WDS mode, meaning it was doing double duty dealing with clients and serving as the backhaul link to Diamond Ledge. Not a very satisfactory arrangement. When the AP radio quit functioning I contacted Rich Moren of Lakes Region Wireless to see if his functioning and under-utilized AP on Ossipee could serve our Ossipee clients. Rich agreed to do that for us, so all 10 Ossipee clients were configured for that network. After fixing some initial front end router glitches, all the clients were running quite well, so I decided to keep things that way. Rich has sporadic helicopter access to the mountain; I don’t and would have had to wait for the snow to leave the mountain before attempting a repair trip. Rich reported after a trip to the top that the backhaul antenna feed horn was broken and the cable was pulled completely out of our AP radio. Severe ice loading up there on the mountain.
Head up Red Hill for another repair trip-damages unknown. Cathy guessed that the omni antenna had ice build-up on one side since people to the north had dropped offline, and that the weight of the ice had pulled the coax cable partially out of its connector. She nailed it on both counts (not that that wasn’t my own theory as well).
It was an absolutely beautiful day-clear blue skies and dead calm. Here is a picture of the icing on the omni antenna and north side of the tower.
Hundreds of pounds of ice were removed from the structure, antennas and cables, and then the coax connector was replaced. A call to HQ confirmed that all 900Mhz clients were back online.
The next picture is a shot of Squam Lake looking west, minus a lot of its winter ice. You may still be a global warming skeptic (good luck with that), but this is March 16th!
Beware the Ides of March. In the headlines:
“The storm, which carried wind gusts of up to 70 mph, came about two weeks after heavy snow and hurricane-force winds left more than 1 million customers in the Northeast in the dark.”
More trouble on Red Hill. Damage-unknown.
Two repair trips were made up Red Hill to repair the 900 Mhz omni antenna. One on 3/3 and another on 3/5. Click here for a thumbnail page and on any thumbnail for larger pictures.
The ice and wind conspired to bend the 900Mhz omni antenna on Red Hill so it was almost pointing at the ground. It’s looking like the grid backhaul antenna on Ossipee suffered much the same fate. It’s amazing people are still connected.
February’s string of bright sunny days come to an abrupt halt with a wicked wet snow and sleet storm causing power interruptions and heavy icing on Ossipee Mtn. and Red Hill. The RF just can’t power through it all. First sunny day or temps above freezing with a wind should knock it off.
The core router in the Library was upgraded this afternoon, but not without the inevitable glitches. The config was ironed out and most people were reconnected by 5:00PM. I’m glad that’s behind me, but some of the routing I had to implement was very counterintuitive. I don’t like that.
Wicked Winds of the West! A branch wipes out the phone line on Quimby Field Rd. taking out all 3 of our T1 lines. I temporarily routed what networks I could up to Red Hill and out to Meredith. Fairpoint showed up within hours and the T1 lines were carrying traffic by 6:00PM.
Here’s a link to a Windows Secrets article on the supposedly Chinese state sponsored attacks on Google and others:
The DNS problems have been resolved after a few sleepless nights and much Googling. Everyone should now be able to call up any webpage withoutout difficulty. At most you may have to reboot your router and computer. There’s one exception-MAC users running Snow Leopard. There’s a bug in that OSX version that causes it to have DNS issues. If you’re running Snow Leopard and having problems, you can try a couple things that have worked for some-not for others. Empty your browser cache. Reboot your router. Release and renew your internet IP address in your Airport Extreme by backspacing out the old IP address, subnet mask, gateway and DNS servers, unplug your router from its power supply and then plug it back in. Reboot your MAC. If all that fails you can run this command in terminal. It’s case sensitive.
sudo killall -HUP mDNSResponder
This will force a reregistering of the DNS cache in your machine. Call me if you would like me to walk you through any of this.
Earlier versions of the MAC OS and Linux are not having any more problems.
I hope that Apple addresses this bug and supplies a fix soon. Oh wait-they did address it. It’s called a “feature”, not a bug.
Currently experiencing difficulties with domain name resolution. It seems to affect MACS and Linux machines more than PCs. A few good minds are looking at it.
Rich Morgan and I connected the two bonded T1 lines to the network and after routing out the usual demons, Little Young and Brown Hill clients are routed out the
Rich Moren sent me these pictures from his webcam on Ossipee. One was taken today, another New Year’s Day, and third one from this fall.
Here’s a photo of Asa and me hauling the generator and charger up Red Hill. We look relatively happy so I guess it must have been less than halfway up. LRCT chair Tony Halsey took the picture. He was headed down and smiling more than we were.
Power on Red Hill is restored after blinking out New Years Eve. Sometimes I forget how (overly) dependent we have all become on internet access. I was reminded of that fact on the evening of 12/31 when one of our subscribers called me on my cell phone shortly after the outage and and said simply “We’ve lost it all”. Internet is getting right up there with hot water on the necessity list. In analyzing the outages over the weekend(Brown Hill, Little Young, Red Hill) I’ve determined that the root cause is the fact that the network is handling on average twice as much data as this time a year ago. Part of that is due to more subscribers, but the real reason is that the internet has become an entertainment channel with ever richer and demanding content. As I posted previously, every packet you send, every packet you receive requires a little electric push to send it on its way. There is a direct relationship between between watts consumed by the access point radios and data throughput. Most of you at some point are sending your data through a solar powered/battery storage access point, and during the cold dark days this time of year, power from photons can be scarce.
The upshot of these latest outages was a trek up Red Hill today by Asa and me hauling a generator up to the fire tower on a sled. I’ll spare you the details, but now it’s up there to charge the batteries when we need to. Gary Floyd did the same on Little Young to get people reconnected to that access point. A wind generator on Red Hill would give us the boost we need during the winter. Rich Moren has one installed on Ossipee, and aside from losing its blades and smashing some solar panels last winter, it’s worked out quite well.
Everyone said that I’d have to be insane to try to run a fixed wireless solar powered network in the mountains of New England. They were correct. All this time I thought I was just being a stubborn Norwegian. Perhaps there’s no difference.
Pete Hoag and I hike up Red Hill and rewire the battery bank in a more democratic configuration.
Cyberpine holds its annual meeting at the Benz Center. Highlights:
Fred Bickford is reelected to the board for a three year term.
Cyberpine has ordered two bonded T1 lines from Fairpoint for additional bandwidth.
Subscribers on Diamond Ledge, Little Young and Brown Hill have been switched over to Deliberant radios.
The rise in video and audio streaming almost doubles the average data throughput on the network from this time a year ago.
Outage until the first support call received falls to about 3 minutes.
The AP radio on Little Young Mtn. and its clients are upgraded with help from Cathy Graham and Asa Berg. A Few routers didn’t like it too much, but other than that it was a smooth transition.
Cathy Graham and I hike Red Hill with ladder and 25 lbs. of tools to adjust the solar panels to a 15 deg. steeper angle so they will shed snow and ice better in the winter. Click for pictures here and here.
Network usage has been running near, at or above capacity for the last two weeks. Average data throughput hits a 5 year high for the week of 8/9 through 8/16
The Brown Hill access point and clients are upgraded. Assisting in the transition were Cathy Graham, Pete Hoag and Tom Wilson. The BreezeCom AP radio on Brown Hill had been in continuous service since September of 2004.
Cathy Graham, Pete Hoag, Ed Maheux and I lowered the two 160 lb. old batteries off the Red Hill fire tower to ground level. The ice-damaged backhaul antenna to Sandwich was replaced with a new panel antenna and the polarity shifted from vertical to horizontal to keep me from stepping on my own toes.
With help from Cathy Graham, Pete Hoag and Eric Morse, 13 of the 15 Diamond AP clients plus the access point itself were upgraded from BreezeCom Pro 11 radios to new Deliberant radios. It went off with only a few glitches and everyone was back online by mid afternoon. This resulted in a 600% improvement in bandwidth available to clients on that access point. We even got a thank you email from 1 (one) customer. Nice.
Little Young access point goes offline after performing flawlessly since Nov. ‘08. The panel/battery terminal block melted into a puddle of shiny plastic due to high resistance. 2 round trips on Thursday and one more on Friday for repairs and battery recharging.
A Deliberant radio is installed for backhaul at Burleigh Farm, replacing the BreezeCom that backhauled from Brown Point. All Burleigh Farm clients are now being routed directly to Red Hill.
Yet another trip up Red Hill to install a LVD (low voltage disconnect). This will disconnect the radio load from the batteries should the voltage drop below 11.5v, protecting them from ruin. The load automatically reconnects at 13v.
Beautiful day for an assault on Red Hill-clear skies, calm wind and temps around freezing. Loaded up the snow machine, sleds, and five new 100 amp hour batteries. With much needed help from Cathy, Pete Hoag, and Evan and Dave Shangraw, all 5 batteries and tools were hauled to the top. It took the snow machine 3 round trips to get them up-only 3 rollovers. The two depleted 160 lb. gel cells were hoisted out of the battery box and the new ones put in place and wired up. Internet service was interrupted for just over an hour during the process. Thanks to Rich Moren for valuable advice and Terry at Specialty Concepts, the manufacturer of the charge controller, for shedding some light on the rather strange controller readouts. Now to get those honkin’ dead batteries back to ground zero...
Loaded up the snow machine to go up Red Hill and replace the charge controller and attempt to make a good track up the mountain. Turned back on Bean Rd. when it started to rain. Back at the office I discovered that although the Red Hill access point had restarted in the weak sun, the radio that backhauls to Meredith had apparently locked up due to a partial reboot. Hitched up the machine again and headed out.
Conditions on the trail were difficult. Very thick fog, drizzle, totally flat light and water droplets on my glasses made for slow going-too slow as it turned out when I rounded a steep uphill blind corner and got the skis up on a boulder and dug the tail in-track off the ground. Wrestled the machine around in the narrow trail so it was headed downhill and hiked to the top.
Strange conditions up on the tower. Very thick fog-but a stiff wind from the south as well. Replaced the charge controller and rebooted the backhaul radio to get people back online. The battery voltage was hovering around 10.5 volts-below completely discharged. The worst case scenario of hauling up a new battery bank had arrived.
1Hr. 10 min. to the Red Hill firetower today. No snowshoes-6,142 steps(Yes-I counted them). Today was the one and only time I’ve been up there when it’s been dead calm. What a joy. You can actually take your time on the tower and get some troubleshooting done without the danger of your glasses blowing off.
The end result of testing left me more mystified than I was before. All voltage/current tests were totally normal-exactly what I would have expected to see with normally functioning power side equipment. The only theory I have at this point is that after the array fuse blew in the charge controller, the batteries started a long slow draw-down from the radio load. This completely discharged them and left the specific gravity of the gel electrolyte close to 1.00-essentially that of water. This froze in the cold temps and separated the gel from the plates, making recharging almost impossible. A worst case scenario actually as it might mean replacing the whole battery bank. Not easy to do even with volunteer help. I’m gearing up for that possibility and will keep you posted.
Parking lot to the Red Hill fire tower on snowshoes in 1 hr. 35 min. Not bad for an old codger. For some as yet to be determined reason, the panel array fuse on the charge controller had blown. I have no way of telling how long the batteries had been getting no charge from the panels, but the voltage of the battery bank was critically low. As of today, Sunday the 22nd, the battery bank has continued a slow rise in voltage, but it may be the case that they will never fully recover. This would mean hauling new batteries to the tower, something I can’t accomplish by myself, so Cyberpine volunteers may need to be pressed into service.
A word about Octoshape. It’s a program that uses the Flash Player installer and enables your machine to be a part of a filesharing network to stream video from sites like CNN Live. Your machine actually becomes a server on the network and can use upstream bandwidth in the 600k range. This is huge. CNN benefits at the expense of ISPs by spreading the streaming video load around. You can read all about it in a Windows Secrets article at http://windowssecrets.com/comp/090205
Needless to say, from my perspective it amounts to peer to peer filesharing which is expressly not allowed on our network. If I detect its fingerprint you will be getting a phone call. Fortunately it’s easy to uninstall. Just go to add/remove programs in XP or programs and features in Vista and remove it.
A 900Mhz Access Point has been put online to serve the downtown area residents. The Federated Church now has high speed internet access to enable them to construct and manage a new website and put the weekly sermons online to reach a wider audience.
Flying off on a related tangent-I’ve always been involved and interested in alternative energy, especially wind and solar. Ossipee Mountain now uses a mix of solar and wind to push your packets through the air. Here’s a picture of a recently installed Southwest Windpower Air X up on Rich Moren’s tower. I installed a similar wind generator, the Air Breeze, for my off-grid home. Chris Read erected a Southwest Windpower Skystream 3.7 at his home in Sandwich. Click here for a short movie. A combination of solar and wind work very nicely together-when one source drops off the other one usually picks up.
If you are interested in windpower, here’s the link to Southwest Windpower’s website:
Service at Red Hill restored at 2:06 PM.
Solar panels clear of snow-battery voltage at 12.8 volts.
Sooo...Do you have flabby abs?
Have I got an exercise routine for you!
Hike to the top of Red Hill and back 3 times a week. And it won’t cost you a cent!
But wait-there’s more!
When you get to the top, your reward is a temperature that’s 20 degrees colder than the parking lot with a 30 mph wind gusting to 40. You’ll be cooled down from your climb in no time at all! Then you have to take off your glasses because they are completely iced up and take off your gloves so you can work on some very sensitive electronics. You’re going to love it!
People, I need to make you aware of something-well-a few things.
Every packet you receive, every packet you send, takes a little electric push to send it on its way. 4 of our access points are solar powered with battery storage; Ossipee, Little Young, Brown Hill and Red Hill. This time of year solar radiation is hard to find. When was the last full day of bright sun? Can’t quite remember? Neither can I. So the next time you think that you absolutely have to watch the latest” family cat boxes family dog” video -think twice. You’re burning up power that’s at a premium these days.
It also appears that from sniffing a few packets here and there that peer to peer file sharing is still active on the Cyberpine network. Per the AUP and laws of the land, sharing copyrighted material is expressly not allowed. Be aware that many copyrighted files are tagged to track where they came from and where they are going. They are also an excellent source of trojans and worms. As an ISP, I’m obligated to turn over the IP address of the person doing it if asked by the RIAA or FBI. I don’t want to be put in the position of ratting on one of my friends or neighbors on the network. Typical fines these days are in the 3000.00 to 5000.00 range. Don’t do it.
Sorry for the rant. Those trudges up mountains to keep things running give me way too much time to think. Happy Holidays to you all.
Happy Solstice! I used to love this time of year. Stay inside by the stove, watch the thermometer bottom out, frost paintings on my windows, winds howling overhead, snow drifting me in. Those days are gone. Now I just keep my fingers crossed for a few stray photons falling on my solar panels and dry clothes in the morning for another assault on a mountain to nurse frozen electronics back to life.
Red Hill has been causing me no end of headaches lately. First ice ups and antenna damage, then power loss, now a Mikrotik router with 3 crucial radios onboard that refuses to talk to anyone-not even me. Its sister router/radio up on the firetower is babbling away at Meredith, but it has about as much to say to the world as Paris Hilton after an all night bender. Ed Maheux, God bless his soul, snowshoed up with me yesterday and allowed us to tap into his battery bank to fire things back up. Hopefully I’ll be able to get all that can’t-live-without content flowing tomorrow on my third trip up there in a week. Until I get it running, all 900Mhz clients and Rockywold are dead in the water-er-ice. Everyone else has been rerouted out the T-1 line, so keep in mind folks that it’s being very heavily shared-maxxed out as a matter of fact. Be kind-be considerate. Light a candle, have some quality time with real people.
That was one nasty ice storm over the weekend, but our neighbors to the south fared far worse with hundreds of thousands of households still without power. Diamond Ledge proved to be the weak link in the chain this time, losing power Friday morning. It ran on my portable generator until 10:00 PM when an overworked crew reset the tripped transformer.
Red Hill and all its antennas was transformed into a block of ice. The 11’ long omni antenna looked like an upside-down icicle, 8-10” at its base tapering to a spike at the top. The directional antennas accumulated many pounds of ice as well.
I trekked up Red Hill today to make sure that the panels weren’t damaged from falling ice and were clear to collect what precious little sun we get this time of year. It turned into an ordeal of course. The first mile or so of trail had started to thaw and was manageable enough, but the conditions got rapidly worse on the higher elevations, so I left the trail and bushwhacked for the top. Lost my ice scraper out of my backpack then my hat out of my pocket, snagged by wicked ornery branches somewhere. Here is what the top looked like upon arriving.
The winds also picked up dramatically at the top and by the time I got to the second platform they were gusting in the 25-35 mph range. I had to keep one hand on a railing at all times to keep my balance and one gust blew my glasses right off my face. Fortunately, they caught in the chicken wire “safety” fence around the platform.
The solar panels were clear of the ice build-up I had observed the day before from the ground, and so were the antennas. The 36” grid backhaul antenna linking to Sandwich suffered some severe damage, most likely from falling ice and was pointed pretty much at the ground. It is functioning, but just barely, and will have to be replaced with a more durable flat panel antenna. Unbelievably severe conditions up there in the winter. I was soaked to the skin in a very short time as the top was in the clouds. The trip down the trail was even worse than the climb up. Picture a mile long bobsled run filled with baby carrot sized icicles. Challenging even for a 59 year old half blind Norwegian. Here’s a picture of the trail.
Cyberpine Cooperative held its annual meeting at the Benz Center. 10 subscribers showed up. This can mean:
A. Nobody really wants to hear me drone on about tech stuff they don’t understand and could care less about, or
B. Most subscribers are happy with their service and think it’s pointless to show up at one more meeting.
Actually, I think it’s a combination of the two.
For those of you who missed it, the highlights.
1. The monthly business subscriber rate of 60.00/mo was dropped. The additional cost to business customers was based on the old Firstbridge rate structure assuming business customers used more bandwidth and were more demanding of tech support. Neither of these have proven to be true. Bandwidth usage is entirely dependent on the number of teenagers behind the connection and tech support demands are dependent on the subscriber’s personality and computer literacy. Monthly email billing was also discussed. The majority opinion was to only email .pdf year-to-date statements quarterly, the other months to remain monthly billing emails.
2. Dale Mayer was re-elected to the Board for a three year term. Although Dale was the only person on the ballot and there were no write-ins, there was barely a majority vote (50% of subscribers) to reelect.
3. Expansion plans were discussed. The only expansion in the works is a 900Mhz access point that will cover most of the downtown area. I spoke about the eventual need to replace the aging BreezeCom equipment with newer and faster radios. This would have to be done in stages converting one access point and all clients using it one at a time.
4. Great refreshments were provided, most of which I took home and enjoyed the rest of the week.
Deliberant AP and Client Bridges were installed on Diamond Ledge, Little Young and Brown Hill, replacing the older FHSS BreezeComs. This created a larger pipe to haul data off those two remote Access Points.
Microsoft has released an unusual “out-of-cycle” patch for a potentially serious security vulnerability in almost all Windows versions. You can read about it at http://windowssecrets.com/comp/081024. I would urge you to manually run Windows Update ASAP to get the patch.
In case you’ve been stuck at your keyboard punching in wrong numbers and haven’t been able to get out in this beautiful fall weather, I thought I better give you some glimpses through my camera lens. These were taken today on my 5th trip up Little Young Mountain getting it ready for winter. If you get a magnifying glass when you put your cursor on the picture, you can left click for a higher resolution. This picture is about a quarter of the way up the trail
In case you thought that entropy was no longer functioning in this universe, here’s what it’s done to a perfectly good Ford Roadster. It will do it to you as well I’m afraid.
Apparently a moose thought the tower was the perfect antler scratching post and in the process he snagged the coax cable and nearly ripped it out of its connectors. Here’s a before and after picture of the damage and repairs
Here’s a shot of the Little Young Access Point decked out with a new solar panel and additional battery to help get an old man through winter.
With thanks to your super woman secretary, Cathy Graham, and the Quimby Trust, another difficult infrastructure improvement has been accomplished. Both the Little Young and Brown Hill access points now have an additional 220 amp hr. battery, 130 watt solar panel and 20 amp charge controller to help get them through the cold dark days ahead. This brings them up to the generating/storage capacity that we have on the Red Hill fire tower, which performed without a hitch during last winter’s extremely snowy conditions.
This month marks year 5 since Pete Hoag and I constructed the network’s first access point on Brown Hill. A cause for celebration if you’re looking for a good excuse!
Another needle in a haystack located. After a long and gradual downhill slide into mindlessness, the AP backhaul radio which serves Little Young and Brown Hill access points was determined to be the cause of deteriorating connections off those APs. It was replaced and the 500mw amplifier it used was replaced with a 1000mw model. Smiles from the remote north of Sandwich.
Since these older BreezeCom backhaul APs carry traffic 24/7 they run fairly warm-OK-hot. After a couple years of this they start fading off frequency. Not a sudden death unfortunately. The funeral procession would be hard to miss. More like dying of old age.
Google releases the beta version of it’s Chrome browser. Read the comic strip write-up at http://www.google.com/googlebooks/chrome/
These guys went outside the box for this one-and scored big time.
Replaced the venerable old BreezeCom Pro 11s on the Ossipee backhaul link with new Deliberant B/G radios after the BreezeCom bridge radio failed up on top of the mountain.
The backhaul links from Brown Hill, Little Young and Brown Point will follow suit as finances allow.
msnbc.co-BREAKING NEWS spam replaces the CNN Alerts. Same malicious thing. Don’t follow the links.
Monday night the 11th the main fiber optic line running from Concord to Laconia was cut affecting phone and data communication for a large area-our network included. Service was restored 24 hours later on Tuesday morning. A deja vu of the Valentine’s Day outage earlier this year. See the 14.2.2008 post.
Beware of bogus CNN Alerts and Daily Top 10 spam emails flooding your inboxes. Following the links will install malicious code on your machine.
Due to even more rainy, dismal weather, everyone is driven indoors and the network traffic hits an all time high of 6.1mbps sustained throughput from 3:00-4:00PM today.
All traffic from the 4 wireless networks that run through Diamond Ledge-Brown Pt/Burleigh Farm, Little Young/Brown Hill, Ossipee and Diamond Ledge AP are now being routed. A QOS implementation called PCQ (per connection queuing) is also being used. This prevents any one bandwidth hog (you know who you are) from dominating an access point to the detriment of other users. Its aim is to maintain connectivity and reasonable data throughput for each connection even during times of heavy traffic.
Lightning continues to be a problem this year. The best way to protect your equipment is to unplug the power from any internet device that is connected to your house wiring ie: radio, router,computer. This isolates that equipment from your house wiring and makes it a much less inviting path to ground for lightning induced surges. It will likely save you money for repairs and spare me a lot of frantic running around fixing things after lightning storms.
So I guess all it takes to bring on devastating lightning strikes is for me to leave town. Makes for a real nice vacation. I love my job.
I see that I haven’t reported any news for almost 2 months. My mentor and good buddy Thom Lawless of Rapid WiFi checks this page regularly to keep tabs on our network and brought this lapse to my attention today. So, to make up for lost time-a tech update and a rant.
The dedicated RDC radio on Red Hill decided to roll over and die for no good reason(other than overwork and fatigue) on June 8th. It had been losing its mind for quite some time, but I was usually able to kick it around enough to make it sit up and take notice. I rerouted their traffic through Brown Point>Diamond Ledge>Library> T-1(like the good old days) and had them reconnected in good time. Then, a couple days later, our first major thunderstorm of the season rolled through. Not a direct hit, but the electrons were grouping enough to make the hairs on my arms stand straight up. Never a good sign. The next morning, after checking all network connections off the APs, everything semed to have survived very well. Then the support calls started to come in. 3 of the 15 900Mhz clients off Red Hill were dead even though the AP up there showed their radios as being connected. Later that night, in my infinite wisdom, I decided that I’d better reboot the router at Red Hill to unscramble any ESD induced anomalies. Well-the reboot button on the web interface is right next to the shutdown button and due to yours truly being way over tired, I accidentally hit shutdown. Great. Now there’s no path for traffic to DSL in Meredith at all. I did an emergency rerouting of ALL network traffic out the T-1 line. Red Hill hike-oh boy.
Up on Red Hill the next morning I took apart the MikroTik router and replaced the RDC radio card. I also discovered the reason for the failed original Red Hill>Meredith link this last winter. A connector(very small and fragile) off the radio card had partially popped off, which was just what I theorized had happened but couldn’t confirm without a complete disassembly. Note to self-hot glue all u.fl connectors to the board. http://en.wikipedia.org/wiki/U.FL The MikroTik and its 3 radios were reassembed and powered up before noon.
Back at ground zero, I routed traffic back through Red Hill and switched RDC back to its dedicated link. 3 of the 900Mhz clients still could not connect. This proved to be blown ethernet ports on their MT boards as a result of the lightning storm. Recommendation- if you have a remote 900Mhz client unit, unplug the POE line going out to your unit at the slightest hint of an impending lightning storm. It appears that it doesn’t matter if the unit is grounded or not-one of the smoked units was and two were not. The storm also tripped many GFIs and blew several power supplies.
The remaining networks coming into Diamond Ledge-Ossipee, Brown Point/Burleigh Farm and Diamond Ledge AP itself- will be readdressed and routed soon, so expect some random downtime while I switch them.
And now-bear with me-the rant.
Teenagers. Unlike my generation, they grew up with internet access and soon took it for granted. Now don’t get me wrong here. It’s absolutely essential for teens to be able to use the internet efficiently and productively. This will be true for generations to come. Teens are typically more tech savvy than their parents and I’ve encouraged them to lead their struggling folks out of the woods on many occasions. BUT-teenagers are absolutely clueless about the impact their favorite apps have on a small network. People of my generation are happy that a search for Apple Brown Betty will produce 10,000+ results in .67 seconds and opening the page with the prize winning recipe takes 4 or 5 seconds. Ahhh-the miracle of the modern internet. Not so with teenagers.
A typical teen these days may have 5-10 Facebook pages loaded and opened, running Limewire file downloads in the background, have multiple IM windows open and active-and wonder why the Youtube video they just clicked on is taking soooo long to play. They are totally clueless about how much bandwidth they are burning up-or how it impacts other users-or who provides it and how it’s being provided. ISPs all over the world are being faced with exponential growth in bandwidth demands-caused for the most part by the apps teenagers typically use. Comcast caught holy hell for shaping bandwidth and throttling certain types of traffic. Time Warner just announced that it will begin instituting data throughput caps and per meg surcharges on usage exceeding their limit in certain areas. The bottom line is that building out infrastructure to increase bandwidth and keep all users happy is very costly and is not an overnight process. Demand is simply exceeding available supply. On a small network such as ours, overuse by a few can have a disastrous effect on many. I just had a suggestion from a Cyberpine member that for every 2 hours of X-Box online use, that person should be required to mow a Cyberpine member’s lawn for 2 hours. Not a bad idea.
So-parents-educate your teenagers on bandwidth usage. Remind them that they are on a non-profit co-op network. As in COOPERATIVE. If they don’t know what that means-have them look it up. In the dictionary. The paper version that’s gathering dust on the shelf. End of rant.
As of this afternoon, Brown Hill, Little Young and Brown Point are running on their own backhaul links back to Diamond Ledge. Even though traffic is going through a switch and is not being routed, the connection speeds for customers on the outlying APs has increased substantially with far fewer collisions and retransmissions. This also means that customers who access the original Diamond Ledge AP directly are no longer competing with traffic from the other access points. This was a network improvement a long time in the making. OK-so I’m a little slow.
Another milestone in Cyberpine network improvements. Your VP, Pete Hoag, and I installed the new backhaul antennas on the Diamond Ledge pine. This is the first step in segmenting the network and routing traffic through the bottleneck there so we can take advantage of the increased bandwidth on the other side. We also dressed up the old “Cyberpine” with a new equipment box and rerouted the coax .
Expect some outages in the coming week as new radios are hooked up and routing is put in place. These should be of relatively short duration unless yours truly really screws up.
Ed Maheux sends the first email from the Red Hill Fire Tower.
Now he can check the radar maps to see what’s bearing down on him before it’s too late!
Our IP address was re-listed on Monday with the CBL and other major blocking lists due to an infected machine spewing out spam email. As I warned everybody previously that draconian measures would be implemented if this bad behavior continued, all outgoing port 25 mail is being dropped unless it is destined for mail.cyberpine.net. To repeat, you MUST have a Cyberpine username and password and authenticate outgoing mail with same AND the outgoing mailserver on your email account has to be mail.cyberpine.net. We will not be acting as an email relay for mass mailing trojans!
Please scan your machines regularly. If you are running Win XP or Vista, turn off System Restore before running scans and then turn it back on after you know your machine is clean.
Link to turn off System restore in Vista or XP:
Hiked up Red Hill and bent the Sandwich backhaul antenna back into alignment. It was pointed to the area between Barville Pond and the Sandwich Beach. Not the best area for transferring data out of the Library. The North side of the fire tower is subjected to some pretty extreme icing and wind. Makes me wonder how the AP on Ossipee survives.
Ed Maheux reports that the 2.4Ghz antenna on the fire tower that links to Sandwich got jarred 90 degrees out of alignment. I had noticed the signal strength drop and was hoping that was the cause and not a radio problem. Fixit trip on the next decent day.
NHVT mailserver goes down. This too has passed.
Service area map updated
Since I get asked how reliable our network is all the time, and you might get the impression from reading previous posts that there are constant outages, here are the 1 year uptime stats for all the access points:
Red Hill- 100% (knock on wood)
Library AP- 99.8%
Diamond Ledge- 99.7%
Burleigh Farm- 99.6%
Brown Point- 99.4%
Young Mtn.- 98.4%
Brown Hill- 97.9%
Ossipee Mtn.- 96.7%
Not bad considering the ice and snow, wind and lightning our equipment is subjected to.
First clients are connected to the 900Mhz access point on Red Hill.
The results are much better than even I expected.
Ohhh...how nice! Someone was thinking of me on Valentine’s Day. Did I tell you how much I like Brown Hill as well?
I think the Eskimos have a few dozen different words for “snow”. I have a few more than that, but only “crusty” and “powder” are printable here.
It was a crystal palace on Brown hill. 1/4” of ice on everything. After slogging my way up and feeling a good deal of sympathy for the deer and moose doing the same thing I was doing-but without the aid of snowshoes-I discovered the outage was caused not by a power failure as is usual, but by a 6” diameter branch that snapped off from 20’ up the tree. It hit the solar panels and equipment box. The panels amazingly did not break, but the coax cable coming into the box was bent at a sharp 90 degree angle, breaking the backhaul link.
Not having the right parts to repair it, I hiked back up at 3:00PM and fixed it, only to discover upon my return that a major Verizon fiber optic line had been cut in 2 different places, knocking out both our T-1 line in Sandwich and the DSL in Meredith, as well as land lines and cell feeds.
There’s apparently no redundancy in their system for our area. Very reassuring.
Attention! Our new DSL IP has been listed in the Composite Blocking List because there is a spam-bot infected machine behind it. All people running PCs on our network MUST run the scanning tool which can be downloaded at:
Hunting down the infected machine is taking way too much of my time so I am relying on you to take the responsibility of tending to your own machines. If it isn’t cleaned up soon I’ll have to resort to fairly draconian measures as this is causing everyone on the network email problems.
Ed Maheux hikes to the fire tower and scrapes 75% of the snow and ice off the solar panels. Who said solar doesn’t work in New England?
The Library and Library AP customers are now routed out the T-1 line. Everyone else is being routed to Meredith via Red Hill.
Two round trips up hills in the same day.
Pictures of the Brown Hill trip here.
Pictures of the Young Mountain trip here
You might have to double click the the picture to see it full size if it displays to fit the page. I don’t get this web stuff. It’s too complicated.
Brown Hill AP-the first and still the best access point- bane of my existence. I see “ordeal” might be over used, so I’ll omit it. Today was one for the books.
Shovel out the snowmobile trailer, chip the snowmobile free of the trailer, do a sliding puzzle shuffle to get the jeep, truck and trailer all pointed down the driveway in the correct order, pack up the generator, snowshoes, battery charger, gas, and shovel, drive over to the Brown Hill trail in the jeep to scope out the situation, collar Ted Adriance and ask him if he’d blast a hole through the snowbank so I could get the snow machine onto the trail, he can’t make it up Brown Hill Road, collar Nancy Collins with her John Deere, she makes it up after stalling out multiple times and scoops out a hole, go back up to my house to swap the jeep for the truck and snow machine, take Pease Hill Rd. to Brown Hill to avoid certain misfortune on Brown Hill Road, unload the machine and strap on the generator, charger and snow shoes, run the snow machine through the beautiful hole that Nancy cleared, get all of 50’ up the trail, bottom drops out, machine on its side mired in snow, shovel it out, wrestle it 180 degrees in the trail and drive back to road, put on snowshoes and hike to the top of the hill, panels iced up and covered in snow, scrape them off, do a manual reset of the radios knowing full well they won’t even make it through the night, snowshoe back down baby step stomping the whole way to pack a trail, get to the bottom and turn the snowmobile 180 degrees again and head up, machine slides off track onto its side multiple times so decide to walk beside it to keep it upright, get to the last steep uphill pitch and have a rare occurrence of good judgement and stop, unpack the generator, charger and.....oh damn, the snowshoes fell off somewhere, trudge up to the top through knee deep snow carrying the charger and generator, fire it up, AP lights up, head back down, decide to leave the machine at the top until tomorrow as I have no energy left.
7000 calories burned, 2000 in stock. Anyone want to join my fitness club?
Finally make it home at 7:00 PM and guess what? I can’t get my boots off.
What a totally insane...er...good old fashioned winter we’re having! I thought being a WISP was sitting at a keyboard and punching in wrong numbers.
Do you suppose an extra panel and battery are in the budget?
That was a non-stop uphill battle, but the network is finally being routed out to DSL in Meredith by way of Red Hill. I was really beginning to wonder if I’d see it in my
lifetime. Next phase is to get downtown Library AP clients back onto the T-1 line, segmenting and routing traffic off Diamond Ledge which remains the choke point, hooking up clients to the 900 Mhz AP on Redhill, breaking off Rockywold onto their own connection, swapping the 2.4 Ghz Library AP for a 900Mhz access point, upgrading customer equipment, and eating a cheeseburger.
Thanks to T Parsinnen at NHVT for tracking down a buried typo in a core router and Thom Lawless at RapidWiFi for mangling your packets enough so they knew where we wanted them to go.
How ‘bout them Patriots?
Wow. What an ordeal. Haven’t had a massive coronary like that in a long time. Right on the heels of totally losing my hard drive. The network outage was caused by condensation in the RJ45 POE connector in the main backhaul radio on the Library. All but 13 customers were affected. Thanks to Pete Hoag who provided his bucket truck with no advance notice.
The DSL circuit has been installed at Meredith Harley and will be given a trial run soon. If it works as expected, all wireless traffic off Diamond Ledge and beyond will be routed to that line. The Library and Library AP customers will continue to use the T-1.
Hiked up Red Hill and replaced the malfunctioning brain with a new perfectly functioning one. Relatively painless operation. I may have to try it myself. The Library to Meredith Harley link was reestablished, paving the way for hookup to DSL feed on or around Jan. 28th.
Thanks to everyone who took the time to run the scan on their machines. The infected machine still has’t surfaced so I may have no other option but to drop all outgoing email packets destined for anywhere other than mail.cyberpine.net. If you are a Cyberpine subscriber, your outgoing mail must be authenticated with your user name and password. Email relaying will not be allowed. Synching of webmail accounts (such as Hotmail, Yahoo Mail, Earthlink etc.) with your local email client (Outlook, Outlook Express) may present some difficulties which I will deal with on a case by case basis.
Our public IP has been relisted in the CBL (composite blocking list) after I had it removed on 1/19, so there is still an infected computer on our network.
Here is an excellent scanner to run and it only takes 3-5 minutes.
PLEASE download and run it and call me if something is found on your machine.
For those of you who want to scan your machines for a mass-mailing virus/trojan/worm, McAfee has a downloadable stand alone scanning program which can be found here:
New MikroTik 532-A/564 core router is installed at the Library, replacing the venerable Cisco 4500/Nokia/mOnOwall combination.
Not too many people think about this, and it has only recently become a hot topic in the networking world, but every packet of information that moves along our infrastructure requires a certain amount of energy to move it to it’s destination. We’re fortunate, from an energy usage perspective, that half our access points are solar powered. The old equipment just replaced at the library used a relatively constant 300 watts. The new MikroTik does the same amount of work and uses about 12 watts. for a savings of about 35.00/month.
And while we’re on the subject of energy usage, it will both save energy, money and help your computer last longer if you simply shut it down when it’s not in use. The added side benefit is that a shutdown and reboot will flush the memory and clear out various snippets of software sludge that inevitably build up. Before I step down from my soapbox, I’d like to alert all of you to the fact that there is an infected machine on our network that is sending out spam. I can’t possibly clean everyone’s machine, so I’m asking that you voluntarily start up in safe mode sometime soon and run a whole system virus scan. If I detect the fingerprint of a mass mailing machine in action, I’ll make one courtesy call, but after that your cord will be cut until you have a clean machine to put online. Thanks.
Elation turns to despair in a matter of 4 short days when it’s discovered that the new MikroTik installed on Red Hill has lost itâ€s wireless interface and 2 of its 3 ethernet ports. Which means-no- I can’t even type it-let’s just say more work and yet another delay. Feets don’t fail me now.
OK folks. I apologize in advance for this long news post. This is THE BIG DAY in my typically mundane blog. Snowmobile halfway up Red Hill today, walk (no snowshoes needed) the rest of the way due to many exposed rocks and ledges (climate change disbelievers please join GW in Crawford, Texas and go cut some brush with him). Replace the 2.4 Ghz dish with a 5Ghz dish and point it by eyeball toward Meredith Harley. Add a new MikroTik router/R52h radio combo and connect it by ethernet to the existing MikroTik 532 with 3 radios in it.
Run a new aerial Cat5 POE line from the Library to the Sandwich MikroTik router/radio to replace the direct burial Cat5 laid by Asa Berg and me in the fall which was severed by who knows what (spring will tell).
Hook into Red Hill from the Library and, here, in painful detail, are the stats:
Red Hill to Meredith Harley Rx signal: -67 db, Tx signal: -66db
Signal to noise: 28 db
Bandwidth test: Receive from Meredith, TCP: 14Mbps
Send to Meredith, TCP: 15.9 Mbps
Tx/Rx CCQ: 93/98%
Library all the way to Meredith Harley: Rx-11.7Mbps, Tx-11.1Mbps
What this all means in English is that this Sandwich/Meredith link is now running and it will easily support the DSL fat pipe which runs at 7Mbpsdown/1.1Mbps up. WooHoo! What a long and painful haul it has been. But not without some very valuable help.
First and foremost, Thom Lawless of Rapid WiFi on Whidbey Island off Seattle for his patience and absolutely critical expertise in IP addressing/routing/nat’ing; Cathy Graham for putting up with”Gunnar’s World” and keeping things glued together; Ed Maheux of the Moultonborough F.D. who has accompanied me and assisted me on every trip up to the Red Hill fire tower; Pete Hoag for his fearless and on a moments notice assistance in all emergencies and some pretty scary antenna installation on top of the Red Hill tower; and of course to all you subscribers who have put your faith in a cabinetmaker to maintain your life line to the world at large. There is some higher speed light at the end of the tunnel after all.
Take the SkiDoo up Little Young at 9:00AM on it’s maiden voyage to find out why it blinked off after Gary’s good charge on Sunday and at 2:45PM in good sun on Tuesday. A faulty charge controller is to blame. 10’ visibility at the top in dense fog. Trail is slushy and the machine gets stuck in washouts several times. Good thing it’s really light. Screw from the ring gear falls out at the top and jams locking up the engine. Dislodge that and come back down in the rain. So much fun-so precious little time.
Hoping for an automatic reset of Brown Hill in bright sun which doesn’t happen, lose patience and snowshoe up for a manual reset at noon
Gary does a preventative battery charge on Little Young.
Cyberpine purchases a 1986 SkiDoo for winter maintenance duties.
Gary Floyd does a manual reset and checks for ice and snow build-up on the solar panels on Little Young.
Snowshoe up Brown Hill at 8:45. 5 below zero. Absolutely the most beautiful snow I’ve seen in years. The panels were indeed 75% covered with ice and snow. Scraped them off and service restored at 9:30. Love that Under Armour.
Snow refuses to slide off the solar panels on Brown Hill and the photons couldn’t get through. It gives up at 5:00PM. Rod Weinberg reports that over 48” of snow fell in December- a 35 year record breaker.
First 900Mhz CPE (customer premise equipment) constructed and test linked successfully with the 900Mhz AP on Red Hill.
Gunnar hauls generator and battery charger up Brown Hill by sled and snowshoes. Quite fun. Even more fun hauling it back down. Ever had a generator on a sled run over the backs of your snowshoes as you’re scrambling to get out if the way? Great visual. Looney Toons has nothing on me.
Note to self: Don’t go “walking” up Brown Hill in the winter without snowshoes. Service restored at 1:45 PM. Is one day of bright sun too much to ask? Merry Christmas to you all.
Gunnar snowshoes up Brown Hill and does a manual reset. Service restored at 11:30 AM. Load cuts out again at 4:30 PM due to low voltage. Anyone else tired of winter yet?
Brown Hill and Little Young Mountain die from lack of sun and no winter vacation. Gary Floyd makes a total of three round trips to the top by snow machine, the last one after dark to swap out the battery charger that wasn’t working. Thank you Gary!
Rich Moren does some kind of voodoo magic on Gene the radio station engineer and gets a helicopter trip up the mountain. He said it was like landing in a cotton field of rime ice. Generator is started, panels cleared, service restored at around 2:30 PM. Thank you Rich and Gene.
It’s decided that the only feasible way to repair the power problem on Ossipee Mountain is to go up on the radio station’s next maintenance trip by helicopter. Rich Moren will try to move that trip as early in January as possible.
Replacement equipment destined for Red Hill tested successfully at Meredith Harley. Dedicated link to Rockywold/Deephaven off Red Hill also passed with flying colors.
Haul generator and battery charger up Brown Hill via ATV. Service reconnected at 10:50 AM. This will be the last trip possible for the ATV given Sunday’s expected 6” of new snow. Does anyone know of a cheap but dependable snowmobile for sale?
Brown Hill shuts down at 1:00 PM
Brown Hill shuts off at 11:30 PM. Resets in partial sun automatically at 10:30 AM
Brown Hill shuts off at 6:20 AM. Hike up to do a manual reset . Back on at 11:45 AM. Whereâ€s global warming when we need it?
Ossipee Mtn. loses power at 10:30 PM.
Little Young Mtn. loses power. Attempted charging trip by ATV proves impossible. Gary Floyd takes the generator and battery charger up by snowmobile and leaves it running overnight.
The 2.4 Ghz antenna and equipment box are removed at Meredith Harley and a 5Ghz antenna and 350 mw radio installed.
Ossipee Mtn. loses power. Rich Moren is able to start the generator remotely.
Due to ice build-up then snow on the solar panels, Brown Hill loses power. Batteries are charged and service restored at 11:00AM.
Realizing the painful yet obvious fact that yet another trip up Red Hill is necessary, Gunnar orders a new MikroTik 532-A and R52H radio and replacement 5Ghz antennas for the Red Hill to Meredith link, avoiding the crowded 2.4Ghz channels found on the initial scan done at Meredith Harley. Word is also received from Rick and Roger Heath that using the hardware store as an alternate endpoint antenna would be a workable possibility.
A test radio is programmed to match the Meredith Harley radio and taken to Greg Berry’s Castle, the top of the intervening hill with the cell tower. The radio links up, but signal strength off Red Hill proves to be extremely weak. The test radio was setup at a point on Rt. 25-B 2 miles closer to Red Hill with similar dismal results.
Gunnar contacts Rick Heath about the possibility of mounting the endpoint antenna on Heath’s Hardware roof, which has an almost identical shot to Red Hill as the Library-Red Hill link.
Gunnar meets with T Parsinnen of NHVT at Meredith Harley. After much troubleshooting it’s determined that location won’t work. Even though NHVT doesn’t have DSL equipment in either the Center Harbor or Moultonboro COs, it’s decided to find a Center Harbor location for the endpoint and lease the DSL line from Verizon.
The Meredith end of the backhaul link is constructed at Meredith Harley. Red Hill only appeared on a scan of networks sporadically and a with a very weak signal strength. Many competing frequencies and an intervening hill 3 miles out is determined to be the most likely cause. A scan from Red Hill does not pick up the Meredith radio.
The Library endpoint of the DSL backhaul link is installed and communication with Red Hill is established at an average throughput of 16mbps.
Gunnar finishes programming the MikroTik router for Red Hill with valuable technical help from Thom Lawless of RapidWiFi, installs it and powers it up.
Cyberpine holds its second Annual Meeting at the Benz Center. The new Board of Directors is elected (Fred Bickford, Dale Mayer and Gunnar Berg) and the by-laws are officially posted on the website here
Pete and Gunnar install the 900Mhz omni antenna and hoist the storage batteries on the fire tower
Cathy and Gunnar construct and install the backhaul and Rockywold antennas on Red Hill.
Gunnar and Asa construct the solar array on Red Hill.
Moultonborough Planning Board accepts Cyberpine’s site plan as drawn.
Moultonborough ZBA grants Cyberpine a special exception to allow commercial use of the Red Hill fire tower
Ed Maheux of the Moultonborough Fire Dept. hauls the equipment up to the fire tower and Jim Hambrook surveys the top in preparation for the Moultonborough ZBA and Planning Board hearings. Meredith Harley agrees to allow Cyberpine the use of their location for the DSL demarc and transmitter.
Gunnar speaks at the NH Telecommunications Advisory Board Meeting on the nature, successes and obstacles in creating a rural wireless network in an underserved remote area.
Representatives from NH DRED and Executive Councilor Ray Burton make an official presentation of the matching grant to Cyberpine at the Sandwich Town Hall.
Cyberpine receives a draft contract from the LRCT allowing installation of the necessary equipment on the Red Hill fire tower
Cyberpine receives a matching grant in the amount of 3000.00 to help construct the Red Hill access point from the NH Division of Economic Development through its Telecommunications Advisory Board.
Don Jones, former owner of the Sandwich Radio Network and valuable advisor, dies of a sudden heart attack.
Don Jones draws up a new network plan incorporating a new backhaul link to DSL in Meredith through Red Hill and recommends scrapping a totally bridged network in favor of a routed one.
Toby Eaton resigns after 1 1/2 years serving as Treasurer. Carroll Bewley volunteers to serve as treasurer. Jocelyn Gutchess invites LRCT President Don Berry for tea and reinforces the importance of high speed access for local businesses and residents, with emphasis on the critical role an access point and relay on Red Hill would be in Cyberpine’s ability to provide service.
Cyberpine office constructed at Sandwich Cabinet Shop
Cyberpine Cooperative, Inc. registered as a non-profit consumer/rural electric co-op with the NH Secretary of State’s Corporate Division, replacing Cyberpine Internet Service Provider, Inc.