Blog In Progress

Bloginprogress.us is back.

When I replaced the hardware I had to fight with AT&T and 2Wire to get the public IP working again. See the post on Posterous.

Should be back in business now.

David

[Update: Connectivity still seems a bit flaky. Comes and goes. If anyone has experience with the 2Wire Router on AT&T DSL with static public addresses I would love to talk to you about it.]

The project begun a couple of months ago to consolidate and virtualize servers and improve availability is coming to fruition. The timing on this isn't random. The company is moving to another building with a much smaller server room -- really just a data closet -- at the end of the month.

We have moved a Domino mail server, BES, Sametime, and just this week Quickr, into a VMWare host at a colocation facility. On site we moved the primary Domino mail and app server (cluster mate of the one at colocation), file server (Novell OES2 Linux) and a couple of small utility servers (DHCP and SMTP that used to run on old laptops) onto a VMWare host. The VMWare box is in a nifty server cabinet on wheels from APC. Also in the cabinet are the UPS, a couple of 1U NetWare servers we already had, and a Cisco Call Manager. That's basically the infrastructure, besides for a small rack of switches, routers and firewalls. And one remaining old-school tower server -- Cisco Unity for voice mail/unified messaging.

Moves are always a challenge but at least physically moving the server room will be a piece of cake! And while the server room is being reassembled we'll still have email, Blackberry mail, Sametime and access to Quickr files via the colo.

What did I learn?

VPNs are amazing things but they do add a layer of complication to any network troubleshooting.

As I mentioned earlier, the habit of working at the server console to "really see what's going on" is hard to break, but except for the couple of NetWare boxes there's almost nothing to touch in the server room. I may as well stay at my desk and get familiar with Remote Desktop, Putty, VNC and VMWare vSphere client. And of course live console in the Domino Admin client.

All of the machines in the VMWare hosts were more or less full installs: install the OS, install the server apps, maybe copy over the Domino folder structure. But the Quickr was done using a physical-to-virtual tool. Two lessons there: 1. Use a really fast disk if you don't want to sit around all night. 2. We caught and updated every place an old IP address was hard coded in Windows network setup, hosts files, Domino connection documents, etc. But we missed one spot in Quickr itself where it referred to itself by IP address to direct LDAP lookups. Oops. We now have a FQDN there.

Besides the LDAP issue on Quickr the only real problem was with BES, which I've already blogged. In a nutshell, BES was happy on its old hardware but after the migration we had to upgrade to the latest versions of everything to keep it running.

Credit where credit is due: The VMWare and infrastructure aspects were all handled by Advanced Computer Technologies in Norwalk, CT. Thanks guys.

Stay tuned for the next pull-out-my-hair challenge: Make Mac OSX an equal option for a corporate desktop.

I recently went to follow the link in this item from a weekly announcement email from IDG Connect:


IBM: Closing the Data Privacy Gap: Protecting Sensitive Data in Non-Production Environments
Find out why protecting data is so important and how to do it.


When I did so I landed on a registration link with the all of the contact information -- name, address, email, phone, job function, industry, etc. -- already filled in. This didn't bother me too much since I'm already registered with IDG. But what did bother me was that most of the information was mine but some was not, including someone else's street address and phone number!

Just an update on my earlier post. Since upgrading the BES software to 4.1 SP7 FP3 (4.1.7.24) and upgrading Domino on the BES and the mail servers to 8.5.1FP3 we've seen no more problems with the BES task hanging. Why this was necessary, since the previous physical server seemed pretty much identical to the new virtual server, we may never know.

By the way, I had somehow missed the fact that there was a SP7 for BES Domino 4.1. The big deal there is that SP7 is supported with Domino 8.5.1 and SP6 is only supported through Domino 8.5. See the April 8, 2010 update of the Compatibility Matrix.

David

VERY CONNECTED - PROCOMM PLUS is the only complete solution to access all your PC communications from one source.


I was cleaning out old hardware and software from shelves in the office and found a complete copy of PROCOMM PLUS in the original carton.  The world awaited, all you needed was modem and the included Winsock 1.1 compatible TCP/IP stack!

Read more

A couple of days ago we migrated a BES to new hardware -- from a stand alone fairly plain vanilla box to a virtual machine in VMWare ESXi. It was BES 4.1.6 on Domino 8.5 and Windows 2003 before and after. The new setup should be faster and have more memory. It's a simple setup for twenty users and we followed RIM's guidelines for a "knife edge" cutover. But, of course, we have a weird problem on the new server which didn't have on the old.

Periodically the NBES.EXE process takes up so much memory that it halts all BES processing, although Domino goes on happily. If you catch it in time and restart BES then the memory footprint goes way down. If you don't then you have to force quit Domino (NSD.EXE -KILL). But then Domino and BES are happy to restart, no reboot needed.

RIM hasn't had much to offer. IBM has an SPR that seems to match our symptoms, but it's for BES 5 on Domino 8.5.1 -- and the recommended fix is to apply FP3 for Domino 8.5.1 when it comes out next week. BES 4.1.6 isn't supported on Domino 8.5.1

The SPR says the trigger is converting bitmaps to GIF on outbound Internet mail. We use a lot of bitmap signature files so that makes sense, but it's not new since we moved the server. It also would match our symptoms if the memory use grows with heavy email traffic, but again, the traffic hasn't changed.

Lovely thoughts to ponder as I spend my holiday weekend periodically restarting the BES task.

Update 5/31:
Upgraded to BES 4.1.7.24. BES 4.1.7 is supported under Domino 8.5.1 so I have Domino up to 8.5.1 FP3. We'll see if we do any better with that combination.

I'm in the midst of some big changes at the Lotus site I support.

We're consolidating most of the servers into two VMWare boxes, one on site and one at a colocation facility. It turned out that we really couldn't move everything off site. The Cisco phone system needed to be on site, and a Domino mail server to receive the voice mail messages needs to be local to the phone system. And it didn't make any sense to move the file and print servers off site. So we're going with the hybrid approach.

File and print was via Novell NetWare. We looked at moving to Windows Server and AD but the disruption to the client side didn't seem worth it, so we have a new Novell OES2 (Linux) server on the VMWare box in the office. It's a very nice solution but it takes a bit of adjusting to and we needed to clean up some things on the NetWare side that we never noticed until we had to talk to OES. For example: missing certificates to make LDAP fully functional. It is pretty neat to log a computer in and see it treat the NetWare and OES boxes the same. We can also connect to the new box without the Novell client using Windows SMB client and, although we haven't tested this yet, should be able to also connect from Mac OSX. And we did some volume consolidation so the machines not running a login script really only need to access a single volume now.

With the VMWare I'm trying to get out of the habit of walking into the server room to work from the physical consoles. I really can do almost everything from my desktop/laptop and from off site. But it's a hard habit to break. There is great reassurance in watching the messages scroll up the Domino consoles and watching the combination screen saver/CPU usage meter on the NetWare machines.

Moving the primary Domino mail server from dedicated hardware onto the on site VMWare was fairly simple and painless. Now I need to move a second Domino mail server, plus a Sametime 8.0.1 server and a BES Domino server onto the off site VMWare. That will be a bit more challenging I think. Among other issues it will involve changing public IP addresses and DNS entries.

Both VMWare servers (HP) are sized so we can add additional virtual machines as the need arises. For example, we think we'll need some sort of SQL server for the next iteration of ERP/Accounting.
And if we ever want to go to Sametime 8.5 at least we'll have a place to put all those servers.

I'll post again when (if?) the project concludes. Advice and encouragement are always welcome. Messages of "You idiot, why didn't you do it the right way which is..." will be tolerated according to the spirit in which they're offered.

There is now a Blackberry application for LinkedIn. Before you could only use LinkedIn on your Blackberry via a fairly limited WAP browser page. To get the new app follow this link on your Blackberry and click "Get it Now"
http://www.linkedin.com/blackberry?trk=BlackBerry

Or you can go here from your computer and ask to have a download link sent to your device:
http://m.linkedin.com/blackberry/email

There's been a lot of complaining about the new app on LinkedIn in the LinkedIn for Blackberry group. I had no problems installing or connecting at first, but as time goes on it seems to be getting a bit flaky with lots of error messages and being unable to refresh. I don't know if that's an issue with more and more people using it. There's also been some discussion about it working better on certain connections. I have a Curve 8320 from AT&T which is pretty slow on EDGE so I try to grab WiFi whenever it's available. I've had some connection issues this weekend from home where the EDGE connection is marginal. But until I test some more I can't say what's the determining factor as to when it works smoothly and when it doesn't.

LinkedIn for BB seems to be a fairly easy way to follow LinkedIn updates or look up a contact. It's not really a substitute for the web interface though.

The app offers to link your Blackberry and LinkedIn contacts but again there's been a lot of conflicting opinions. Since my Blackberry contacts links back to my Notes contacts, and from there to the company CRM system, I'm reluctant to play around too much with that feature.

The internal address of the primary Domino mail server is 192.168.1.2. It seems some folks have their home router set to assign that address. When the user brings his laptop into the office the address is recognized as unavailable and the laptop is assigned a new address. It takes about 2 seconds (see DHCP server log except below).

6:48:40 AM DHCP:DHCPDISCOVER from 00:16:d3:25:15:ab via 192.168.1.67
6:48:41 AM DHCP:DHCPOFFER from 192.168.1.208 from 00:16:d3:25:15:ab from 192.168.1.67
6:48:41 AM DHCP:DHCPREQUEST from 192.168.1.208 from 00:16:d3:25:15:ab via 192.168.1.67
6:48:41 AM DHCP:DHCPACK from 192.168.1.208 from 00:16:d3:25:15:ab from 192.168.1.67

But that's enough for anyone working on a mail message in their server replica to get a "Server not responding". The Notes client does not offer to switch to the cluster mate. Instead the user must hit Ctrl-Break, and can then continue working normally.

Anyone else seen this? Is this working as designed?