Low Budget High Availability Network

[popeyespappy]

I work for a company owned by a small Native American nation. The tribe has several companies that specialize in different things. One that does logistics. One that does facilities support. One that does weather forecasting. Another that does security work. One that just provides back office support to the other companies. That kind of thing. Most of but not all of our business is with the government. 

Fucking Government IT security regulations are killing us. The expense has been astronomical. A couple of months ago the boss pointed at me and said, "Find a way to cut costs!" I wrangled a budget out him, got a lot of help from a couple of guys that don't know the first thing about systems administration, and we came up with this.







A bunch of used hardware running open source software. We are still testing and it hasn't gone live yet, but we are hoping to move everything over to this hardware in a couple or 3 months. 

There are 6 separate networks. A guest network. One for VOIP. A redundant LAN. A redundant LAN that connects the servers to the SAN's. A management network that just connects management ports. And a separate security management network.  

We have two separate ISP's. WAN A  is a symmetrical 1 Gig fiber. WAN B is copper based 1 Gig down and 35 Meg up. The routers are Dell R610's running PFSense. The servers are Dell R710's running CentOS. They are setup as a high availability cluster with four Dell R510's configured as SAN's. Each of the 510's has 24 TB of drives, but each box is setup for for RAID 6 and there are two mirrored pairs so about 40 TB of usable storage. I don't even know what the hardware is for the security server. It is whatever we were using before we bought the Dell we are using now. It is running Security Onion. It is the master node and has a VM storage node for logging. Each of the routers and servers have a VM configured as a forward node for NIDS running Snort and Bro, and HIDS running Wazuh. The Security Onion master does the analysis. It integrates the Sguil, Squert, Kibana and CapMe tools into a single console. 

One the harder problems we are working on is each company's VM has to be a stand alone domain. That requirement is written in stone due to the kind of organization we are. While we employees often do things for multiple companies, the companies have to be completely separate entities. We can't even appear to look like divisions of the same company. 

Right now we have spent about $20,000 on hardware including memory upgrades, lots of additional NIC ports, and a dozen reconditioned 2200 VA UPS units. That's only about a month's worth of our current IT costs. If we can get it running, and keep it running (the later being harder for us than the former) we will replace all the hardware over the next couple of years with new stuff. The UPS units alone for that will probably run us $30,000. 

Anyway that's where we are going right now, but if any of you who got through this TLDR post have suggestions on how we should do things differently I'd be happy to hear listen. As I said this is all being setup by a a few guys with little to no experience in systems administration, and we are learning as we go...