Had an interesting time today - installed the Windows forwarder on our Domain Controllers. I know Security event logs are busy but holy shit do those things log some stuff even for an SME like ourselves Does anyone know how to tell the agent not to send all the existing stuff in the event logs to Splunk when you install the agent as that put us way over the 500MB trial license in no time. I do not know how to exclude past logs other than clearing them before you install the agent. Even not taking the import of past logs, our domain controllers generate WAY WAY more data than our paid license can handle.
Shame too, as I know that data is really interesting. A lot of it is just noise though; we have a number of regexes on our indexers that will nullQueue specific Windows Security log events. We've got 5,000 users on a daily basis and we're able to get the DC's logs (plus those from all our other servers/switches) into about 3-4GB a day. A lot of what we filter out is just the machine-to-machine auth - stuff like a workstation authenticating to a DC using its computername$ account. One specific example is that we have a transform on the indexers that is set to watch for event code 4624 (account logon), then do a regex to see if the username matches our computer name convention, followed by a $ (indicating it's the machine account). What ends up being left are just logins with AD user accounts. I will admit that for as much as we've pruned out, we've added back in from other logs.
For example, we enabled additional AD logging so that we can audit what kinds of changes our staff are doing to user accounts and groups (password resets, adding people to groups, etc.). Does anyone know how to tell the agent not to send all the existing stuff in the event logs to Splunk when you install the agent as that put us way over the 500MB trial license in no time. Speaking of which, does anyone have any even vague guidelines for expected sizing? We're an SMB with a dozen server VMs that matter and 200 workstations, and I just have no clue how much logs these things would be shooting over to a Splunk install. The only way I could get The Man to buy me paid Splunk would be to show off the awesome with the trial license, but I haven't got the time if it turns out we're generating 300GB of logs per day. Gut reaction is that 500MB/day won't do it.
For just 3 DCs (one is at a ROBO so light usage) we're averaging about 1.2GB/day I think. VMware and our switches is minimal. I still think it depends what your objective is - if it's to log everything it can easily get expensive, if it's to log stuff from a particular viewpoint i.e. Security it can easily get expensive, but it's possible to be a little more clinical in what you throw at it. Splunk Light may also be worth a look.
No apps or AD integration, works out basically at 50% the price of Enterprise. Does anyone know how to tell the agent not to send all the existing stuff in the event logs to Splunk when you install the agent as that put us way over the 500MB trial license in no time. Speaking of which, does anyone have any even vague guidelines for expected sizing? We're an SMB with a dozen server VMs that matter and 200 workstations, and I just have no clue how much logs these things would be shooting over to a Splunk install.
The only way I could get The Man to buy me paid Splunk would be to show off the awesome with the trial license, but I haven't got the time if it turns out we're generating 300GB of logs per day. You're not to far off in size from my network. Logging everything off of the 20 Windows servers blew so far past the daily quota.
The amount of logs that Windows servers generate is. Or oppressive depending on your job. That's why we whitelist the IDs we really want to see; for instance: the file delete events I want to track and be able to search. Knowing who and when deleted or moved something on a file server is extremely useful. On the free license past maybe 5-8 Windows servers, you are going to have to control input with whitelisting or blacklisting.
Does anyone know how to tell the agent not to send all the existing stuff in the event logs to Splunk when you install the agent as that put us way over the 500MB trial license in no time. Speaking of which, does anyone have any even vague guidelines for expected sizing? We're an SMB with a dozen server VMs that matter and 200 workstations, and I just have no clue how much logs these things would be shooting over to a Splunk install. The only way I could get The Man to buy me paid Splunk would be to show off the awesome with the trial license, but I haven't got the time if it turns out we're generating 300GB of logs per day. If you are into VMware, it would be worth taking a look at their Log Insight product. It will quite likely to work out cheaper. We ingest about 10-15GB per day into Splunk mainly for application log analysis.
It was looked at for the SIEM requirement but we'd have to scale up to about 200GB per day for just a first round of useful info. That was too much by the time we looked at infrastructure to run that in HA and the license was about double the hardware costs. Instead we went for a SIEM option that was appliance based and I'm sure that shortened the time to usefulness by a month and has a UI that seems more responsive than our Spunk front ends. Saying that it is a pretty cool piece of software that can produce some great info from all your data. We ingest about 10-15GB per day into Splunk mainly for application log analysis.
Splunk Software Download
It was looked at for the SIEM requirement but we'd have to scale up to about 200GB per day for just a first round of useful info. That was too much by the time we looked at infrastructure to run that in HA and the license was about double the hardware costs. Instead we went for a SIEM option that was appliance based and I'm sure that shortened the time to usefulness by a month and has a UI that seems more responsive than our Spunk front ends. Saying that it is a pretty cool piece of software that can produce some great info from all your data. When implementing Splunk's Enterprise Security offering, getting some professional services for the deployment is a no-brainer. At $EMPLOYER-1, our security team was over the moon on day one of the ES install with the visibility they were getting.
They started with a 150 GB/day purchase and doubled it to 300 GB/day six months later. Where I'm at now, I'm administering an 1800 GB/day Splunk install.
Splunk is expensive, both in licensing and hardware requirements. But the ROI is very good, especially if you give key users the training and time to really dig into the potential of the product. We ingest about 10-15GB per day into Splunk mainly for application log analysis.
It was looked at for the SIEM requirement but we'd have to scale up to about 200GB per day for just a first round of useful info. That was too much by the time we looked at infrastructure to run that in HA and the license was about double the hardware costs. Instead we went for a SIEM option that was appliance based and I'm sure that shortened the time to usefulness by a month and has a UI that seems more responsive than our Spunk front ends.
Saying that it is a pretty cool piece of software that can produce some great info from all your data. When implementing Splunk's Enterprise Security offering, getting some professional services for the deployment is a no-brainer. At $EMPLOYER-1, our security team was over the moon on day one of the ES install with the visibility they were getting. They started with a 150 GB/day purchase and doubled it to 300 GB/day six months later. Where I'm at now, I'm administering an 1800 GB/day Splunk install. Splunk is expensive, both in licensing and hardware requirements. But the ROI is very good, especially if you give key users the training and time to really dig into the potential of the product.
Comparatively speaking, compared to other players in that space, Splunk is on the cheaper side. We ingest about 10-15GB per day into Splunk mainly for application log analysis.
It was looked at for the SIEM requirement but we'd have to scale up to about 200GB per day for just a first round of useful info. That was too much by the time we looked at infrastructure to run that in HA and the license was about double the hardware costs. Instead we went for a SIEM option that was appliance based and I'm sure that shortened the time to usefulness by a month and has a UI that seems more responsive than our Spunk front ends. Saying that it is a pretty cool piece of software that can produce some great info from all your data. When implementing Splunk's Enterprise Security offering, getting some professional services for the deployment is a no-brainer. At $EMPLOYER-1, our security team was over the moon on day one of the ES install with the visibility they were getting.
They started with a 150 GB/day purchase and doubled it to 300 GB/day six months later. Where I'm at now, I'm administering an 1800 GB/day Splunk install. Splunk is expensive, both in licensing and hardware requirements.
But the ROI is very good, especially if you give key users the training and time to really dig into the potential of the product. Comparatively speaking, compared to other players in that space, Splunk is on the cheaper side.
Splunk Enterprise License
I keep forgetting how expensive SEIM solutions are. I'm more used to Splunk as a centralized logging tool. There's some sticker shock from that perspective. I know every time I start talking hardware requirements for a deployment, I get funny looks. I now have a 'simple' decision to make - Light or Enterprise? The greedy part of me says go Light and get the additional capacity.
The part of me that is thinking 'analysis' thinks it probably wouldn't take long before I'll miss being able to use apps and do the smarter stuff I know Splunk can do, even if we're not doing it right now. I'd check with sales to see what exactly they mean by 'Packaged Apps', because it may just mean the big for-pay apps Splunk sells.
(Enterprise Security, VMWare, ITSM, etc) If that's the case, those are way beyond the capabilities of a single-server install anyway. If the free/community apps are fair game, that's fine for an initial install. The biggest concern I have with Splunk Light is that a single-server install isn't what I would consider the minimum for a production Splunk deployment. If there is any expectation of growth, I would recommend starting with a dedicated search head and a single-node indexing cluster. With that set-up, you're starting with the framework of a distributed deployment, and can scale quite a bit without really needing to re-architect or shuffle data around. (It's way easier to add an indexer to a one-node cluster than it is to migrate from a standalone install to a cluster) Beyond that, the lack of SSO and role-based access controls, the 5 user limit, and the lower support tier are all considerations.
Gut response is to say that if Enterprise is even remotely on the table, you should go that route. Splunk is like crack, and it's usage will grow faster than you think.
(Also, if you go with an Enterprise deployment, I highly recommend springing for some Professional Services for the initial deploy). I'm giving this a test run. Is there way to make Splunk only go out and grab log file info every 15 minutes instead of continuously? I am sure the network load on the targets is light, but I still don't have need for Splunk to hit all my servers 24/7. I think the relation is the other way around. The Splunk server is receiving the data, it's not going out to your servers. As a rSyslog setup, the server daemon gets the input for log and it writes and sends out the network port to Splunk at the same time.
My experience is only with getting info to Splunk with syslog and the universal forwarder, so it could be that there is a forwarder that does poll but I'm not aware of one. I did a P.O.C. Of Splunk by spitting out the logs from our Palo Alto Networks firewalls as well as from a few file servers and domain controllers. I think Splunk has a slick interface, but on the whole it seems incredibly 'fiddly'.
It took me quite a bit of time just getting the domain controller logging to work correctly. The amount of data generated was way beyond the free license and we had to get a trial license. Unfortunately, they don't do trial licenses for the for-pay plugins, which I think is a little ridiculous. Eventually, something happened with our VM and it just would get a kernel panic on boot. I didn't feel like fixing it and just killed the VM. We still have no SIEM and are looking. LogRhythym, Splunk, and IBM Qradar are all possibilities.
Anyone have any experience with LogRhythym or Qradar? They seemed really decent from the presentations we saw, especially Qradar. Unfortunately, they don't do trial licenses for the for-pay plugins, which I think is a little ridiculous. Eventually, something happened with our VM and it just would get a kernel panic on boot. I didn't feel like fixing it and just killed the VM. Get connected with their sales drones.
(Who will then get you connected with a sales engineer) There are definitely trial licenses for their paid apps, but you can't just get them from the website. I've had them assist with setting up POC's for the VMWare app and Enterprise Security. (Granted, we were existing customers, but we were evaluating those add ons). Unfortunately, they don't do trial licenses for the for-pay plugins, which I think is a little ridiculous. Eventually, something happened with our VM and it just would get a kernel panic on boot. I didn't feel like fixing it and just killed the VM.
Get connected with their sales drones. (Who will then get you connected with a sales engineer) There are definitely trial licenses for their paid apps, but you can't just get them from the website. I've had them assist with setting up POC's for the VMWare app and Enterprise Security. (Granted, we were existing customers, but we were evaluating those add ons) I was talking to sales.
They wouldn't allow me to do a trial on the security plugin for whatever reason. The guy told me they do trials on the core data ingestion licenses, but not for anything else. Unfortunately, they don't do trial licenses for the for-pay plugins, which I think is a little ridiculous. Eventually, something happened with our VM and it just would get a kernel panic on boot. I didn't feel like fixing it and just killed the VM.
Get connected with their sales drones. (Who will then get you connected with a sales engineer) There are definitely trial licenses for their paid apps, but you can't just get them from the website. I've had them assist with setting up POC's for the VMWare app and Enterprise Security. (Granted, we were existing customers, but we were evaluating those add ons) I was talking to sales. They wouldn't allow me to do a trial on the security plugin for whatever reason.
The guy told me they do trials on the core data ingestion licenses, but not for anything else. Guess that changed then. EDIT: Looking back, we didn't get a trial on ES.
I misremembered. Probably because it's a beast to implement.
(We had a professional services rep on site for three weeks to manage the initial rollout) We did POC VMWare, but that's relatively simple to set up. (They've got pre-defined images that handle a majority of the collection) I would also say that ES would be.really. jumping into the deep end of Splunk.
You need to be collecting everything to get value out of it. But if you are, it's pretty fantastic. We're starting to explore options for centralised logging of 'stuff'.
Words like SIEM have been mentioned but I suspect that may be a simple route to a never ending journey - in reality I think we simply need some decent visibility of what our 'stuff' is doing, and the ability to flag on certain events, stuff being: Firewalls Switches Servers (mostly Windows with some Linux) Windows Endpoints - well do people really collect data from these? And events being. Well I'm not 100% sure if I'm honest which is where you start to think about SIEMs that supposedly have nice templates and 'correlation engines' built in that do the work for you - magic beans?
I started off thinking Splunk would be crazy expensive, but when I think that they only care about ingest and not about users, endpoints, devices, cores, sockets etc. It kind of looks like it might actually be crazy cheap. It took about 15 minutes to get working and 10 of those were setting up the VM to install it on and registering to download it - I'd love some example of what you're using it for and how you're using it to generate useful, actionable notifications and alerts. Proceed with caution with Splunk and large implementation.
Their bias is for you to log more crap, and let the search functionality do it's thing. Your interest is to log what you need.
So for SIEM, follow the NSA guidance for Windows servers and customize from there. Is Graylog worth looking at for an open-source solution? We're a small business with just a handful of servers and applications that we are beginning to realize we need to keep better track of. Probably a dozen VMs, some AWS servers, a bunch of Cisco phone gear, routers and switches, and 60 users' worth of other associated stuff. We could pay for a top-tier solution if we really needed it, but I wonder what's out there for folks that just need something small. If you don't need the analytics that splunk offers and just want a good centralized logging solution, I highly recommend it over something like the ELK stack.