What is IP Log Management and why does it matter?

Every internet service provider sits on a firehose of connection records. IP log management is the discipline of capturing those records reliably, storing them safely, and being able to answer one deceptively simple question at any moment: who was using this IP address at this time?

The record you're required to keep

When a subscriber's device opens a connection, the router sees a flow: a source IP and port, a destination, a MAC address, and — behind it all — the PPPoE account that owns the session. In Bangladesh, the BTRC requires ISPs to retain this activity and produce it on demand. That means the record has to be complete (PPPoE ID, MAC, source and NAT IP, ports, precise timestamps), tamper-resistant, and retained for the mandated window.

Get any part of that wrong and the log is worthless when it's needed most — during an investigation or an audit, when you can't go back and re-capture what already happened.

Why it's harder than it sounds

At ISP scale the volume is enormous. A network of 20,000 subscribers can generate tens of millions of flow records a day. A naïve log store fills disks in weeks and grinds to a halt the first time someone runs a real search. So IP log management is really three problems at once:

• Capture — record every flow, enriched with the subscriber behind it, without dropping lines under load.
• Storage — keep months of it affordably, which means serious compression, not raw text files.
• Retrieval — find one subscriber's session among billions of rows in seconds, not hours.

What "good" looks like

A healthy IP log system captures in real time, resolves any IP-and-moment back to a subscriber, and hands you a clean, timestamped export you can give a regulator without editing. It stays fast as history grows, and it's honest about retention — you should know exactly how far back you can answer.

That's the bar InfinityLog is built to clear: real-time capture from your Mikrotik, cutting-edge storage at roughly 16 bytes per event, and sub-second search across a full day of traffic.

See it on your own traffic →

How to choose the right IP log management tool for your ISP

Not all log servers are equal, and the differences don't show up until you're mid-audit or mid-investigation. Here's what actually decides whether a tool will serve you for years — or become the thing you dread.

1. Compliance fit, first

Start with the record itself. Does the tool capture everything the BTRC expects — PPPoE ID, MAC, source and NAT IP, ports and accurate timestamps — out of the box, with templates tuned for Mikrotik? A tool that captures "most" of the fields is a tool that fails the one audit that matters.

2. Search speed at your scale

Ask for a demo on realistic volume, not a toy dataset. A search over a full day of a 20,000-subscriber network should return in about a second. If it takes minutes on a demo, it will take forever in production — and your team simply won't use it.

3. Storage economics

Retention is a cost, and compression is the lever. A legacy log store can need several times more disk than a modern, efficient one for the same logs — roughly 8× in our deployments. That’s the difference between a 16 TB HDD and a 2 TB SSD — and between a system that's slow on spinning rust and one that's fast on flash.

4. Real-time visibility

Compliance is the floor, not the ceiling. Live log streaming turns the same data into an operational tool — you spot an issue as it happens instead of reading about it afterwards.

5. Access control & accountability

Multiple people will touch the system. Role- and group-based permissions with activity tracking mean every lookup is scoped and auditable — important when the logs themselves are sensitive.

6. Who's behind it

A log server is infrastructure. You want a vendor who deploys it, supports it, and has done so for years across many ISPs — not a one-off install you're left to babysit.

InfinityLog was built against exactly this checklist, and it's run in production for 350+ ISPs across five countries.

Compare packages →

Top 5 benefits of implementing an IP log management solution

Beyond ticking the compliance box, a proper log server changes how an ISP operates. Here are the five wins operators tell us about most.

1. Audit-ready, always

When the regulator asks, you answer in minutes with a clean export — not a scramble through half-complete files. The record is already complete and already searchable.

2. Faster investigations

IP-to-subscriber attribution turns "who had this address?" from an afternoon of grep into a one-second query. That speed matters when a lawful request has a deadline.

3. Lower storage cost

Our cutting-edge storage shrinks the same logs by roughly 8×, so you keep more history on less — and cheaper — disk. Many operators move from bulk spinning disks to a small, fast SSD.

4. Real-time operational insight

Live streaming and traffic analytics surface top destinations, busy hours and anomalies — the same compliance data, working double duty for your NOC.

5. Peace of mind

Append-only storage, per-source retention and role-scoped access mean the record is trustworthy and the system is accountable. You stop worrying about the logs and get back to running the network.

Book a demo →

Why we rebuilt InfinityLog's storage from the ground up

For years InfinityLog stored logs the obvious way — a relational database, a table per day, a row per event. It worked, until it didn't. Here's why we tore out the storage layer and rebuilt it on a cutting-edge storage technology, and what changed for the ISPs running it.

The problem with a fat row per event

The old approach keeps every field of every event together on disk, wrapped in per-row overhead and indexes. For a handful of records that's fine. For an ISP writing tens of millions of flows a day it's brutal: disks fill in weeks, and a query for one subscriber has to walk enormous per-day tables that were never built for fast scans. Operators told us the same thing — the logs were there, but searching them was something you avoided.

A smarter way to store an event

Our cutting-edge storage technology groups similar values together and compresses them, instead of paying full price for a bloated row every time. The same event that once cost well over a hundred bytes now lands in roughly 16 bytes on disk. A network that used to need a 16 TB HDD comfortably fits on a 2 TB SSD — about 8× smaller — and because a search only reads exactly what it needs, a full day of traffic comes back in under a second. We don't publish the internals, and deliberately so: the efficiency is the product, and it's a hard thing for a competitor to copy.

Migrating without losing a single line

The catch: you can't take a compliance log server offline to re-platform it, and you can't lose history mid-cutover. So the migration is a dual-write: new events flow into the new store and the legacy store at the same time, reads are served from whichever holds the range you ask for, and only once the new store is proven do we retire the old write path. Ingestion never stops. No day goes dark.

What the operator actually feels

• Storage bills drop — months of retention on a small SSD instead of a rack of spinning disks.
• Search becomes something you use — investigations that were an afternoon are now a query.
• Nothing changes at the router — the same syslog stream, the same fields, the same compliance record.

None of this shows up on a spec sheet the day you sign. It shows up six months later, the first time a lawful request lands with a deadline and you answer it before the call ends.

See it on your own traffic →

How much storage does a BTRC-compliant log server really need?

It's the single question that decides your log-server hardware bill — and the one most vendors answer with a scary round number instead of the math. Quote you 10 TB of disk and the invoice takes care of itself. Here is the actual calculation, so you can size storage on your traffic instead of someone's spec sheet.

What actually drives the number

Log storage isn't driven by how many subscribers you have — it's driven by how many events they generate. Every NAT translation, every connection your router logs, is one record. A typical broadband subscriber produces on the order of 3,000–4,000 logged events a day. So the real formula is simple:

• Daily events = subscribers × events-per-subscriber-per-day
• Daily storage = daily events × bytes-per-event
• Total storage = daily storage × retention days

Two of those three numbers are fixed by your network. The one a vendor actually controls — the one that separates a 2 TB server from a 16 TB one — is bytes per event. That's where the technology choice lives.

The expensive way: a fat row per event

Older log servers store every event as a bulky database row or a line of text — timestamp, source IP, translated IP, ports, protocol — wrapped in per-row overhead and indexes. In practice that lands around 130–180 bytes per event on disk. Run the math for a modest 20,000-subscriber network:

• 20,000 subs × 3,500 events = 70 million events/day
• 70M × 150 bytes ≈ 10.5 GB/day
• Six months of retention ≈ 1.9 TB — and that's before backups, indexes and headroom

Now you see where "you'll need a 10 TB drive" comes from. Add a bigger customer base or a longer retention window and the requirement balloons into double-digit terabytes fast. It isn't that your network is enormous — it's that each event is being stored inefficiently.

The efficient way: cutting-edge technology

InfinityLog was rebuilt on a cutting-edge storage technology designed for exactly this shape of data. Instead of a heavy row per event, similar values are grouped and compressed together, so the same NAT record that cost 150 bytes elsewhere lands in roughly 16 bytes on disk — close to 8–10× smaller. The same 20,000-subscriber network above:

• 70M events/day × 16 bytes ≈ 1.1 GB/day
• Six months of retention ≈ 200 GB — comfortably a single small, fast SSD

A network that a legacy server says needs a 16 TB rack of spinning disks fits on a 2 TB SSD with room to spare. Same logs, same compliance record, same fields at the router — a fraction of the storage.

A quick sizing table

Rough six-month retention, comparing a legacy row-store to InfinityLog's cutting-edge storage:

• 5,000 subscribers — legacy ≈ 470 GB · InfinityLog ≈ 50 GB
• 20,000 subscribers — legacy ≈ 1.9 TB · InfinityLog ≈ 200 GB
• 50,000 subscribers — legacy ≈ 4.7 TB · InfinityLog ≈ 500 GB
• 100,000 subscribers — legacy ≈ 9.5 TB · InfinityLog ≈ 1 TB

These are planning figures — your exact events-per-subscriber and retention policy move them — but the ratio holds: the technology, not the subscriber count, decides whether you're buying gigabytes or terabytes.

Don't forget retention is a dial

BTRC compliance sets a minimum window you must keep; it doesn't force you to hoard forever. InfinityLog lets you set retention per source, so you keep exactly what the rules require and no more. Because each event is so small to begin with, even a generous retention window stays affordable.

The takeaway: before you buy disk to solve a log-storage problem, check whether you actually have a storage-efficiency problem. Most operators are being asked to pay for terabytes because their log server was never built to compress this data. Ours was.

Size it on your own traffic →

Terabytes vs. gigabytes: why ISP log servers buy too much disk

A storage-hungry log server doesn't just cost more the day you buy it. It quietly taxes you every month — and then it fails you at the exact moment you can't afford it to. If a vendor's first answer to "what do I need?" is a stack of multi-terabyte drives, that's worth a second look.

The bill you see on day one

When a log server stores every event as a bulky row, terabytes become mandatory. That means bigger drives, more of them, and usually spinning disks instead of flash — because SSD at that capacity is expensive. So you're paying for hardware you only need because the software is inefficient. It's a cost, but it's the smallest one.

The bills you don't see

• Backups multiply. Every terabyte of primary storage is another terabyte to back up. A 16 TB log store needs a 16 TB backup target; a 200 GB store fits on media you already own.
• Search gets slow. Investigations run against big spinning disks are painful — a lawful-intercept request that should take a second turns into minutes of grinding through enormous per-day tables. Storage bloat isn't just a capacity problem; it's a speed problem.
• Growth compounds. Add subscribers or extend retention and an inefficient store grows terabytes at a time. An efficient one grows gigabytes at a time. The gap widens every month you operate.

The failure mode nobody quotes

Here's the one that actually hurts: a full disk stops your logging. When a storage-heavy server fills up — and oversized stores fill faster than anyone plans for — ingestion halts. The router keeps sending, but nothing lands. That's not an inconvenience; on a compliance log server it's a gap in the record, exactly the thing the whole system exists to prevent. We've seen it happen in the field, and the root cause is almost always the same: a store that was always going to run out because it was never efficient to begin with.

The lever is efficiency, not capacity

The fix isn't to keep buying bigger drives — it's to store each event in a fraction of the space. InfinityLog's cutting-edge storage technology compresses the same compliance data into roughly 16 bytes an event, about 8× smaller than a traditional row-store. That single change cascades through every cost above:

• Months of retention live on a small, fast SSD instead of a rack of HDDs.
• Backups become trivial because there's simply less to copy.
• Search returns in under a second because it reads only what it needs.
• The disk stops being the thing you nervously watch, because it's no longer filling.

How to size it honestly

When you compare log servers, don't compare total storage — compare storage per subscriber. Ask each vendor: for my subscriber count and my retention window, how many bytes per event do you store, and what disk do you actually require? The answer tells you whether you're buying a well-built system or subsidising an inefficient one with hardware.

The takeaway: terabytes aren't a sign of a serious log server — often they're a sign of an inefficient one. The right technology turns a terabyte problem into a gigabyte one, and takes the "will it fill up?" question off your plate for good.

Compare InfinityLog packages →