[Ntop] n2disk automated capture archive workflow

[Raoul Duke]

Hi list,

I'm evaluating n2disk for a specific use-case but I'm not clear how to map
my workflow to its mental model so I wanted to ask your advice.

my desired workflow is:

* I start capturing with n2disk. I'm using the -I option to make
on-the-fly index
* when a certain session event occurs in my application I want to archive
the pcap for that session (in the event I will know which remote UDP
ip/port were used in the session)

I have had success performing the above steps manuall/independently but my
attempts to automate them have become protracted and I wanted to ask if
perhaps I'm missing something simpler way to do this.

the main issue I have is that when my application event occurs (lets say I
know that a given application session $sessionid just ended involving
remote $ip and $port) then I'd like the knowledge of that application
session ending to trigger a npcapextract for $ip and $port to archive the
session under a filename of $sessionid. I have all these variables and
have a script which is ready to automatically run the extract...

the problem I have is:
* n2disk has not flushed the data to disk yet and so I can't run the
extract yet. so how can I know when it is safe to run the extract?
* I read in release notes that it was possible to us " kill -USR1 to close
and flush the current pcap in order to make live traffic immediately
available" which works but I notice every time I call it it generates a new
index file. Which then leads me to the question of: how do I know which
index file to run npcapextract against? e.g. if the latest index was 1.idx
and I do a kill -USR do I have to guess that my application events would be
found in 1.idx / 1.pcap or is there a another way to do this?

* looking at all this another way. I'd be happy to defer the npcapextract
until the data is naturally flushed to disk. but this leads me to 2
questions:
- how can I know when all the relevant data is flushed to disk so I can
take action on the npcapextract? e.g. is there some concept of a
hook/trigger I can call when pcap / index data is flushed to disk?
- how can I know which index file is the "current" one

I am new to n2disk so I am likely just coming at this with some flawed
mental model. I hope at least my question articulates what I'm trying to
accomplish.

I'd be very grateful for any input on how I can accomplish my use-case.

Thanks,
RD

[Raoul Duke]

Hi Alfredo,

Post by Raoul Duke
* I read in release notes that it was possible to us " kill -USR1 to
close and flush the current pcap in order to make live traffic immediately
available" which works but I notice every time I call it it generates a
new index file. Which then leads me to the question of: how do I know
which index file to run npcapextract against? e.g. if the latest index
was 1.idx and I do a kill -USR do I have to guess that my application
events would be found in 1.idx / 1.pcap or is there a another way to do
this?
I recommend you to enable the timeline, and just specify the time interval
in npcapextract, using the timeline as data source instead of the specific
pcap/index.

Thanks for this useful info. I will experiment with timeline.

Post by Raoul Duke
* looking at all this another way. I'd be happy to defer the npcapextract
until the data is naturally flushed to disk. but this leads me to 2
- how can I know when all the relevant data is flushed to disk so I can
take action on the npcapextract? e.g. is there some concept of a
hook/trigger I can call when pcap / index data is flushed to disk?
You probably need to know what is the timestamp of the last packet dumped
to disk, maybe we can write it under /proc/net/pf_ring/stats/<n2disk
stats>. If this works for you we can add it to the features list.

this seems like a good feature to have in the general case could be
potentially used in my case.

another idea I had was: is there an option to control the flush frequency?
e.g. lets say I captured 100 packets but they have not yet flushed to disk
and no traffic happens for (say) 10 more minutes. would there still be no
flush to disk during that 10 minutes? i.e. would the flush only happen
when sufficient traffic has occured to fill up the memory buffer or is
there a way to say "always flush to disk every X seconds". in that way I
could defer the npcapextract for X seconds after I know the application
session has ended and could guarantee the packets would be flushed to disk
by then.

Thanks so much for such a quick and helpful response.

RD