[Ntop] PF_RING FT API implementation

Discussion:

Harish Patil

2018-06-08 02:00:35 UTC

Hi,

I am new to ntop/ndpi.
I'm looking for source code of PF_RING FT such as implementations of:
pfring_ft_create_table() or pfring_ft_process() etc which uses ndpi lib.

I only see the function defines in userland/lib/pfring_ft.h but not the
implementation.
Could you pls provide some pointers?

Thanks,

Harish

Harish Patil

2018-06-11 16:10:31 UTC

Permalink

Hi Emanuele,
Thanks for the response.
I didn't know that it was not open sourced and didn't expect it either
given that other modules are open sourced.
Are there any plans of open sourcing them?

Thanks.

Hello,
Sorry but PF_RING FT is not open source.
Regards,
Emanuele
Hi,
I am new to ntop/ndpi.
pfring_ft_create_table() or pfring_ft_process() etc which uses ndpi lib.
I only see the function defines in userland/lib/pfring_ft.h but not the
implementation.
Could you pls provide some pointers?
Thanks,
Harish
_______________________________________________
_______________________________________________
Ntop mailing list
http://listgateway.unipi.it/mailman/listinfo/ntop

Harish Patil

2018-06-12 23:40:11 UTC

Permalink

Hi Alfredo,

Thanks for the response.
We are planning to use PF_RING FT in our projects.
We only have shared libs. How can we integrate and make sure it works in
all cases (platforms/GCC)?
If we had the sources we could compile along with sources and would be much
better.

Thanks.
Harish

Hi Harish
no plan atm, however we tried to create a flexible framework, meaning
that people should be able to extend it using the API we created.
Please let us know if something is missing for your use case.
Thank you
Alfredo
Hi Emanuele,
Thanks for the response.
I didn't know that it was not open sourced and didn't expect it either
given that other modules are open sourced.
Are there any plans of open sourcing them?
Thanks.

_______________________________________________
Ntop mailing list
http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
http://listgateway.unipi.it/mailman/listinfo/ntop

Harish Patil

2018-06-22 02:32:29 UTC

Permalink

Thanks Alfredo/Emanuele.

I am seeing the below message:

#########################################################################
# ERROR: You do not seem to have a valid PF_RING FT 7.3.0.180617 license
[ECFE456A76066A02]
# ERROR: Please get one at http://shop.ntop.org/.
#########################################################################
# We're now working in demo mode with packet capture and
# transmission limited to 5 minutes

Is there a need to buy a licensed version? Is this newly introduced in the
later releases?

Few questions regarding the PF_RING FT API behavior:

1) Does pfring_ft_process() makes a copy of the packet using the pointer
passed to it? If so, does it copy only the header or entire packet
including payload?

2) Is it safe for me to release/free the buffers after calling
pfring_ft_process()?

3) Do you have performance numbers doing inline DPI for 1G or 10G links?

4) Does PF_RING FT/nDPI maintain state information to detect a flow OR does
it just uses the packet passed to parse L7 info?

Thanks

Harish

Hi Harish
we are currently using those libs to compile our applications on many
distributions
including Ubuntu 14/16/18, Centos 6/7, Debian 7/8/9, please let us know if
you
experience issues with the compilation, and we will find a way to handle
that.
Regards
Alfredo
Hi Alfredo,
Thanks for the response.
We are planning to use PF_RING FT in our projects.
We only have shared libs. How can we integrate and make sure it works in
all cases (platforms/GCC)?
If we had the sources we could compile along with sources and would be
much better.
Thanks.
Harish

Harish Patil

2018-06-22 14:45:07 UTC

Permalink

Thanks Alfredo.
Replies below.

Post by Harish Patil
Thanks Alfredo/Emanuele.
#########################################################################
# ERROR: You do not seem to have a valid PF_RING FT 7.3.0.180617 license
[ECFE456A76066A02]
# ERROR: Please get one at http://shop.ntop.org/.
#########################################################################
# We're now working in demo mode with packet capture and
# transmission limited to 5 minutes
Is there a need to buy a licensed version? Is this newly introduced in the
later releases?
Yes, PF_RING FT, included in latest stable, requires a license.

So at least for now if we stick to the older version then we don't need a
license, right?

Post by Harish Patil
1) Does pfring_ft_process() makes a copy of the packet using the pointer
passed to it? If so, does it copy only the header or entire packet
including payload?
No packet copy at all.

In that case, I can only store packet headers in my local DPDK mempools,
right? The idea is to minimize mempool sizes.

Post by Harish Patil
2) Is it safe for me to release/free the buffers after calling
pfring_ft_process()?
Yes.

Post by Harish Patil
3) Do you have performance numbers doing inline DPI for 1G or 10G links?
You should be able to do 10Gbit line-rate, however this also depends on
your hw. Please note you can use multiple threads (with RSS or ZC software
distribution) if 1 core is not enough.
4) Does PF_RING FT/nDPI maintain state information to detect a flow OR
does it just uses the packet passed to parse L7 info?
It keeps some state, as 1 packet is often not enough.

Can you please provide any pointers on that?
Another question: Do you know if there is any issues using pf_ring ft lib
as a dynamic module in DPDK secondary process?

Post by Harish Patil
Regards
Alfredo
Thanks
Harish
On Wed, Jun 13, 2018 at 12:14 AM, Alfredo Cardigliano <

Hi Harish
we are currently using those libs to compile our applications on many
distributions
including Ubuntu 14/16/18, Centos 6/7, Debian 7/8/9, please let us know
if you
experience issues with the compilation, and we will find a way to handle
that.
Regards
Alfredo
Hi Alfredo,
Thanks for the response.
We are planning to use PF_RING FT in our projects.
We only have shared libs. How can we integrate and make sure it works in
all cases (platforms/GCC)?
If we had the sources we could compile along with sources and would be
much better.
Thanks.
Harish
On Mon, Jun 11, 2018 at 9:20 AM, Alfredo Cardigliano <

Harish Patil

2018-06-27 17:08:14 UTC

Permalink

Hi Alfredo,
Thanks for the response.
Regarding performance question #3.
When you say 10G line rate, are you saying 10G line rate with single core
and is it true for all frame sizes?
Assuming standard Intel 10G NIC (like 82599) which can do line rate @64B
over DPDK/PF_RING, do we have test results that shows how many ports/cores
needed for inline DPI?

Another question: I happen to run some HTTP traffic over DPDK (doing just
L2 forwarding) and did not see any flows detected (using PF_RING FT APIs).
Do you know what typically could cause that and any debug info that can be
gathered?

Thanks,
Harish

Post by Harish Patil
Thanks Alfredo.
Replies below.

Post by Harish Patil
Thanks Alfredo/Emanuele.
#########################################################################
# ERROR: You do not seem to have a valid PF_RING FT 7.3.0.180617 license
[ECFE456A76066A02]
# ERROR: Please get one at http://shop.ntop.org/.
#########################################################################
# We're now working in demo mode with packet capture and
# transmission limited to 5 minutes
Is there a need to buy a licensed version? Is this newly introduced in
the later releases?
Yes, PF_RING FT, included in latest stable, requires a license.

So at least for now if we stick to the older version then we don't need a
license, right?
1. PF_RING does not require a license, if you enable PF_RING FT, which is
an extension, that requires a license,
2. PF_RING FT was not present in older versions of PF_RING,
3. you can use PF_RING FT without PF_RING for packet capture

In that case, I can only store packet headers in my local DPDK mempools,
right? The idea is to minimize mempool sizes.

Post by Harish Patil
2) Is it safe for me to release/free the buffers after calling
pfring_ft_process()?
Yes.

So it means pfring_ft_process() is synchronous, but where as the callbacks
can be asynchronous.
Ex: pfring_ft_set_flow_export_callback(ft, processFlow, NULL);
The processFlow() callback is asynchronous, will it be scheduled on the
same core?
Callbacks are called inline by the main packet processing thread, both
while processing packets
(with process_packet as in the example at https://github.com/ntop/PF_
RING/blob/dev/userland/examples_ft/ftflow.c#L253)
or when idle (see pfring_ft_housekeeping in https://github.com/ntop/PF_
RING/blob/dev/userland/examples_ft/ftflow.c#L255 ).
3) Do you have performance numbers doing inline DPI for 1G or 10G links?

Post by Harish Patil
You should be able to do 10Gbit line-rate, however this also depends on
your hw. Please note you can use multiple threads (with RSS or ZC software
distribution) if 1 core is not enough.
4) Does PF_RING FT/nDPI maintain state information to detect a flow OR
does it just uses the packet passed to parse L7 info?
It keeps some state, as 1 packet is often not enough.

Can you please provide any pointers on that?
This is done internally by FT, it is not visible through the API, you just
get teh L7 protocol as soon as it is detected.
Another question: Do you know if there is any issues using pf_ring ft lib
as a dynamic module in DPDK secondary process?
I do not see issues, but I am not 100% sure I got what you mean with "as a
dynamic module in DPDK secondary processâ.
Regards
Alfredo

Post by Harish Patil
Regards
Alfredo
Thanks
Harish

Hi Harish
we are currently using those libs to compile our applications on many
distributions
including Ubuntu 14/16/18, Centos 6/7, Debian 7/8/9, please let us know
if you
experience issues with the compilation, and we will find a way to handle
that.
Regards
Alfredo
Hi Alfredo,
Thanks for the response.
We are planning to use PF_RING FT in our projects.
We only have shared libs. How can we integrate and make sure it works in
all cases (platforms/GCC)?
If we had the sources we could compile along with sources and would be
much better.
Thanks.
Harish

Hello,
Sorry but PF_RING FT is not open source.
Regards,
Emanuele
Hi,
I am new to ntop/ndpi.
pfring_ft_create_table() or pfring_ft_process() etc which uses ndpi lib.
I only see the function defines in userland/lib/pfring_ft.h but not
the implementation.
Could you pls provide some pointers?
Thanks,
Harish
_______________________________________________
_______________________________________________
Ntop mailing list
http://listgateway.unipi.it/mailman/listinfo/ntop

Harish Patil

2018-06-29 05:32:21 UTC

Permalink

Thanks Alfredo,
Regarding last question, please ignore it, it was a mistake on my side.

Hi Harish
my comments inline
Alfredo
Hi Alfredo,
Thanks for the response.
Regarding performance question #3.
When you say 10G line rate, are you saying 10G line rate with single core
and is it true for all frame sizes?
over DPDK/PF_RING, do we have test results that shows how many ports/cores
needed for inline DPI?
single core,
as of DPI, it is hard to say it would be able to do line-rate at any
traffic condition as it
depends on several factors, including the number of new flows per second,
the l7
protocol type (some dissectors use more cpu than others), etc. What I can
say is that
with mixed internet traffic (what we used for testing) it can keep up with
10G full rate.
Another question: I happen to run some HTTP traffic over DPDK (doing just
L2 forwarding) and did not see any flows detected (using PF_RING FT APIs).
Do you know what typically could cause that and any debug info that can be
gathered?
Uhm, did you try to process the same traffic using PF_RING or libpcap? Or
could you
provide a pcap file that we can use to reproduce it?
Regards
Alfredo
Thanks,
Harish

Post by Harish Patil
Thanks Alfredo.
Replies below.

Post by Harish Patil
Thanks Alfredo/Emanuele.
############################################################
#############
# ERROR: You do not seem to have a valid PF_RING FT 7.3.0.180617 license
[ECFE456A76066A02]
# ERROR: Please get one at http://shop.ntop.org/.
############################################################
#############
# We're now working in demo mode with packet capture and
# transmission limited to 5 minutes
Is there a need to buy a licensed version? Is this newly introduced in
the later releases?
Yes, PF_RING FT, included in latest stable, requires a license.

In that case, I can only store packet headers in my local DPDK mempools,
right? The idea is to minimize mempool sizes.

Post by Harish Patil
2) Is it safe for me to release/free the buffers after calling
pfring_ft_process()?
Yes.

So it means pfring_ft_process() is synchronous, but where as the
callbacks can be asynchronous.
Ex: pfring_ft_set_flow_export_callback(ft, processFlow, NULL);
The processFlow() callback is asynchronous, will it be scheduled on the
same core?
Callbacks are called inline by the main packet processing thread, both
while processing packets
(with process_packet as in the example at https://github.com/ntop/PF_
RING/blob/dev/userland/examples_ft/ftflow.c#L253)
or when idle (see pfring_ft_housekeeping in
https://github.com/ntop/PF_RING/blob/dev/userland/examples_
ft/ftflow.c#L255 ).
3) Do you have performance numbers doing inline DPI for 1G or 10G links?

Can you please provide any pointers on that?
This is done internally by FT, it is not visible through the API, you
just get teh L7 protocol as soon as it is detected.
Another question: Do you know if there is any issues using pf_ring ft lib
as a dynamic module in DPDK secondary process?
I do not see issues, but I am not 100% sure I got what you mean with "as
a dynamic module in DPDK secondary processâ.
Regards
Alfredo

Post by Harish Patil
Regards
Alfredo
Thanks
Harish

Hi Harish
we are currently using those libs to compile our applications on many
distributions
including Ubuntu 14/16/18, Centos 6/7, Debian 7/8/9, please let us know
if you
experience issues with the compilation, and we will find a way to
handle that.
Regards
Alfredo
Hi Alfredo,
Thanks for the response.
We are planning to use PF_RING FT in our projects.
We only have shared libs. How can we integrate and make sure it works
in all cases (platforms/GCC)?
If we had the sources we could compile along with sources and would be
much better.
Thanks.
Harish

Hello,
Sorry but PF_RING FT is not open source.
Regards,
Emanuele
Hi,
I am new to ntop/ndpi.
pfring_ft_create_table() or pfring_ft_process() etc which uses ndpi lib.
I only see the function defines in userland/lib/pfring_ft.h but not
the implementation.
Could you pls provide some pointers?
Thanks,
Harish
_______________________________________________
_______________________________________________
Ntop mailing list
http://listgateway.unipi.it/mailman/listinfo/ntop