DataSurgeon

DataSurgeon copied to clipboard

Hi!

Currently DataSurgon can't handle CIDR-ranges:

Example:

<ul>
  <li>100.64.0.1</li>
  <li>100.64.0.0/10</li>
  <li>40.88.21.235/30</li>
  <li>169.254.0.0/16</li>
  <li>224.0.0.0/4</li>
  <li>2002::1234:abcd:ffff:c0a8:101</li>
  <li>2002::1234:abcd:ffff:c0a8:101/64</li>
  <li>2001:db8:3333:4444:5555:6666:7777:8888</li>
  <li>2001:db8:3333:4444:5555:6666:7777:8888/64</li>
</ul>

Result

❯ ds -C -f test.txt
ip_address: 100.64.0.1
ip_address: 100.64.0.0
ip_address: 40.88.21.235
ip_address: 169.254.0.0
ip_address: 224.0.0.0
ipv6_address: 2001:db8:3333:4444:5555:6666:7777:8888
ipv6_address: 2001:db8:3333:4444:5555:6666:7777:8888

(Also, by testing this I found a bug where it doesn't detect double colon in ipv6 urls) 😁 https://www.ibm.com/docs/en/i/7.5?topic=concepts-ipv6-address-formats

Hi,

Thanks! I will look into this....

Best, Drew

Hi,

Is this the output you were looking for? (edit: I see the IPV6 error you were talking about and im looking into it)

Contents of data.txt

<ul>
  <li>100.64.0.1</li>
  <li>100.64.0.0/10</li>
  <li>40.88.21.235/30</li>
  <li>169.254.0.0/16</li>
  <li>224.0.0.0/4</li>
  <li>2002::1234:abcd:ffff:c0a8:101</li>
  <li>2002::1234:abcd:ffff:c0a8:101/64</li>
  <li>2001:db8:3333:4444:5555:6666:7777:8888</li>
  <li>2001:db8:3333:4444:5555:6666:7777:8888/64</li>
  <li>2001:db8:3333:4444:5555:6666:7777::/64</li>
</ul>

Results

dalleman@drew-ubuntu:~/Projects/Software/DataSurgeon/target/release$ ./ds -Cf data.txt 
ip_address: 100.64.0.1
ip_address: 100.64.0.0/10
ip_address: 40.88.21.235/30
ip_address: 169.254.0.0/16
ip_address: 224.0.0.0/4
ipv6_address: 2002::1234
ipv6_address: 2002::1234
ipv6_address: 2001:db8:3333:4444:5555:6666:7777:8888
ipv6_address: 2001:db8:3333:4444:5555:6666:7777:8888/64
ipv6_address: 2001:db8:3333:4444:5555:6666:7777

Best, Drew

@Defirence I saw your now deleted comment. This project is not malicious and has various practical uses :laughing: . Your Github ticket will be ignored.

Hi,

@borestad I think I got the Regex correct how does this look?

drew@ubuntu:~/Projects/Software/DataSurgeon$ ./target/release/ds -f target/release/data.txt -C
ip_address: 100.64.0.1
ip_address: 100.64.0.0/10
ip_address: 40.88.21.235/30
ip_address: 169.254.0.0/16
ip_address: 224.0.0.0/4
ipv6_address: 2002::1234:abcd:ffff:c0a8:101
ipv6_address: 2002::1234:abcd:ffff:c0a8:101/64
ipv6_address: 2001:db8:3333:4444:5555:6666:7777:8888
ipv6_address: 2001:db8:3333:4444:5555:6666:7777:8888
ipv6_address: 2001:db8:3333:4444:5555:6666:7777

Best, Drew Alleman

Hi @Drew-Alleman !

Sorry for late reply, my inbox (work related) had 1600 messages, and I missed that you wrote the other day!

Looks great. 👍 I wonder .... if one would like to split these into multiple groups later in another version?

like:

ip_address: 100.64.0.1
ip_cidr: 100.64.0.0/10
ip_cidr: 40.88.21.235/30
ip_cidr: 169.254.0.0/16
ip_cidr: 224.0.0.0/4
ipv6_address: 2002::1234:abcd:ffff:c0a8:101
ipv6_cidr: 2002::1234:abcd:ffff:c0a8:101/64
ipv6_address: 2001:db8:3333:4444:5555:6666:7777:8888
ipv6_address: 2001:db8:3333:4444:5555:6666:7777:8888
ipv6_address: 2001:db8:3333:4444:5555:6666:7777

... but for now, this is excellent! I tried to write/improve the regexp myself yesterday evening, but was apparently to tired.

Great work! 😊

Hi,

Glad to hear it works for you. I am going to update the IPV4 and IPV6 regex for what I have now.

drew@ubuntu:~/Projects/Software/DataSurgeon$ ./target/release/ds -f target/release/data.txt -C
ip_address: 100.64.0.1
ip_address: 100.64.0.0/10
ip_address: 40.88.21.235/30
ip_address: 169.254.0.0/16
ip_address: 224.0.0.0/4
ipv6_address: 2002::1234:abcd:ffff:c0a8:101
ipv6_address: 2002::1234:abcd:ffff:c0a8:101/64
ipv6_address: 2001:db8:3333:4444:5555:6666:7777:8888
ipv6_address: 2001:db8:3333:4444:5555:6666:7777:8888
ipv6_address: 2001:db8:3333:4444:5555:6666:7777

I might create a plugin for extracting CIDR information specifically. Edit: After seeing your new issue I probably will do this, and have -i and -6 return standard IP addresses then have a separate argument for extracting CIDR addresses. How does that sound to you?

Best, Drew Alleman

This sounds great! :)

Hi,

I will try to get this rolled out this weekend.

Best, Drew