CSGQ: CSG1 Quarterly
home | idtech | ps2 | turtle | codebin | gossip
More efficient software means we could probably run tomorrow's software
with yesterday's hardware. Instead, because of bloat, we're stuck running
yesterday's software with tomorrow's hardware.
slashdot.org comment

Several programs you may find useful.

Quick passwords

To get me some quick dirty passwords, there is pretty handful Linux/Cygwin commands combination, like this:

$ tr -dc '0-9A-Za-z' </dev/urandom | fold -w8 | head -n1

(Yes, I'm pretty good at memorizing such garbage. No, for hosting accounts there was another sequence used).

FPU FILD instruction in shell script (what?!)

This one is really weird, and I doubt you will ever need something similar. But it contains some perculiarities within, which are valuable by itself, such as translating hex number to binary (a bit string) and back, or circular shift, without bc, all is done with shell (bash actually) and awk. There was some embedded system, with busybox and awk, but without bc or perl, and I need to read and compare some ADC data, well, whatever. All it does is taking integer (as 32-bit hex) and converts it to IEEE floating point number, presented as 32-bit hex as well, the same you get when typing in C program int ival=92348; float fval=(float)ival; so that the FPU instruction (FILD) is called on type convertion.

The code is based upon great article Implementing int2float(), it works for little-endian systems only, and performing like this:

$ perl -e 'printf "%08x", (-92347987 & 0xffffffff)' | bash fild.sh | { read ins; echo -n -e
	"\x${ins:6:2}\x${ins:4:2}\x${ins:2:2}\x${ins:0:2}"; } | od -f | head -n1 | awk '{print $2}'

Here's the script (1.2K).

Of course, you should simply use awk when you have mixed integer/float calculations in shell, such as follows:

$ echo "527708 4.7683729E-006" | awk '{print $1 * $2}'

Show bits set in arbitrary number

When it is needed to see which bits are set in a number, I use very simple perl script combined with bc output, something like this:

$ echo "obase=2;ibase=16;400001D3" | bc | perl bitpos.pl

which would produce "30 8 7 6 4 1 0" string in result, e.g. that helps to check which flags are set in CPSR register of ARM processor, and works even better with 64-bit or 80-bit values.

Here's the script (0.4K).

Windows shell zip operation helper

When deploying something on Windows, I use small VBScript (yikes!) helper, named wshzip.vbs (1.0K) which works as follows:

C:\Documents and Settings\All Users>cscript //NoLogo wshzip.vbs
Usage: wshzip <copy|expand> <zipfolder> <file1> [<file2>...]

That is, if you want to expand some zipped folder, you should do

cscript //NoLogo wshzip.vbs expand zipfile output_path

Searching hexadecimal dump for a pattern

Often when analyzing network traffic dumped with Wireshark, I ended up with packet data exported as hexadecimal text. I got simple utility named schxd which is scanning through the hex dump for the string of bytes requested. For example,

$ od -Ax -tx1 -v <somefile> | schxd -x "1f 25 ac fc c0 f3"

searches for some bytes inside (binary) file. Which is kind of dumb because you're using bgrep (or whatever) for this already. But sometimes you got hex dump that can't be converted back to data simply by calling xxd, such as network traffic capture exported as plain text with full details including decrypted packet payload. And when captured traffic is so big that Wireshark choked on it, tshark is the only option.

$ tshark -r <packetdumpfile> -P -x | schxd -x "02 74 66 04 69 6e 66 6f"

59:0040  00 00 00 00 07 6c 65 69 70 7a 69 67 02 74 66 04   .....leipzig.tf.

$ tshark -r <packetdumpfile> -P -x | sed -n "52,61p"

  7   0.266933 ->   DNS 89 Standard query 0x1aba  A leipzig.tf.info

0000  00 0c 29 3f af db 00 0c 29 10 9e 91 08 00 45 00   ..)?....).....E.
0010  00 4b 00 c8 40 00 7f 06 d5 a8 0a 00 08 9b 0a 00   .K..@...........
0020  08 a2 04 2c 00 35 88 f0 89 1a 3b ec e8 f5 50 18   ...,.5....;...P.
0030  fa f0 b7 44 00 00 00 21 1a ba 01 00 00 01 00 00   ...D...!........
0040  00 00 00 00 07 6c 65 69 70 7a 69 67 02 74 66 04   .....leipzig.tf.
0050  69 6e 66 6f 00 00 01 00 01                        info.....

Here's the source (10K).

Recently I was up to the task of finding portion (if any) of large file transmitted across the network, and here's one-liner I came up with:

$ for((i=0; i < $FILESIZE; i += 64)) do od -v -tx1 -An -j$i -N64 $FILENAME | tr -d '\r\n' |
	 xargs -0 -I {} schxd -x {} $TRAFFICHEXDUMP; done | grep -v "Found 0 results"

It really helped. Not to the exact point, but ended up being very close to actual packed where the transmission of portion in question started. FILESIZE, FILENAME and TRAFFICHEXDUMP are self-explaining I guess. Though I'm sure, 64, which is an arbitrary number, should be defined as a variable too; it depends on size of packet - 128 bytes in my case.

Putting junk on the wire

I worked with computer networks a lot, and surprisingly often it was needed to put some single packet on the wire, usually tweaked in some details from actual data being previously captured; i.e. you capture the traffic, then save one packet (as raw data, from Wireshark), then change some fields e.g. port numbers, fixing all checksums, and then resend it. I use small pcap-based program, named penwr, it puts whatever junk you specified on the wire, bypassing OS TCP/IP stack (run it under root).

# ./penwr -w 08:00:27:51:86:ae <somefile>

It's rather inconvenient to specify network adapter with MAC-address, but under Windows systems other ways are much worse; you can skip it completely, if your system has only one network interface. You have to install pcap-devel package on your system (if it's UNIX-like), then build it with

$ gcc penwr.c -o penwr -lpcap

For Windows, you should get WinPcap with Developer's Pack from here. Wireshark installs WinPcap driver, so if you have it, download and extract only WpdPack - headers and libs (can be found here). I don't like any visual tools from Microsoft, thus am building the program on Windows XP with outdated Driver Development Kit (DDK, it contains C compiler); in addition I got Platform SDK installed (outdated as well). The next goes in a single line:

cl /MT /O2 /W3 penwr.c /DWIN32 /D_CRT_SECURE_NO_WARNINGS /IC:\WINDDK\3790~1.183\inc\crt /IC:\WpdPack\Include /IC:\PROGRA~1\MICROS~2\Include /link /LIBPATH:C:\WINDDK\3790~1.183\lib\wxp\i386 /LIBPATH:C:\WINDDK\3790~1.183\lib\crt\i386 /LIBPATH:C:\WpdPack\Lib bufferoverflowU.lib wpcap.lib Packet.lib

Yes, it looks terrible, mainly because of multiple include and library paths to various parts of SDK and DDK. Your method (or paths at least) should differ. Btw, did you know, dumpbin utility is missing in DDK because it's included into link and can be invoked with link /dump? And to get short path (with tildes) you should run dir /X? And to find a file you should use dir /S C:\filename? And to "cat" a file (the other one half of use-cases for UNIX cat utility beyond concatenations) you can run type filename? MS-DOS sometimes feels OK after all.. But for me, Windows without Cygwin is pretty useless.

Anyway, here's the source (8K).

Almost forgot - when building on Windows, that bufferoverflowU.lib is necessary because I'm using DDK compiler, otherwise I'd get linking errors with __security_cookie in consequence of automatically switching buffers check option (/GS) along with DDK runtimes which are missing the code required; on conventional Visual Studio installation you'll need only wpcap.lib and Packet.lib from WpdPack.

"OK, I'm on Windows, and don't have any building environment ready nor time to prepare it, and want to run it immediately!" - here's (27K) binary built on Windows XP, 32-bit, linked statically, so you should have only WinPcap installed.

To fake application clients

This one is more complex/advanced (and interesting) than the above. I've used several traffic generators, such as BRUTE (site) and pktgen (site); they are all great, but for some specific cases I needed tool not necessarily powerful performance-wise (pktgen can give you numbers very close to theoretical limits regarding packets per seconds, pps) but rather capable of doing the trick to pretend there are many endpoints, with various (phantom) IPs. For TCP transport. And I got it! :) stmpde is rather simple and primitive program once again built upon pcap interface and exploiting promiscuous mode with network adapter. Since it's running in user mode it has a little value in throughput tests (at least until several copies run from different network interfaces), in my experience it cannot provide more than 200K pps with packets of minimal size. However, in some application tests where you need to analyze particular firewall or proxy-like functionality it really shows its benefits.

The scheme below is rather excessive - in basic experiment you can get along with only two hosts, one for traffic generation and the other one to accept the load. But splitting the duties among physical hosts helps to narrow down principal traits of the problem under study. Suppose we need to check a packet filtering or some other kind of functionality regarding application-level traffic being forwarded by our firewall; or maybe we're developing a proxy server; suppose it's FTP traffic. And we need to know how it handles requests varied not only by the port numbers but also by the source IPs. We'll use four hosts: one running application server - it can be any FTP-server with anonymous (ftp/ftp) access, tagged host_swamp below; one to generate FTP-requests with stmpde copy, tagged host_spawn; firewall or transparent proxy which forwards requests to host_swamp and tagged host_twit (the one which behavior we're examining actually); and the last one tagged host_dump to discard all the packets that going from host_swamp in reply to generated requests. The last three - host_spawn, host_twit and host_dump should share single collision domain. To clarify this, and the presence of the host_dump at all - host_twit will be fooled with all the requests coming from the host_spawn they are from external network segment so that the replies should be routed to host_dump; this should help to reduce the load on host_spawns TCP/IP stack.

+------------+                                +------------+                         
|            | role: traffic generator        |            | role: traffic disposer  
| host_spawn | default_gw: host_twit          | host_dump  | default_gw: none        
|            |                                |            |                         
|            |                                |            |                         
|            |                                |            |                         
|            |     host_spawn, host_dump,     |            |                         
+------------+     host_twit are in one       +------------+                         
      |eth0        collision domain                 |eth0                            
      |                                             |                                
      |                                             |                                
+------------+                                +------------+                         
|            | role: firewall/proxy server    |            | role: application server
| host_twit  | default_gw: host_dump          | host_swamp | default_gw: host_twit   
|            |                                |            |                         
|            | our goal is to put it          |            |                         
|            | under stress in terms of       |            |                         
|            | connections per second         |            |                         
|            |                                |            |                         
+------------+                                +------------+                         
      |eth0                                         |eth0                            
      |                                             |                                
      |                                             |                                

On the host_dump we apply the following iptables rules:

# iptables -P INPUT DROP
# iptables -P FORWARD DROP

This makes it deaf to any incoming packet so that there wouldn't be any ARP or ICMP responses from host_dump.

To build stmpde on Linux, make sure pcap-devel package is installed, and use the following command:

$ gcc stmpde.c -o stmpde -lpcap -lrt

To run dns client making 1000 requests per second from 10000 IPs with generated name:

# ./stmpde -t 1000 -s -l

To run ftp client making 100 requests per second with anonymous connection:

# ./stmpde -t 100

To run http client:

# ./stmpde -q favicon.ico

To run in raw mode (to send junk):

# ./stmpde -a 1 -q "00 a1 b2 c3 d4 e5 f6" udp://

To run echoing server:

# ./stmpde -a 0
# ./stmpde -a 0 tcp://

The second case (with TCP) is not echoing anything back actually, but simply accepts connection, then acknowledges every datum sent until it is closed/dropped.

Currently it's IPv4 only, doesn't work with VLANs, and runs only on Linux. I'll update source (20K) with further changes/fixes.

Nearest one to do: there's silly thing about TCP connection rate, specifically when running stmpde with timeout and a number of connections

# ./stmpde -t 100 -n 30

that is, 100 ftp requests per seconds, and every connection should last for half a minute, then eventually actual connection rate will void to minimum - because of very dumb way the existing connections are handled. Promise to fix it really soon.

Next thing to develop - put it into kernel mode, to overcome context switching, and to get more pps and connections per seconds (current numbers: 200,000 pps for UDP and 1,000 TCP conn per seconds on 1 GigE).

On the side note: whatta wonderful page is this! Honestly, I despise JavaScript garbage, and strongly opposed to all the direction web development is taking, suggesting to delete a web page if it cannot prove its merits in lynx browser ;-) however, that page is heavenly good; lewish (author) is really good, check out his code on github B-)

1 Constructive Solid Geometry

© playerstartinsolid media, 2019