strace, ktrace, and truss

Network and system tools

wget and curl

For our purposes, wget and curl do roughly the same thing: fetch the contents of a URL to the console.

We can use either to retrieve the rendezvous or relay seeds using the seeding URL. In this case we choose the familiar Pubnet seeding URLs.

Using wget

http://www.gnu.org/software/wget/

$ wget -O - http://rdv.jxtahosts.net/cgi-bin/rendezvous.cgi?2
--18:17:45--  http://rdv.jxtahosts.net/cgi-bin/rendezvous.cgi?2
           => `-'
Resolving rdv.jxtahosts.net... 209.128.126.130
Connecting to rdv.jxtahosts.net|209.128.126.130|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]

    [<=>                                  ] 0             --.--K/s             
http://209.128.126.120:9700
tcp://192.18.37.36:9701
tcp://192.18.37.37:9701
tcp://192.18.37.38:9701
tcp://209.128.126.120:9701
tcp://209.128.126.120:9711

Using curl

http://curl.haxx.se/

$ curl http://rdv.jxtahosts.net/cgi-bin/rendezvous.cgi?2
http://209.128.126.120:9700
tcp://192.18.37.36:9701
tcp://192.18.37.37:9701
tcp://192.18.37.38:9701
tcp://209.128.126.120:9701
tcp://209.128.126.120:9711

strace, ktrace, and truss

The three tools strace, ktrace, and truss, available on Linux, OS X, and Solaris, respectively will one dark day be your best friend. They trace all system calls made by a running process and its children. Most often one of these tools will inform you that your app is trying to find some important system file, but cannot find or open it.

Consider this very simple example, whereby under Linux we seek to get a directory listing of the file foo. We intentionally arrange for foo not to exist, so we can show you what strace has to say about it.

First, let's just get a directory listing of the current working directory, just to prove foo does not exist:

$ ls -al
total 8
drwxr-xr-x 2 petrovic users 4096 2006-11-25 06:32 .
drwxr-xr-x 9 petrovic users 4096 2006-11-25 06:32 ..
-rw-r--r-- 1 petrovic users    0 2006-11-25 06:32 iamnotfoo

Now, let's intentionally try to get a listing for foo

$ ls -al foo
ls: foo: No such file or directory

So ls tells us foo does not exist, but not all applications will be as helpful. So let's ask strace to help diagnose. strace produces a lot of output, so we'll filter out anything that does not contain foo. Using strace is a simple matter of prefixing the command of interest, with its arguments, with strace:

$ strace -f ls foo 2>&1 |grep foo
execve("/bin/ls", ["ls", "foo"], [/* 38 vars */]) = 0
stat64("foo", 0x80765fc)                = -1 ENOENT (No such file or directory)
lstat64("foo", 0x80765fc)               = -1 ENOENT (No such file or directory)
write(2, "foo", 3foo)                      = 3

The -f switch to strace forces strace to trace child processes of ls, which is normally what you want to do when using any of the "trace" family of tools. So we see in the output that the C system call stat64() returns -1 with an error indicating there is 'no entry' for file foo. stat64 is the system call used to determine a file's "status", which includes file size and other information.

strace can also be used to observe low-level opening of network connections, although depending on what you're doing, lsof or netstat may be better choices

$ strace -f curl http://www.jxta.org 2>&1 |grep connect 
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("24.205.1.14")}, 28) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("24.205.1.14")}, 28) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("24.205.1.14")}, 28) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("64.125.133.201")}, 16) = -1 EINPROGRESS (Operation now in progress)

wherein we see curl trying to connect to the "name server caching daemon" nscd, doing DNS lookups on port 53 on the URL on the command line, and finally trying to connect to the resolved address on port 80. Connecting to nscd is totally system dependent, so if you don't see this attempt on your machine, it's purely optional for our purposes here. Had some local name caching prefetch succeeded, we probably would not have seen the calls to the nameservers on port 53, and the app would have gone straight to connecting to the cached address on port 80, but I guess that's a digression.

The implication of all this is that you can do

$ strace -f java -classpath ...

to trace the system calls made by your Java/JXTA app.

Things to know or do before you post

Clear your cache.

Guarantee that before you post, you have removed the JXTA cache in $JXTA_HOME/cm. A populated cache at startup time can lead to unexpected results. I've been there. We all have.

$ rm -rf $JXTA_HOME/cm

Are you running on the Pubnet?

If you are running on the Sun Pubnet, please say so. If you don't know if you are running on the Pubnet, post your $JXTA_HOME/PlatformConfig file. The PlatformConfig contains the URL from which relay and rendezvous seeds are determined.

More specifically, the Service <Svc> elements below in PlatformConfig tell the story, in particular the <Parm> child elements. We see the signature jxtahosts.net URLs. This is a Pubnet configuration.

<Svc>
   <MCID>
      urn:jxta:uuid-DEADBEEFDEAFBABAFEEDBABE0000000605
   </MCID>
   <Parm type="jxta:RdvConfig" config="client" xmlns:jxta="http://jxta.org">
      <seeds>
         <addr seeding="true">
            http://rdv.jxtahosts.net/cgi-bin/rendezvous.cgi?2
         </addr>
      </seeds>
   </Parm>
</Svc>
<Svc>
   <MCID>
      urn:jxta:uuid-DEADBEEFDEAFBABAFEEDBABE0000000F05
   </MCID>
   <Parm type="jxta:RelayConfig" client="true" xmlns:jxta="http://jxta.org">
      <client>
         <seeds>
            <addr seeding="true">
               http://rdv.jxtahosts.net/cgi-bin/relays.cgi?2
            </addr>
         </seeds>
      </client>
      <server/>
   </Parm>
</Svc>

As an interesting aside, take a quick look in ./platform/binding/java/api/src/net/jxta/peergroup/PeerGroup.java and one finds the origin and meaning of the magic ModuleClassIDs (<MCID>) found in various parts of PlatformConfig, e.g.

/**
* Well known module class identifier: rendezvous service
*/
public final static ModuleClassID rendezvousClassID = (ModuleClassID) ID.create(URI.create(WK_ID_PREFIX + "0000000605"));

Turn Multicast Off.

This is almost the 11th Commandment. Multicast should be used only if you know what you are doing. Multicast on can make it look like your app is working when it isn't. Multicast config is here:

<Svc>
   <MCID>
      urn:jxta:uuid-DEADBEEFDEAFBABAFEEDBABE0000000905
   </MCID>
   <Parm>
      <jxta:TransportAdvertisement xmlns:jxta="http://jxta.org" 
          type="jxta:TCPTransportAdvertisement">
         <Protocol>
            tcp
         </Protocol>
         <ConfigMode>
            auto
         </ConfigMode>
         <Port start="0" end="0">
            0
         </Port>
         <MulticastAddr>
            224.0.1.85
         </MulticastAddr>
         <MulticastPort>
            1234
         </MulticastPort>
         <MulticastSize>
            16384
         </MulticastSize>
      </jxta:TransportAdvertisement>
   </Parm>
</Svc>

The presence of <MulticastOff/> indicates that multicast is off. If the element is absent, multicast is on. Therefore, in the config above, multicast is on. When you use NetworkConfigurator (and you should) to configure your peers, turn multicast off using

networkConfigurator.setUseMulticast(false);

Private net considerations

If you are running under a private net, you should change the NetPeerGroupID, NetPeerGroupName?, and NetPeerGroupDescription? via NetworkConfigurator. Then, when you have a reference to the NetPeerGroup, print these parameters

Set the private net parameters like this:

NetPeerGroupFactory factory;
try {
   factory = new NetPeerGroupFactory(
      (ConfigParams)configurator.getPlatformConfig(),
      new File(jxtaHome).toURI(),
      IDFactory.fromURI(new URI(NetPeerGroupID)),
      NetPeerGroupName,
      (XMLElement) StructuredDocumentFactory.newStructuredDocument(MimeMediaType.XMLUTF8,
         "desc", NetPeerGroupName)
   );
}
...
netPeerGroup = factory.getInterface();

Then print as a sanity check that the values 'took':

System.out.println("NetPeerGroup ID: " + netPeerGroup.getPeerGroupID());
System.out.println("NetPeerGroup Name: " + netPeerGroup.getPeerGroupName());

The various JDK j commands

There are a bunch of useful j-commands in your JDK bin/ directory. Use your JDKs jps command to check the process ID of your running Java programs

$ jps -l
17909 sun.tools.jps.Jps
13350 org.apache.tools.ant.launch.Launcher
13368 privatenet.Rendezvous

Network tools

Here are some handy tools and techniques to master to examine your network in a JXTA context.

lsof

Most Unix boxes have lsof (list open files)

http://www.akadia.com/services/lsof_intro.html

I'm not sure what the Windows equivalent is, but I believe it exists.

mach:~> lsof -r -i :9701
COMMAND   PID     USER   FD   TYPE  DEVICE SIZE NODE NAME
java    26579 petrovic   12u  IPv6 1204783       TCP *:9701 (LISTEN)
java    26579 petrovic   60u  IPv6 1204785       TCP mach:9701->cw:50191 (ESTABLISHED)
java    26579 petrovic   61u  IPv6 1204808       TCP mach:9701->pluto:34684 (ESTABLISHED)
java    26579 petrovic   83u  IPv6 1204809       TCP mach:9701->cw:50192 (ESTABLISHED)
java    26579 petrovic   87u  IPv6 1204817       TCP mach:9701->pluto:34685 (ESTABLISHED)

shows two peers connected to mach's JXTA listener on port 9701. The "-r" will give you a rolling update, while the "-i" gives you "internet files" (read sockets). The ":9701" restricts output to port 9701.

netstat

Alternatively, you can use netstat, for similar information

This invocation of netstat is unix-specific. Command line switches will vary on the Mac.

mach:~> netstat -cat |grep 9701
tcp6       0      0 *:9701                  *:*                     LISTEN     
tcp6       0      0 mach:9701               pluto:34684             ESTABLISHED
tcp6       0      0 mach:9701               pluto:34685             ESTABLISHED
tcp6       0      0 mach:9701               cw:50191                ESTABLISHED
tcp6       0      0 mach:9701               cw:50192                ESTABLISHED

Verifying the JXTA greeting is available

Sometimes it's good to see JXTAHELLO on a JXTA listener:

mach:~> telnet mach 9701
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
JXTAHELLO tcp://127.0.0.1:33102 tcp://192.168.0.6:9701 urn:jxta:uuid-79B6A084D3264DF8B641
867D926C48D9F8BA10F44BA74475ABE2BB568892B0DC03 0 1.1

Cool.

Packet sniffing

Sometimes you want to see packets go by

The venerable tcpdump: http://www.tcpdump.org/

Available for Windows, too.

You can use Ethereal if you must.

mach:~> sudo tcpdump -Xlns0 port 9701 |tee x.out  # outputs to stdout and saves to x.out
13:21:54.399237 IP 192.168.0.6.9701 > 192.168.0.8.50192: P 2011:2523(512) ack 479 win 225
2 <nop,nop,timestamp 368990230 988681059>
        0x0000:  4500 0234 21fd 4000 4006 9568 c0a8 0006  E..4!.@.@..h....
        0x0010:  c0a8 0008 25e5 c410 3add 3757 4aad f9bf  ....%...:.7WJ...
        0x0020:  8018 08cc b809 0000 0101 080a 15fe 5816  ..............X.
        0x0030:  3aee 1363 0c63 6f6e 7465 6e74 2d74 7970  :..c.content-typ
        0x0040:  6500 1661 7070 6c69 6361 7469 6f6e 2f78  e..application/x
        0x0050:  2d6a 7874 612d 6d73 670e 636f 6e74 656e  -jxta-msg.conten
        0x0060:  742d 6c65 6e67 7468 0008 0000 0000 0000  t-length........
        0x0070:  01c1 006a 786d 6700 0001 0005 7265 6c61  ...jxmg.....rela
        0x0080:  7900 046a 7865 6c02 0100 0872 6573 706f  y..jxel....respo
        0x0090:  6e73 6500 1874 6578 742f 706c 6169 6e3b  nse..text/plain;
        0x00a0:  6368 6172 7365 743d 5554 462d 3800 0000  charset=UTF-8...
        0x00b0:  0963 6f6e 6e65 6374 6564 6a78 656c 0201  .connectedjxel..
        0x00c0:  0005 6c65 6173 6500 1874 6578 742f 706c  ..lease..text/pl
        0x00d0:  6169 6e3b 6368 6172 7365 743d 5554 462d  ain;charset=UTF-
        0x00e0:  3800 0000 0733 3630 3030 3030 6a78 656c  8....3600000jxel
        0x00f0:  0101 0015 456e 6470 6f69 6e74 536f 7572  ....EndpointSour
        0x0100:  6365 4164 6472 6573 7300 1874 6578 742f  ceAddress..text/
        0x0110:  706c 6169 6e3b 6368 6172 7365 743d 5554  plain;charset=UT
        0x0120:  462d 3800 0000 1674 6370 3a2f 2f31 3932  F-8....tcp://192
        0x0130:  2e31 3638 2e30 2e36 3a39 3730 316a 7865  .168.0.6:9701jxe
        0x0140:  6c01 0100 1a45 6e64 706f 696e 7444 6573  l....EndpointDes
        0x0150:  7469 6e61 7469 6f6e 4164 6472 6573 7300  tinationAddress.
        0x0160:  1874 6578 742f 706c 6169 6e3b 6368 6172  .text/plain;char
        0x0170:  7365 743d 5554 462d 3800 0000 b774 6370  set=UTF-8....tcp
        0x0180:  3a2f 2f30 2e30 2e30 2e30 3a30 2f45 6e64  ://0.0.0.0:0/End
        0x0190:  706f 696e 7453 6572 7669 6365 3a75 7569  pointService:uui
        0x01a0:  642d 3842 3333 4530 3238 4230 3534 3439  d-8B33E028B05449
        0x01b0:  3742 3842 4639 4134 3436 4132 3234 4231  7B8BF9A446A224B1
        0x01c0:  4646 3032 2f75 7569 642d 4445 4144 4245  FF02/uuid-DEADBE
        0x01d0:  4546 4445 4146 4241 4241 4645 4544 4241  EFDEAFBABAFEEDBA
        0x01e0:  4245 3030 3030 3030 3046 3035 2f75 7569  BE0000000F05/uui
        0x01f0:  642d 3739 4236 4130 3834 4433 3236 3444  d-79B6A084D3264D
        0x0200:  4638 4236 3431 3836 3744 3932 3643 3438  F8B641867D926C48
        0x0210:  4439 4638 4241 3130 4634 3442 4137 3434  D9F8BA10F44BA744
        0x0220:  3735 4142 4532 4242 3536 3838 3932 4230  75ABE2BB568892B0
        0x0230:  4443 3033                                DC03

-- Main.mspetrovic - 21 Nov 2006