PCP API Documentation and more changes

During the last weeks I made lots of changes to the PCP code. Most of it was refactoring of internal stuff. First, I implemented a C Buffer "class", inspired by OpenSSH's buffer.c. I really like it. It can be used to incrementally fill a buffer, the buffer resizes automatically and makes boundary checks on every call to avoid buffer overflows. I can also directly put numbers of different sizes into it and multibyte numbers can be converted to big-endian automatically. The same prinziples can be used in the reverse, it's possible to read from a buffer little chunks. It remembers the last read offset and also supports host-endian conversion.

On top I wrote a C stream "class". This one is even nicer, since it can be used to read/write from/to files OR buffers. It behaves almost like the FILE interface, you can use ps_write() to put data out or ps_read() to fetch data in. That way I can do blockwise encryption on files and memory buffers. In fact, the encryption API uses those streams. But the latest addition to the stream class is even better: it can transparently encode to Z85 encoding. And in read-mode it can decode and determine automatically if the input is encoded. I'm not using it in the API yet, but the sample program already uses it and it works like a charm.

As you can see from the various links in this post, I also added some fairly amount of API documentation which is available here for online reading. It's generated using doxygen and I love it.

What else? I changed the sign and crypt mode of pcp, it signs the recipient list and a hash of the original content, and encrypts the signature. I re-factored some of the Z85 code to be more reliable and fault tolerant. I changed the native format of PCP public key exports. Public keys are now exported in RFC4880 (OpenPGP) format. While my exports are incompatible with OpenPGP (which is not intended anyway), the format is much more flexible than my old format. Previously I just dumped the pubkey C-structure to disk. While that worked well, it's completely impractical if I ever change that structure (which I did a lot so far). Now with the RFC4880 format, exported keys are indepentent from the internal structure, therefore I can change it as I wish without ever getting incompatible to old exports.

While I was at it, I also changed the export format of secret keys. While I don't use RFC4880 for this, I use at least a much more formalized format and not just a structure dump. Also, the whole export is now encrypted, not just the secret key blobs inside.

Last but not least, you can now export public keys in a couple of programming languages, such as a perl structure, or C code or YAML. That way an exported key can be used in small programs without the hassle of generating and maintaining keys. Just use PCP for key management.

As always latest source is on Github.

Update 2014-03-04:

I’ve got the last big change done, I removed -P and -S, now keys of any type are imported with -K. The new importer uses the Pcpstream decoder as well, so the last remaining part which isn’t using it, is the clearsig reader. I also fixed more bugs in the decoder.<p>Update 2014-03-02:</p>While I was at it, I fixed a couple of other encoding related bugs, added unittests for it, enhanced the commandline a little and added a verbose key listing feature (someone on cypherpunks requested it). Usually a key listing looks like this:

pcp1 -l
Key ID               Type             Creation Time        Owner
0xB5B64D99AE73F3BE   primary secret   2014-03-02T23:14:23  Mallory <mally@foo.bar>
0x629AFD2418EFA3BA   secret           2014-03-01T18:50:06  Alicia <alicia@local>
0x969D5931D7B409C6   valid public     2014-03-01T18:50:07  Bobby <bobby@local>
0x4EF5795E2874AD8D   valid public     2014-03-01T18:50:09  Bart <bart@local>

Note the validity new flag for public keys.</p>

Now, the new verbose listing:

pcp1 -L
Key ID               Type             Creation Time        Owner
0xB5B64D99AE73F3BE   primary secret   2014-03-02T23:14:23  Mallory <mally@foo.bar>
    88b3a815 49c28236 7e6e3c31 17c286c5 7905c7a7 ec78911f 1fd76563 5688e4c0 
    encrypted: yes, serial: c940b8f0, version: 6

0x629AFD2418EFA3BA   secret           2014-03-01T18:50:06  Alicia <alicia@local>
    076f002c 37b39ab5 cb0818b7 1fe33168 38b4d7d6 1b6e52c2 25229159 5405ec86 
    encrypted: yes, serial: 5386733f, version: 6

0x969D5931D7B409C6   valid public     2014-03-01T18:50:07  Bobby <bobby@local>
    2d1efc28 ef294913 06a914be 986975d9 869d01e1 82ea026f a4c16c98 b6a2e2bb 
    signed: yes, serial: f7cb26b4, version: 6, signature fingerprint:
    324fde54 3f6725ee a8c74f67 998e5b61 10a6f2db cdb2f282 1a689be2 3af1e514 

0x4EF5795E2874AD8D   valid public     2014-03-01T18:50:09  Bart <bart@local>
    9b660a1b a688d8fa 4a3b3a02 78b75363 70d01656 30045245 55d74944 f08bb5ab 
    signed: yes, serial: 1b4ed012, version: 6, signature fingerprint:
    5449b8f4 9f0fe50e 3e46c1be e9225e26 aa1354bf 6bd105c3 147a9870 8a531161 
Basically it displays the fingerprint of the keys, some flags and - if present - the key signature fingerprint. I store the signature anyway but didn't use or display it yet.

Update 2014-03-02:

So, finally I reverted the tilde stuff successfully. Now, PCP uses hyphens again. While I was at it, I enhanced the decoder and parser a lot. It's now more robust and parses the input linewise. Both the pcpstream decoder and the string decoder (z85_readstring()) now use the same framework. Previously they were independent from each other, so in fact I had two parsers. This was odd anyway so I generalized it.

The only remaining bad thing are clear signatures. They are parsed directly in ed.c and not by the stream decoder. I need to extend the stream decoder to be able to work on that stuff as well.

This Github commit was the last change to get into a stable state again. All unittests pass again. Thanks god (and my wife for her patience!)

Update 2014-02-28:

During the last days I made another large change, I changed the header markers for Z85 encoded data. Until now I had something like
----- BEGIN blah...-----
just as it's being used in PEM and elsewhere. This is no problem at all for small files like keys. But I wanted to have armored encryption as well, and there I fell heavily on my nose, including bleeding, crying and cursing of course.

The thing is, that the hyphen is a legitimate Z85 character and it may happen that ----- is legitimate Z85 encoded content. Proof:

$ perl -e 'foreach ((0xc6, 0x5a, 0x0b, 0x13)) { print chr($_) }' | pcp1 -z
And since I'm reading input in streaming mode, it may happen that such a marker crosses block boundaries. So, I had to solve this. And my solution was quite genious or so I thought *g*. I just used the tilde character as marker which is not part of the Z85 characterset. One tilde starts a comment, another finishes it. Really easy. I had not much to change and it worked like a charm, even via block boundaries.

However, I spoke with Pieter about the issue and basically he told me, that my idea was bad (he said: "badly-designed parser"). Well. As of this writing the tilde-mode code is on github, but I already started to rewrite it. Again! Holy shit.

So in the next iteration (hopefully the last one, it's a boring business), I'll use hyphens again but read the streams in a totally different way. In fact if an input stream is considered Z85 encoded, it will be read linewise by the decoder. The decoder parses every line then and that way it's easily possible to detect headers, comments and footers. Once a Z85 block is complete, it will be decoded and put into the internal read cache. If there is something left on the line, it will be saved for the next iteration. So in fact, there are now 2 caches, one for decoded data (used by the caller) and one for undecoded data (kind of read ahead cache).

It’s not done yet and I’m sure this stuff will steal me another couple days. Damn. However, at least it’s currently in a state where it can parse and decode a complete stream. But I didn’t try it with different blocksizes and I didn’t even dare to run the unittests yet.</p>

The new code isn't pushed to github so far, because yesterday I had a major problem with git. Thanks god I managed to solve it. "git merge", goddamnshit.

23 February 2014 | #source


Auferstanden von den Toten

Vor vielen Monaten, eher einem halben Jahr, habe ich von einem Freund einen Kammdornwels übernommen. Er hat sein Becken abgeschafft, alle Fische abgegeben, nur ihn wollte niemand haben. Ich hatte ihn damals in mein Becken getan und seither nie wieder gesehen. Mir wurde zwar versichert, dass der sich selten blicken lässt und wenn, dann nur nachts. Aber ich habe ihn nie zu Gesicht bekommen.

Tatsächlich war ich felsenfest davon überzeugt, dass er gestorben ist. Vielleicht war er irgendwo eingeklemmt und ist verhungert oder es war ihm zu kalt. Tja. Bis gestern nacht.

Ich machte meine übliche Runde, kam am Becken vorbei und bekam einen Mordsschreck: da trieb sich irgendein Riesenvieh im Vordergrund herum. Taschenlampe geholt, nachgeschaut und siehe da: der olle Knurrwels! Unfassbar. Ich frage mich echt, wie der die letzten Monate überlebt hat.

Eigenartig ist, dass er ausgerechnet jetzt aufgetaucht ist. Am Mittwoch habe ich nämlich das Becken umgebaut. Unter anderem habe ich auch eine Wurzel entfernt. Vielleicht war er darunter eingeklemmt und ich hab ihn befreit. Oder er hat so eine Art Winterschlaf gemacht und ich hab ihn geweckt. Wer weiss :)

Hier ist er also: mein Agamyxis pectinifrons, Alter unbekannt, Name Erwin:

2014-02-21 - Erwin 7:

2014-02-21 - Erwin 6:

2014-02-21 - Erwin 5:

2014-02-21 - Erwin 4:

2014-02-21 - Erwin 3:

2014-02-21 - Erwin 2:

2014-02-21 - Erwin 1:

21 February 2014 | #aquarium


Aquarium Umgebaut

Heute hab ich das Aquarium etwas umgebaut, eine Wurzel entfernt und diverse Pflanzen. Und gegärtnert. Nun haben die Elritzen hoffentlich genug Schwimmraum. Immerhin ist eine von ihnen schon in Paarungsfärbung (orange), man sieht sie auf den Bildern hier und da.

2014-02-19 - Aquarium Umbau 3:

2014-02-19 - Aquarium Umbau 1:

2014-02-19 - Aquarium Umbau 1:

19 February 2014 | #aquarium


Becken vom 08.02.2014

Frisch gegärtnert:

2014-02-08 - Becken vom 08.02.2014:

08 February 2014 | #aquarium


pcp/pbp compatibility progress

It's getting better and better. Not only do I have some very nice and secure sign+crypt functionality in PCP, but public key export and import from pcp to pbp and vice versa works now finally. From pbp issue#10:


$ pbp -b . -x --self Tom > tom.pbp 
Passphrase for decrypting master key for Tom: 

$ pcp1 -V vx -P -I tom.pbp -b
key 0x61E05C2AA4803742 added to vx.

 $ pcp1 -V vx -l              
Key ID               Type      Creation Time        Owner
0x61E05C2AA4803742    public   2014-02-06T07:31:00  Tom <>


$ pcp1 -V va -b -p -O alice.pbp   
Enter passphrase to decrypt your secret key for signing the export: 
public key exported in PBP format.

$ pbp -b . -X -i alice.pbp 
Success: imported public keys for Alice

$ pbp -b .  -l
invalid 9163 3781 9b14 ea5b 010b 7487 61fd dd46 Alice
valid 9275 5a5d 5375 bb49 d096 e0c5 1261 a575 Bob

Pure happyness!

One drawback does it have though: public key crypto doesn't work yet. I suspect the recipient list computing is incompatible since symetric crypto already works. At least pcp says that it cannot find a matching public key. And the other direction doesn't work too. But that's the smallest of all possible problems.

Update 2014-02-06:

And. Now. Finally.

# bob exports his pk
bobby@io: % pbp -x --self Bob > bob.pbp
Passphrase for decrypting master key for Bob: 

# alice exports her pk
alicia@io: % pcp -p -b -O alice.pbp
Enter passphrase to decrypt your secret key for signing the export: 
public key exported in PBP format.

# bob imports alice' pk
bobby@io: % pbp -X -i alice.pbp 
Success: imported public keys for Alicia

bobby@io: % pbp -l
valid b888 026a 38e2 cdf7 f0a6 6486 63a5 0fea Bob
invalid ed32 1935 0310 fe6f 35c6 b44d be6b 3ca8 Alicia   [1]

# alice imports bobs pk
alicia@io: % pcp -P -I bob.pbp -b
key 0x87358A0988953A67 added to ~/.pcpvault.

alicia@io: % pcp -l
Key ID               Type      Creation Time        Owner
0xB497AFF45654CD98   primary   2014-02-06T19:58:09  Alicia <>
0x87358A0988953A67    public   2014-02-06T18:58:02  bob <>

# bob encrypts to alice
bobby@io: % echo "HALLO ALICE, KNUTSCHI" > msg
bobby@io: % pbp -c -i msg -o encrypted -r Alicia -S Bob
Passphrase for decrypting encryption subkey for Bob:

# alice decrypts it
alicia@io: % pcp -d -I encrypted 
Enter passphrase to decrypt your secret key: 
Decrypted 22 bytes successfully

# other way around, alice encrypts to bob
alicia@io: % echo "ACH, SCHNUCKI" | pcp -e -O encrypted -r Bob
Enter passphrase to decrypt your secret key: 
Encrypted 164 bytes for:
bob <>

# and bob decrypts it
bobby@io: % pbp -d -i encrypted -S Bob
Passphrase for decrypting encryption subkey for Bob: 
good message from Alicia

06 February 2014 | #source