How openssh stores RSA key pairs

Posted on March 31, 2019 by usrmisc

This blog post assumes that you are familiar with the following principles at least a some general level

  • public key cryptography
  • ASN.1
  • base64 encoding

If you are not use your favorite search engine to find some introductory articles.

While no longer leading edge RSA keys continue to be widely used for ssh. Let’s look into what the key files actually contain an how the fingerprints are calculated. I’ll use standard Linux tools to do that.

Before doing that we need to recall a couple of storage formats, which predate XML, JSON, yaml, etc. which most programmers are more likely to see these days.

  • BER: Basing Encoding Rules describe how to binarily encode and store ASN.1 data
  • DER: Distinguished Encoding Rules are a special variant of BER. While BER does not generally guarantee a binarily unique encoding for given ASN.1 data, DER does. Think of leading zeros as the most obvious difference. Binary uniqueness is important if we want to compare fingerprints.
  • PEM, Privacy-Enhanced Mail is a format to store and send binary data (particlurly cryptographic keys and certificates) in ASCII only. It encodes the binary data using base64 and adds header and a footer line. Look into any PEM encoded file and you immediately understand the principle.

So let’s start with openssh private key files. While their purpose is different host key pairs and user key pairs are encoded exactly the same way. So for the purpose of this blog post we can just talk about the private key (file) and the public key (file). Just don’t do that when you ask someone else to send a public key or verify a fingerprint. Then you should always be clear on whether you mean the host or the user key.

The private key file is a PEM-encoded file. Note that since openssh 7.8, released August 2018 a new storage format is the default. Because most of my keys are older I just handle the old format for now. For simplicity I only look at private key files which are not protect by a password, i.e they are not encypted. While openssh does not use any ending (file extension), if you create a user key pair using Amazon EC2 console, they will give it the ending .pem. As usual Linux tools don’t take the ending into account.

$ cat id_rsa_test1
-----BEGIN RSA PRIVATE KEY-----
MIIEpgIBAAKCAQEAyPRkF0N/BdSRFy/+YBDfPd4NbrdBhRJUDAqizMZLcNnxHKVY
30j/b1b87ZpHVEmI8MFCUQiTaN5AdgPP1m2MI4cz3lDsKsuxDPVpzuxOWUMQpH/i
AwX4Eh51J8YT+scrg5KoDvrwsk6FF86tnuqsxyFwgP23WDEUQPlOr2DFehJlpzgw
vf5WzhKPd4VmZU6tIEL4+QBTmvnYqvCa2IWUt354h/znByer8RmGhdIvnppSSLLa
...
3fSVevllbKfe9z5Wd82Yvb33aqxi6Hpjm+3UySoHTY7/295yWQCzRtJFAum9WxAf
ZxXu9wD1AoGBAMM/ts1U5jUFxvuTOR3TdrzZDIIHW1yjxAFUrP5ML2nxGMYvbA9S
ctZOJ3OUKL/wdsbRD8a3xeCWWOgkNPl4HOSUePjHAaji0K/0lBKaLHPler874ELj
aBOQgTrLS6/A4uYgXB2iFUKskecjDUOrI+/BRw/SolxIbkwM0wPk7KNB
-----END RSA PRIVATE KEY-----

Let’s decode that

$ grep -v "RSA PRIVATE" id_rsa_test1 | base64 | hexdump -C
00000000  30 82 04 a6 02 01 00 02  82 01 01 00 c8 f4 64 17  |0.............d.|
00000010  43 7f 05 d4 91 17 2f fe  60 10 df 3d de 0d 6e b7  |C...../.`..=..n.|
00000020  41 85 12 54 0c 0a a2 cc  c6 4b 70 d9 f1 1c a5 58  |A..T.....Kp....X|
00000030  df 48 ff 6f 56 fc ed 9a  47 54 49 88 f0 c1 42 51  |.H.oV...GTI...BQ|
00000040  08 93 68 de 40 76 03 cf  d6 6d 8c 23 87 33 de 50  |..h.@v...m.#.3.P|
00000050  ec 2a cb b1 0c f5 69 ce  ec 4e 59 43 10 a4 7f e2  |.*....i..NYC....|
00000060  03 05 f8 12 1e 75 27 c6  13 fa c7 2b 83 92 a8 0e  |.....u'....+....|
00000070  fa f0 b2 4e 85 17 ce ad  9e ea ac c7 21 70 80 fd  |...N........!p..|
...

Ok, that doesn’t look very helpful.

But If we know that the data is ASN.1 in DER encoding…

$ grep -v "RSA PRIVATE" id_rsa_test1 | base64 -d | openssl asn1parse -inform DER
    0:d=0  hl=4 l=1190 cons: SEQUENCE          
    4:d=1  hl=2 l=   1 prim: INTEGER           :00
    7:d=1  hl=4 l= 257 prim: INTEGER           :C8F46417437F05D491172FFE6010DF3DDE0D6EB7418512540C0AA2CCC64B70D9F11CA558DF48FF6F56FCED9A47544988F0C14251089368DE407603CFD66D8C238733DE50EC2ACBB10CF569CEEC4E594310A47FE20305F8121E7527C613FAC72B8392A80EFAF0B24E8517CEAD9EEAACC7217080FDB758311440F94EAF60C57A1265A73830BDFE56CE128F778566654EAD2042F8F900539AF9D8AAF09AD88594B77E7887FCE70727ABF1198685D22F9E9A5248B2DA98870BB65448BBE113406770CFAC6FF7AD7691A08CB74FCD37125ECBA93A64639B764CE362572E442613A4F52CD73DC898BD0476EF29383983D5D71EBEFE20C37153D225A118F5778E23363B
  268:d=1  hl=2 l=   3 prim: INTEGER           :010001
  273:d=1  hl=4 l= 257 prim: INTEGER           :B34064C3E79E37BA2C0E0DEE8F0C9DC5D6FC0E6583194B871312ADB391B21BA11C951BDCD19790DA6532CA53130B5968C0B0F311CB5F8A6CADFD575B1E7FAB40C9CA77A5A1746F2BA84E852A61D9658010295C3563D3003C5154D9BCB377C00FCD9695387F9912C43DAE45DAD365ABE718A9FD35D444E3CA98468A6CA01AA6CE8AAEC239051E8662995B2BBE9B4F2ED14289ABD642041F28573414E67C1F2A9F7A1C084492A084FF8C488D5053AAD2EF869A5039F8BFE534B589027912BB79DD282ADA78028B9853D534DB82E55AFBDD6575A7B30A3755C42DD03E16281BE15D2962A695607C7846C5DA2773A1D6D9FE39FE78CDFBACD70B6D7ED0D8755072E9
  534:d=1  hl=3 l= 129 prim: INTEGER           :F2D7124D7ACAAC2E430C6492D946750EE66D508E5714FBD417D5E98F28328F749E76236166D1540F5DDB391B93653A3BB5326E5165B8BFCEDB5D62301895FA8A98CF0572A996838C237323C911FAF6E8843F3E37B20677A4C19728C6AB2F56D12B917F22CDD434DAABA597F16655F4B1EE4BB6160C59DF3CF4337C2BDB86E5DF
  666:d=1  hl=3 l= 129 prim: INTEGER           :D3D83DB6129BC85CC8DEAF64A403FBC71CC424E17C3AD41D912AF21C6404F46CCD54812A5EE2766D7B0E94903AE11DFDADD67FDCFD2DF45F103723DA959E7D5834003B591E5CA9F646A8C9C9F13E331A977810350555AE4C78E3AB0631051DB50B08CBC7D176165E10F752534244E0571A7FF72E11A463F7E1BAC7B51399A325
  798:d=1  hl=3 l= 129 prim: INTEGER           :A4C664A7E81ADF8C2078A741B1668A854ABB7FFEA57E1A86468A2289BDD7D8D963B07BBF5A99CD3504157D81859919536C56C4DE3C6C88D1DEAD55B396EB256EA7D3493A0D7290DE252BBA6B73E4DB66D85D65653B4A0222EC2D1A40FBE50A3EB2166EB2FA00F4C02FDA13E87BECF5354AA15AF348FC2E6AD8B49A9BD3C08BF7
  930:d=1  hl=3 l= 129 prim: INTEGER           :A704645AE8BEE32FABBA4D43A63FF1BFE0810FA6AA7FE2FDD096B03D0BEA101EBB9F751A47A679C204F3D0D30968B4716D1DA0BF44E877327FA149662AF1C256C8E0A9E9B013547872EADDF4957AF9656CA7DEF73E5677CD98BDBDF76AAC62E87A639BEDD4C92A074D8EFFDBDE725900B346D24502E9BD5B101F6715EEF700F5
 1062:d=1  hl=3 l= 129 prim: INTEGER           :C33FB6CD54E63505C6FB93391DD376BCD90C82075B5CA3C40154ACFE4C2F69F118C62F6C0F5272D64E27739428BFF076C6D10FC6B7C5E09658E82434F9781CE49478F8C701A8E2D0AFF494129A2C73E57ABF3BE042E3681390813ACB4BAFC0E2E6205C1DA21542AC91E7230D43AB23EFC1470FD2A25C486E4C0CD303E4ECA341

… we see that the file contains a sequences of 9 integers.

And openssl has even support to show us the semantics of those integers:

$ grep -v "RSA PRIVATE" id_rsa_test1 | base64 -d | openssl rsa -inform DER -text
Private-Key: (2048 bit)
modulus:
    00:c8:f4:64:17:43:7f:05:d4:91:17:2f:fe:60:10:
...
    1e:be:fe:20:c3:71:53:d2:25:a1:18:f5:77:8e:23:
    36:3b
publicExponent: 65537 (0x10001)
privateExponent:
    00:b3:40:64:c3:e7:9e:37:ba:2c:0e:0d:ee:8f:0c:
...
    fe:39:fe:78:cd:fb:ac:d7:0b:6d:7e:d0:d8:75:50:
    72:e9
prime1:
    00:f2:d7:12:4d:7a:ca:ac:2e:43:0c:64:92:d9:46:
...
    ab:a5:97:f1:66:55:f4:b1:ee:4b:b6:16:0c:59:df:
    3c:f4:33:7c:2b:db:86:e5:df
prime2:
    00:d3:d8:3d:b6:12:9b:c8:5c:c8:de:af:64:a4:03:
...
    10:f7:52:53:42:44:e0:57:1a:7f:f7:2e:11:a4:63:
    f7:e1:ba:c7:b5:13:99:a3:25
exponent1:
    00:a4:c6:64:a7:e8:1a:df:8c:20:78:a7:41:b1:66:
...
    2f:da:13:e8:7b:ec:f5:35:4a:a1:5a:f3:48:fc:2e:
    6a:d8:b4:9a:9b:d3:c0:8b:f7
exponent2:
    00:a7:04:64:5a:e8:be:e3:2f:ab:ba:4d:43:a6:3f:
...
    4d:8e:ff:db:de:72:59:00:b3:46:d2:45:02:e9:bd:
    5b:10:1f:67:15:ee:f7:00:f5
coefficient:
    00:c3:3f:b6:cd:54:e6:35:05:c6:fb:93:39:1d:d3:
...
    91:e7:23:0d:43:ab:23:ef:c1:47:0f:d2:a2:5c:48:
    6e:4c:0c:d3:03:e4:ec:a3:41
...
writing RSA key
-----BEGIN RSA PRIVATE KEY-----
...

These are the 8 last numbers we have seen in ASN.1 decoding above. The first one was zero, I’d guess it could be a version number, or possibly an indicator that there is no encryption, but I did not check the source or spec.

And of course we could have the same results a bit easier, because openssl assumes PEM even by default:

$ openssl rsa -text <id_rsa_test1

For efficiency reasons openssh does not store just the private key as required by the original RSA algorithm but a couple of precalculated values used by the implementation. For us not going into the cryptography here it is enough to know that the first 2 two numbers make up the public key, the remaining 6 numbers need to be kept private.

Yes, that means that the whole key pair is contained in the private key file. You
can lose/ignore/delete your public key file and always recreate it from the private key file:

$ ssh-keygen -y -f id_rsa_test1
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDI9GQXQ38F1JEXL/5gEN893g1ut0GFElQMCqLMxktw2fEcpVjfSP9vVvztmkdUSYjwwUJRCJNo3kB2A8/WbYwjhzPeUOwqy7EM9WnO7E5ZQxCkf+IDBfgSHnUnxhP6xyuDkqgO+vCyToUXzq2e6qzHIXCA/bdYMRRA+U6vYMV6EmWnODC9/lbOEo93hWZlTq0gQvj5AFOa+diq8JrYhZS3fniH/OcHJ6vxGYaF0i+emlJIstqYhwu2VEi74RNAZ3DPrG/3rXaRoIy3T803El7LqTpkY5t2TONiVy5EJhOk9SzXPciYvQR27yk4OYPV1x6+/iDDcVPSJaEY9XeOIzY7

If you compare it with the original key file id_rsa_test1.pub the only difference is an additional comment at the end. Functionality is the same regardless of the comment. This also explains why the AWS EC2 console only knows about pem files and no public key file is ever mentioned. They just produce it when needed.

This brings us to the next question. What is the format of an openssh public key file? Obviously it is ASCII and it contains 3 fields

  • the type of key
  • some base64 encoded data
  • an optional comment

So what does the base64 part contain?

$ awk '{print $2}' id_rsa_test1.pub | base64 -d | hexdump -C
00000000 00 00 00 07 73 73 68 2d 72 73 61 00 00 00 03 01 |....ssh-rsa.....|
00000010 00 01 00 00 01 01 00 c8 f4 64 17 43 7f 05 d4 91 |.........d.C....|
00000020 17 2f fe 60 10 df 3d de 0d 6e b7 41 85 12 54 0c |./.`..=..n.A..T.|
00000030 0a a2 cc c6 4b 70 d9 f1 1c a5 58 df 48 ff 6f 56 |....Kp....X.H.oV|
00000040 fc ed 9a 47 54 49 88 f0 c1 42 51 08 93 68 de 40 |...GTI...BQ..h.@|
00000050 76 03 cf d6 6d 8c 23 87 33 de 50 ec 2a cb b1 0c |v...m.#.3.P.*...|
00000060 f5 69 ce ec 4e 59 43 10 a4 7f e2 03 05 f8 12 1e |.i..NYC.........|
00000070 75 27 c6 13 fa c7 2b 83 92 a8 0e fa f0 b2 4e 85 |u'....+.......N.|
00000080 17 ce ad 9e ea ac c7 21 70 80 fd b7 58 31 14 40 |.......!p...X1.@|
00000090 f9 4e af 60 c5 7a 12 65 a7 38 30 bd fe 56 ce 12 |.N.`.z.e.80..V..|
000000a0 8f 77 85 66 65 4e ad 20 42 f8 f9 00 53 9a f9 d8 |.w.feN. B...S...|
000000b0 aa f0 9a d8 85 94 b7 7e 78 87 fc e7 07 27 ab f1 |.......~x....'..|
000000c0 19 86 85 d2 2f 9e 9a 52 48 b2 da 98 87 0b b6 54 |..../..RH......T|
000000d0 48 bb e1 13 40 67 70 cf ac 6f f7 ad 76 91 a0 8c |H...@gp..o..v...|
000000e0 b7 4f cd 37 12 5e cb a9 3a 64 63 9b 76 4c e3 62 |.O.7.^..:dc.vL.b|
000000f0 57 2e 44 26 13 a4 f5 2c d7 3d c8 98 bd 04 76 ef |W.D&...,.=....v.|
00000100 29 38 39 83 d5 d7 1e be fe 20 c3 71 53 d2 25 a1 |)89...... .qS.%.|
00000110 18 f5 77 8e 23 36 3b                            |..w.#6;|

This is a proprietary format invented by openssh, so this time we cannot
decode ASN.1. Still it’s pretty easy to read, so I skip writing a decoder.

  • a four byte big-endian length: 7 Bytes
  • 7 ASCII Bytes reading “ssh-rsa”
  • a four byte big-endian length: 3 Bytes
  • 01 00 01: This is the public exponent we have seen above in ASN.1 of the private key
  • a four byte big-endian length: 257 Bytes
  • the 257 bytes of public modulus we have seen above in the ASN.1 of the private key

This matches all nicely. So why do they need 257 bytes to store a 2048 bit modulus? This comes from ASN.1, which stores signed integers as 2’s complement. If the first bit of a positive integer is set a leading zero byte needs to be inserted to mark it as a positive integer.

The last question is how the fingerprints are calculated. The openssh fingerprint always refers to the public key. (After all private keys should not be shared so comparing fingerprints makes little sense.) The fingerprint is just calculated on the base64 decoded data field of the public key in openssh’s proprietary storage format:

$ awk '{print $2}' id_rsa_test1.pub | base64 -d | md5sum
c5cada07a97ff5b50a9ca3d0e76266b2 -
$ ssh-keygen -l -E md5 -f id_rsa_test1.pub
2048 MD5:c5:ca:da:07:a9:7f:f5:b5:0a:9c:a3:d0:e7:62:66:b2 foo@bar (RSA)

To get the same format using colon separators

$ awk '{print $2}' id_rsa_test1.pub | base64 -d | openssl md5 -c
(stdin)= c5:ca:da:07:a9:7f:f5:b5:0a:9c:a3:d0:e7:62:66:b2

Of course md5 is considered obsolete, so let’s do it using SHA256 (which has been selected by default in openssh for a while). Here we need to know that openssh does not show a hex string, but a base64 encoded result:

$ awk '{print $2}' id_rsa_test1.pub | base64 -d | openssl sha256 -binary | base64
nIw6/G+YUBiao6JPufR1JX+Mq/+kh44eYcMnmSrtSGw=
$ ssh-keygen -l -E sha256 -f id_rsa_test1.pub
2048 SHA256:nIw6/G+YUBiao6JPufR1JX+Mq/+kh44eYcMnmSrtSGw foo@bar (RSA)

AWS EC2 users might have wondered why the fingerprints shown on the EC2
console never match to anything shown by openssh. The reason is that AWS does not calculate the fingerprint from the openssh proprietary format of the public key, but from the private key in PKCS#8 format. Well, that’s a bit of a special case, you are indeed sharing your private key with AWS.

openssl pkcs8 -in AWS1.pem -nocrypt -topk8 -outform DER | openssl sha1 -c
(stdin)= 08:44:96:40:4a:35:65:5a:81:a2:fd:b1:cf:11:fa:82:46:77:98:2a

This post has already gotten too long so I’ll leave it to dig deeper into what this PKCS#8 contains. The following 2 commands should be enough of a hint for the curious reader:

$ openssl pkcs8 -in AWS1.pem -nocrypt -topk8 | openssl asn1parse
$ openssl rsa -outform DER


 

Posted in troubleshooting, underTheHood | Tagged , , , | Leave a comment

Local mail on Linux workstations

Posted on June 24, 2018 by usrmisc

Local mail on Linux workstations can still work very much like 20+ years ago on Unix. Every user has a mailbox under /var/spool/mail/. Some local services can produce administrative mails, namely cron jobs and security mails from sudo.

However, computing has changed a lot in the last 20 years. Most Linux workstations have only one (human) user. Often she or he is also the administrator (root). The user might also have several machines, at home, at work, a lightweight for travelling, a workhorse for software development. Many of these machines have no valid DNS name or it might change while travelling. And many users don’t want to and would never remember to check local mailboxes, whether there could be any output from a failed cron job. However, everybody should have a mail account somewhere in the “cloud” or at their ISP (unless they run their own mailserver). So the easiest would be just to have all local mail delivered to that single mailbox somewhere on the net.

In Ubuntu installations there is often no mailer (MTA) installed at all. In the past I have used ssmtp to get mail to the local root user delivered to the net. Unfortunately it does not work for mail from “normal” users (uid ≥ 1000). Also the program does not look like it’s actively maintained. So that might be a dead-end solution.

In OpenSUSE you always have postfix installed. I have tried uninstalling it in the past because it’s a massive overkill if your daily mails are handled somewhere else. However, the dependencies were too complicated so I have given up on that approach. And it might be useful to know if some daemon produces mail. So I have configured postfix to send all mail to my “normal” mailbox on the net.

First, having a mail server running on the internet might be dangerous, if you don’t know what you are doing and don’t spend enough time and effort to maintain and monitor it. Luckily in the default configuration in OpenSUSE makes postfix to listen on the loopback network interface only, so I don’t think you need to worried too much that it will be misused in some spam sending operations.

Unless the mail provider you are using accepts mail from senders without a valid DNS name (mine doesn’t), the first thing you need to do is to make sure that your mail gets a valid sender addresses. In postfix you do this by adding the desired mappings to the file /etc/postfix/sender_canonical

jdoe j.doe+from-jdoe-at-mach1@my-sp.foo
root j.doe+from-root-at-mach1@my-sp.foo

Here I assumed that

  • only users root and jdoe will ever send mail. I you have other senders, add respective lines
  • the mail account j.doe@my-sp.foo supports tags separated by a plus sign (e.g. gmail does that). This way you can easily see where some mail originated from. (Without the tags you still dig out all the information from the mail headers)

Next you have to make sure that mails sent to local accounts end up in your desired mail account. Analogously to the above a file /etc/postfix/recipient_canonical can be used. However, at least the default postfix configuration in OpenSUSE does not use such a file. So you need to do 3 steps

  1. Create the file /etc/postfix/recipient_canonical

    jdoe j.doe+to-jdoe-at-mach1@my-sp.foo
    root j.doe+to-root-at-mach1@my-sp.foo

    Again I assumed that only 1 user and root will ever receive local mail. However, the postfix configuration already comes with list a of aliases (like postmaster and abuse) which are mapped to root. So I’d assume mail to those would also be forwarded, although I haven’t seen the need to test it.

  2. Because this file is not previously configured in postfix you need to do so by adding the following lines to /etc/postfix/main.cf

    recipient_canonical_classes = envelope_recipient, header_recipient
    recipient_canonical_maps = hash:/etc/postfix/recipient_canonical

  3. Postfix requires the text file /etc/postfix/recipient_canonical to be converted into binary before usage. This can be done using the postmap command. However, OpenSUSE converts all known files automatically when the postfix service is (re)started. So it’s enough to add the name recipient_canonical into the POSTFIX_MAP_LIST in the file /etc/sysconfig/postfix

Here consumer grade Internet connections are not able to use port 25 freely in order to prevent private PCs being used as spam distributors. Outgoing traffic to that port is limited to the MTA of your own ISP. A final step is still needed to make the forwarding of mail work. Set the nexthop in /etc/postfix/transport.

* smtp:smtp.my-isp.foo

Of course this might be very different depending on country and ISP.

Finally restart the postfix service

sudo systemctl restart postfix

For testing send yourself a mail

$ mail -s "test mail" root
This mail was sent to root, but should end up in my "normal" mailbox
.
$

To read the logs you can use

$ sudo journalctl -b -u postfix

Open issues

  • TLS encryption
  • Authentication at SMTP server
Posted in HOWTO | Tagged , , | Leave a comment

Locking out the crackers

Posted on April 10, 2013 by usrmisc

A lot of crackers have their scripts and probably also botnets running to break in into machines listening on the internet via more or less known vulnerabilites and trivial passwords.

I recently happend to see 2 standard Linux tools to lock them out.

fail2ban scans the logs and upon certain failure message execeeding a given threshold it will block the source IP address of such activity in the firewall.

pam_tally2 counts authentication failures on PAM level and locks the user account if a threshold is execeeded.

Of course both approaches allow to configure the threshold, automatic unlocking etc. fail2ban is pretty freely configurable for many different purposes.

Another one I have just seen is http://denyhosts.sourceforge.net/ Looks like it has not been updated for a while.

(I’m not actively responsible for any machine listening openly on the internet, so I don’t follow that field explicitly. I wouldn’t be surprised if much more advanced tools exist.)

Posted in noteToSelf | Tagged , , | Leave a comment

Linux sessions, process groups, processes, threads and the ps command

Posted on February 19, 2013 by usrmisc

I don’t want to repeat the description how sessions, process groups, processes and threads all hang together in Linux. This description is pretty clear, there are plenty of others on the net.

I keep forgetting what’s the main feature of sessions and process groups though. So here is my reminder:

  • A session can contain one or more process groups.
  • A session can optionally have a controlling tty.
  • Signals can be delivered to a process group.
  • A shell with job control puts all processes of a pipe into one process group.
  • Signals caused by the terminal (like Ctrl-C) go to all processes in the foreground process group.

The following ps command shows the relevant information from sessions down to threads.

ps -emo pid,ppid,sess,tty,pgid,tid,cls,rtprio,nice,cputime,comm,cmd

Posted in HOWTO, underTheHood | Tagged | Leave a comment

Integer sizes in C on 32 bit and 64 bit Linux

Posted on December 27, 2012 by usrmisc

Q: How big is an int, long int etc. in C?

A: It depends. (The standard leaves it completely up to the compiler, which also means the same compiler can make it depend on options and target architecture.)

In practice I have not used anything else but gcc on Linux for a couple of years, so for myself the answer is a bit easier. However, because I don’t program C/C++ that often these days, each time I do so I soon tend to hit the question how big was that integer again, especially if interfacing with some low-level stuff and the code should work correctly on both 32 bit and 64 bit machines. At the moment I mostly use Intel architecture, so let me limit this post to Intel. (I have used a lot of ARM in the past and this week glibc with support for AArch64 came out, maybe the results can be checked against ARM later.)

type \ executable[1] 32 bit 64 bit
short int 16 bit 16 bit
int 32 bit 32 bit
long int 32 bit 64 bit
long long int 64 bit 64 bit
size_t 32 bit 64 bit
void* [2] 32 bit 64 bit

[1] A 32 bit executable can be used in a 64 bit user space (supposed a 32 bit loader and required shared libraries have been installed), a 32 bit user space can run on a 64 bit kernel and a 32 bit kernel can run on a 64 bit processor. So it’s really the word length of the executable that counts.
[2] In exotic cases pointers can have different lengths, http://stackoverflow.com/questions/6751749/size-of-a-pointer So, I’m not sure whether sizeof (void *) isn’t in fact undefined by the C standard. At least gcc compiles it without warning and returns a value, which looks correct for gcc on the Intel systems covered here.

The results where produced by the following piece of code:

#include <inttypes.h>
#include <stdio.h>

int main() {
  printf( "    short int: %zd\n" , sizeof(short int) ) ;
  printf( "          int: %zd\n" , sizeof(int) ) ;
  printf( "     long int: %zd\n", sizeof(long int) ) ;
  printf( "long long int: %zd\n", sizeof(long long int) ) ;
  printf( "       size_t: %zd\n", sizeof(size_t) ) ;
  printf( "        void*: %zd\n\n", sizeof(void *) ) ;


  printf( "PRIu32 usage (see source): %"PRIu32"\n" , (uint32_t) 42 ) ;
  return 0;
}

Slightly related the code also shows 2 features for 32/64 bit portable usage of printf. The “z” length modifier refers to size_t, see printf(3) for a couple of similar ones. The PRIu32 macro makes sure that a constant word length is used regardless of the compiler specific length of the integer types. This and several similar macros are in fact standardized in C99, they are defined in header inttypes.h.

P.S. A previous version of this post contained stupid copy paste errors resulting in wrong results. Hopefully all of them are fixed now.

Posted in programming, underTheHood | 8 Comments

Converting Epoch

Posted on August 17, 2012 by usrmisc

It shouldn’t be big news that Linux (like Unix) uses Epoch as its internal calendar. Epoch is number of seconds since Jan 1, 1970 00:00 GMT. Wikipedia can tell more.

The problem is just that as a humam it is not really possible to estimate what the epoch values stand for. From sometimes in 2011 to to sometimes in 2014 they start with 13 and have eight more digits. This month they even nicely start with 1345 and 6 more digits. But at least I don’t know of any other rules of thumb to make any more sense out of them.

Of course conversion can be programmed in any language. Or you can go to this online service.

But as a Linux user the following 2 shell commands might be the most handy solution:

$ date -d @1345678901
Thu Aug 23 02:41:41 EEST 2012
$ date -d "Thu Aug 23 02:41:41 EEST 2012" +%s
1345678901

Posted in HOWTO | Tagged , , , , | Leave a comment

Access control lists in Linux

Posted on January 7, 2011 by usrmisc

Well, I have seem them some 20 years ago in AIX. Access control lists (ACLs) for Unix/Linux files. At least they existed, but I have never seen them used. I have used Linux for quite some while and it has never appeared to me that they actually exist and are even used.

Normally access to a file is only granted in 3 levels: One for the owner, the second one for one group of users and the third one for everything else (“world”)

With ACLs you can give individual rights to many different users, not only to the owner. And you can give individual rights to many groups, not only one.

So how do you know an ACL is used?

$ ls -l /dev/dri/card0
crw-rw----+ 1 root video 226, 0 2011-01-07 21:25 /dev/dri/card0

There are 1 + 3 + 3 + 3 + 1 + 1 = 12 characters at the beginning of each listing.

  • The first one is the file type
  • Then there are 3 * 3 characters for owner’s, group’s and world’s access rights.
  • number 11 is the sticky bit
  • number 12 is a space in most cases, so I have never noticed it before. However if it’s not a space but a plus sign, the access rights are controlled by an ACL.

So how can you check the contents of the ACL? Interestingly enough although the Linux kernel supports ACLs, my Linux distro (Kubuntu) doesn’t seem to contain any user space tool to display them.

Well, that’s not a problem, the tool is easily installed.

$ sudo apt-get install acl

Using the following command you can check, which of your files actually have an explicit ACL:

$ sudo getfacl -R -s /

In my system the list is short. Just a couple of devices related to sound and the graphics adapter. So although these devices are owned by root, the currently logged in user gets rights to use these devices. Without ACL this would be not really be possible. If you wanted to still have them owned by root, then you would need to make them accessible to a whole group or even world.

So nothing dramatic here. Just interesting to see how things work.

If you want to know more, read the man page acl(5). (It comes with the installation above)

Posted in underTheHood | Tagged | 1 Comment

server4you — experiences after 2 days

Posted on June 26, 2010 by usrmisc

I ordered a virtual root server. My goal was just to get a machine that is always running and do various experimenting. I don’t expect much traffic and don’t need much resources. So I chose server4you, which was the cheapest offer I could find. Basically 54 EUR for the first year. And you need to remember to cancel the contract after 9 months in writing, if you don’t want to pay the double price for the next year.

Of course you get what you pay for. So for (one of) the cheapest on the market, I somehow expected some surprises. Additionally if somebody makes business with such an aggressive marketing and such price level, I’d always expect some strings attached. So I felt more comfortable to do business with a German company than e.g. with an American one, because German happens to be my mother tongue.

So ordering the server from http://www.server4you.de was somewhat more complicated than necessary. The German pages didn’t allow to enter any address outside Germany. Finally I managed to do so by changing the language to English. The other unpleasant feeling came from the fact that they have mandatory fields in their order form, which according to German law they are not allowed to ask (According to German law you are only allowed to ask data necessary to do the business in question)

There was no confirmation of the the contract sent by email. I would not call this good way to do business.

After 15 minutes or so I received an email that my server had been configured. Well, that was quick. But they hadn’t even asked me what operating system I wanted. Weird. And the email did not contain the address of the server. Even weirder. I waited another 15 minutes and then I got the idea that I could try to login to my account on their web site (you need to choose a user name and password during ordering)

Well, it didn’t let me in. Ok, maybe the database is just not updated yet. After 2 more hours it still didn’t let me in. Oh, they even have a free support hotline. (A toll free number in Germany, which you can reach fro free using Skype even from outside of Germany. Not sure whether they would serve you in any other language than German there.) They answered in less than a minute. Well, the only idea they had was wrong password. So they recommended to use the reset password functionality. Thanks and goodbye!

Hmm, but the resend password functionality claims that my user name is unknown. Another call to the hotline, another quick answer and some deeper investigations this time. Ahh, they have 2 different systems. http://www.server4you.de and http://www.server4you.net. And because I had to make my order in English, they had obviously created my server in the dot net system, although I had never entered that URL (and hadn’t even known that it existed)

Logging in to the dot net server, and yes, it works. Well, some unnecessary hassle, but quick and free customer service that actually solved the problem. That’s clearly better than average these days.

The first message when logging in: Your server has not yet been installed. Ahh, that’s how they got the server ready so quickly without even asking me about the OS.

Searching around a bit how things work and installing a minimal debian system.

After a couple of minutes I can log in. OK it’s really quite slim, only 160 packages and 426 MB of data. Nice to see such a clean system.

Se let’s see what it is. Configuration file /etc/apt/sources.list.d/debian.list tells me it is etch. Oh yeah, could be newer. And the sources file is not updated, the German mirror specified there has no etch repositories anymore. Well, I can update it myself. And indeed, the installation is pretty out of date, apt-get dist-upgrade pulls in 45 upgrades (out of 160 installed). And I guess etch hasn’t got many updates recently.

So how do I upgrade this beast to lenny? Ok debian has really good release notes. Hmm, the kernel. That was said in server4you’s FAQ, you can’t change the kernel of the virtual server. What kernel do I have? 2.6.9. Whaaat? I couldn’t even parse that number. we are at 2.6.35 these days. Ubuntu Hardy 08.04 LTS uses 2.6.28 and that sounds already very old to me. 2.6.9 is from October 2004!!! Googling a bit for the full string 2.6.9-023stab052.4 leads to http://kb.parallels.com/en/8556. Ahh, this is actually a rather recent version of the RHEL 4 kernel. Welcome to the real (slow) world of enterprise computing???

Anyway lenny seems to require only 2.6.8 or higher, I should be on the safe side. So updating sources.list to lenny, apt-get update and apt-get dist-upgrade. And now a reboot. Yes!!! It works.

In conclusion the feelings are mixed. Why do they provide such an old etch system even with non-functional sources.list configuration? But if you know how, you can relatively quickly upgrade it. Maybe they do that only that you can feel good over your own achievements… But definitely not the right choice for somebody who doesn’t want to repair and upgrade the system on day 1.

This posting is already too long, but there is one more story. The firewall. You can configure it from the management interface in the web. (Which will also apply the settings after reboot. Even though this nice feature is again undocumented) But to make a long story short, their web console is broken. The help in the web console tells how it opens outgoing connections if you have blocked incoming ones. But even if the help tells you how it does it, the implementation just doesn’t do it. Yet another call to the support. Quick answer again. But he knows nothing about firewalls, needs to connect to a colleague. After 5-7 minutes of waiting somebody seems to know something about firewalls. Well, yes it’s broken he admits after a while. He’ll pass it on to the development department.

Summary: It’s cheap. (Well if they didn’t manage to cheat me and I only pay what I expect). The user (or should I say admin) guidance could be better. Debian etch is a bit too old, especially such a broken configuration that you cannot install security updates right away. The kernel base version is extremely old, but maybe that’s common industry standard? The firewall configuration has a bug, wonder why a new customer finds that after a couple of hours. The free customer support worked much better for me so far than with many other companies (even those with generally good reputation)

Posted in diary | Tagged , , | 8 Comments

Solving a Dragon video player problem

Posted on June 15, 2010 by usrmisc

When Ubuntu Lucid 10.04 LTS came out recently I finally decided to
give KDE a try and installed Kubuntu.

One of the nice features for the newcomer (and probably everybody else
who can use a keyboard and is tired of overly long menu choices) is
the search field in the “K” menu. They call it “Kickoff Application
Launcher”, windows users would call it the “Start” menu. If you type a
few characters it will show all possible matches. The text doesn’t
need to match the program name, but will also be found in the program
description. Typing “vid” will reveal that there is a video player
called dragon.

That’s all nice, but unfortunately dragon did not play many of my
video clips. Well actually it played the video, but the sound was an
nearly unrecognizable noise.

Googling for the problem produced only one possible hit at
http://lists.debian.org/debian-kde/2010/04/msg00087.html, but the
solution suggested there did definitely not help.

After investigating many things I came to the conclusions

  • many multimedia formats did actually work fine. But clips using the
    FLV container format and an AAC audio stream didn’t. And because
    this combination is widely used on Youtube, they also happened to be
    in my collection.

  • it’s not a hardware problem, because when I boot the same machine to
    Ubuntu, totem videoplayer can play the sound without problems. The
    AAC audio codec is in libfaad and actually the shared library’s MD5
    checksum is identical in both my working Ubuntu installation and an
    the broken Kubuntu installation

So what next? Unfortunately I had no clue about dragon’s
architecture. So I decided to take an ltrace and see what it does,
while playing the undesired noise. (Interesting enough ltrace comes with
Kubuntu, no need to install it.)

ltrace -C -o dragon.ltrace dragon

To my surprise there was nothing related to codecs or pulseaudio in
the trace. (Is there some limitation that ltrace doesn’t show calls
from one shared library to another???) The few calls clearly related
to multimedia were all in phonon namespace.

Google lead me to http://phonon.kde.org/. OK, phonon is the top-level
multimedia framework in KDE. And it needs a backend to really play
media. Several backends are supported, but browsing the list of
installed packages reveals that xine-lib is used in Kubuntu.

xine-lib is at http://www.xine-project.org/home. The newest version is
1.1.18.1, but Kubuntu Lucid uses 1.1.17.1. The 1.1.18 release notes
mention promising changes indeed:

* Flash audio bug fixes, mostly concerning AAC.

Ok. So let’s just download and build lib-xine 1.1.18.1. But ./configure
--help is not a pleasant reading for the newcomer. There is a plethora
of configuration options and which ones should I use the get a version
that just works with my existing system???

Well, remember that Kubuntu is Debian based and (re-)building a
package is always the same command in Debian. So if I’d rebuild the
existing 1.1.17.1 based debian package, copy paste the configuration
to the 1.1.18.1 tarball things should be easy without really
understanding the details.

So here we go:

apt-get source libxine1
cd xine-lib-1.1.17
dpkg-buildpackage -b -rfakeroot

(Although I have done some building on this machine before and already
had quite some development packages installed I needed to install about
100 packages more, before building succeeded. However, all dependencies
were declared correctly, so I just needed to read the complaints about
missing packages. No intellectual work required here.)

By running ./config.status -V we can see the configuration chosen by
the Debian package.

So I copied that, just changed the installation prefix (because I prefer
to have tests with unknown outcome under my home directory clearly
separated from the installed software)

./configure '--build' 'i486-linux-gnu' '--host' 'i486-linux-gnu' \
'--prefix=/home/usrmisc/try/xine-lib/install' \
'--with-external-libmad' '--with-external-a52dec' \
'--with-external-libdts' '--with-external-ffmpeg' \
'--with-external-libmpcdec' '--enable-ipv6' '--with-jack' \
'--with-pulseaudio' '--with-libflac' '--with-wavpack' \
'--with-freetype' '--disable-vidix' '--enable-directfb' \
'--disable-nosefart' \
'LIBMODPLUG_LIBS=-lmodplug' 'CFLAGS=-g -O2 -g' \
'build_alias=i486-linux-gnu' 'host_alias=i486-linux-gnu' \
'LDFLAGS=-Wl,-Bsymbolic-functions' 'CPPFLAGS='

And now test it by overriding the system’s library path:

LD_LIBRARY_PATH=/home/usrmisc/try/xine-lib/install/lib dragon

Great! It just works. Problem was already fixed in the new version.

P.S. The next step is to find out how to get the new version of
lib-xine into Kubuntu. Ubuntu uses launchpad, but bugs for KDE
components are reported to KDE’s own bugzilla. But there is no bug in
the current version of the KDE application, just Kubuntu has an older
buggy one. How is that handled??? Let’s see whether understanding the
bureaucracy is quicker than solving the technical problem in this
case.

Posted in diary, troubleshooting | Tagged , , , , | Leave a comment

Other than English keyboard layout?

Posted on May 31, 2010 by usrmisc

Do you have an English keyboard? Well, you can probably skip this post, unless you are doing very international stuff.

The following are my experiences with various flavors of Ubuntu.

Normally the keyboard is selected during the installation. However, if you use LiveCDs, VMs, terminal servers (LTSP) it happens sometimes that the target machine assumes an English keyboard and it gets difficult to enter – / + ? and similar punctuation.

Here is how you change the layout.

1. Inside X-Windows things are easy. The command

setxkbmap layout

is all you need. I don’t remember where the layout parameter is defined, but for my Finnish keyboard it is “fi”. I guess Google is your friend when you can’t guess your own layout’s code.

2. On the console it’s not that easy.

The command is

sudo dpkg-reconfigure console-setup

That’s the easy part. It will walk you through many questions, I didn’t find all of them easy to answer.

Have a look at file /etc/default/console-setup to see your current settings and maybe even take a backup before running the command.

The process is not that fast, so if you only have to enter a couple of characters you might be faster to type blindly…

I guess internally reconfiguring runs the command setupcon. It has options to set only keyboad layout and skip the fonts (or vice versa). I have not tried it at all, not sure whether theres is any difference compared to invoking dpkg-reconfigure

Posted in HOWTO | Tagged , | Leave a comment