Yongbing's Blog

A personal technical note.

Verify the Signature of a X.509 Certificate

| Comments

Get the certificate

1

$ openssl s_client -showcerts -connect www.google.com:443 </dev/null > www.google.com.crt

then extract the top two certificates from www.google.com.crt:
Google’s (call it Google.pem):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

-----BEGIN CERTIFICATE-----
MIIEdjCCA16gAwIBAgIIOPbIrrvyFtIwDQYJKoZIhvcNAQEFBQAwSTELMAkGA1UE
BhMCVVMxEzARBgNVBAoTCkdvb2dsZSBJbmMxJTAjBgNVBAMTHEdvb2dsZSBJbnRl
cm5ldCBBdXRob3JpdHkgRzIwHhcNMTUwMzI1MTUyNTE0WhcNMTUwNjIzMDAwMDAw
WjBoMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwN
TW91bnRhaW4gVmlldzETMBEGA1UECgwKR29vZ2xlIEluYzEXMBUGA1UEAwwOd3d3
Lmdvb2dsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCSTWLd
zWsBLTP2zxKl/WKRtJCGerseT381b6i71CRADCg+57sU2o0SaCOYFKSQ99iOul6j
qe2N/ua2/4QUBalk1HsfdttvOaaZD9AkBgJI+oiVgCcdbdpqet6xU4QlJo+qjC2C
SGXaPzD3jUiIfUyQ2G8KipXwqcyVG3Vi/JjskdEywclBgYIEVlK/UQ01fxeSIBEm
3yMM12vKWLNnowCI5mBv57gCZVgr2aI182rGCipsJmd1l5tSUBNVxRx5J6ul3Q/6
+jnfokCz8hgmGIqYGLXpdh044VrKmA7xK6FL7gDKPYx/ZhErVs7c4sqL5n3YrKi1
lNkGyudSIX9nQZ5ZAgMBAAGjggFBMIIBPTAdBgNVHSUEFjAUBggrBgEFBQcDAQYI
KwYBBQUHAwIwGQYDVR0RBBIwEIIOd3d3Lmdvb2dsZS5jb20waAYIKwYBBQUHAQEE
XDBaMCsGCCsGAQUFBzAChh9odHRwOi8vcGtpLmdvb2dsZS5jb20vR0lBRzIuY3J0
MCsGCCsGAQUFBzABhh9odHRwOi8vY2xpZW50czEuZ29vZ2xlLmNvbS9vY3NwMB0G
A1UdDgQWBBRRAvQ0zUd+a1XkMqsEqYzLoPJ6FDAMBgNVHRMBAf8EAjAAMB8GA1Ud
IwQYMBaAFErdBhYbvPZotXb1gba7Yhq6WoEvMBcGA1UdIAQQMA4wDAYKKwYBBAHW
eQIFATAwBgNVHR8EKTAnMCWgI6Ahhh9odHRwOi8vcGtpLmdvb2dsZS5jb20vR0lB
RzIuY3JsMA0GCSqGSIb3DQEBBQUAA4IBAQB+xWJQGlk2iKfjkj694iLpUPYqDW9z
fg/BAcfZa8gpwL7TmBCFxAxxxW84+22gQM8sH9pqeeHpNCTmTLbtx6XREH+UKeHn
u1zFhYFTrc0WDGXOR9RbZGEiXESCEwmZKL1/3/s9+rFfX3AV1BS7ickw3QzNHqzW
d8xvLuSdkCGui0/q+UXX2sGkJET9ZTT9fuy6UGA7zOOCMaAuGtXQaiZMCwo5IRmO
godUOFa70CaggZm9s4Cci0+JzwIBl7T7459NgtZaIvoRj/DVByboBaBFNkfcJmzb
cIDZKTG45MHEWZJNzacNjUg+KPCqf1bTMYKjTxSs1bkYy4cTYrg1NvHv
-----END CERTIFICATE-----

and the issuer’s (call it Issuer.pem):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

-----BEGIN CERTIFICATE-----
MIID8DCCAtigAwIBAgIDAjp2MA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
YWwgQ0EwHhcNMTMwNDA1MTUxNTU1WhcNMTYxMjMxMjM1OTU5WjBJMQswCQYDVQQG
EwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzElMCMGA1UEAxMcR29vZ2xlIEludGVy
bmV0IEF1dGhvcml0eSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AJwqBHdc2FCROgajguDYUEi8iT/xGXAaiEZ+4I/F8YnOIe5a/mENtzJEiaB0C1NP
VaTOgmKV7utZX8bhBYASxF6UP7xbSDj0U/ck5vuR6RXEz/RTDfRK/J9U3n2+oGtv
h8DQUB8oMANA2ghzUWx//zo8pzcGjr1LEQTrfSTe5vn8MXH7lNVg8y5Kr0LSy+rE
ahqyzFPdFUuLH8gZYR/Nnag+YyuENWllhMgZxUYi+FOVvuOAShDGKuy6lyARxzmZ
EASg8GF6lSWMTlJ14rbtCMoU/M4iarNOz0YDl5cDfsCx3nuvRTPPuj5xt970JSXC
DTWJnZ37DhF5iR43xa+OcmkCAwEAAaOB5zCB5DAfBgNVHSMEGDAWgBTAephojYn7
qwVkDBF9qn1luMrMTjAdBgNVHQ4EFgQUSt0GFhu89mi1dvWBtrtiGrpagS8wEgYD
VR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAQYwNQYDVR0fBC4wLDAqoCig
JoYkaHR0cDovL2cuc3ltY2IuY29tL2NybHMvZ3RnbG9iYWwuY3JsMC4GCCsGAQUF
BwEBBCIwIDAeBggrBgEFBQcwAYYSaHR0cDovL2cuc3ltY2QuY29tMBcGA1UdIAQQ
MA4wDAYKKwYBBAHWeQIFATANBgkqhkiG9w0BAQUFAAOCAQEAJ4zP6cc7vsBv6JaE
+5xcXZDkd9uLMmCbZdiFJrW6nx7eZE4fxsggWwmfq6ngCTRFomUlNz1/Wm8gzPn6
8R2PEAwCOsTJAXaWvpv5Fdg50cUDR3a4iowx1mDV5I/b+jzG1Zgo+ByPF5E0y8tS
etH7OiDk4Yax2BgPvtaHZI3FCiVCUe+yOLjgHdDh/Ob0r0a678C/xbQF9ZR1DP6i
vgK66oZb+TWzZvXFjYWhGiN3GhkXVBNgnwvhtJwoKvmuAjRtJZOcgqgXe/GFsNMP
WOH7sf6coaPo/ck/9Ndx3L2MpBngISMjVROPpBYCCX65r+7bU2S9cS+5Oc4wt7S8
VOBHBw==
-----END CERTIFICATE-----

then extract the signature from Google.pem, and the RSA public key (used to decode the signature) from Issuer.pem:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

$ openssl x509 -in Google.pem -noout -text
    Signature Algorithm: sha1WithRSAEncryption
         7e:c5:62:50:1a:59:36:88:a7:e3:92:3e:bd:e2:22:e9:50:f6:
         2a:0d:6f:73:7e:0f:c1:01:c7:d9:6b:c8:29:c0:be:d3:98:10:
         85:c4:0c:71:c5:6f:38:fb:6d:a0:40:cf:2c:1f:da:6a:79:e1:
         e9:34:24:e6:4c:b6:ed:c7:a5:d1:10:7f:94:29:e1:e7:bb:5c:
         c5:85:81:53:ad:cd:16:0c:65:ce:47:d4:5b:64:61:22:5c:44:
         82:13:09:99:28:bd:7f:df:fb:3d:fa:b1:5f:5f:70:15:d4:14:
         bb:89:c9:30:dd:0c:cd:1e:ac:d6:77:cc:6f:2e:e4:9d:90:21:
         ae:8b:4f:ea:f9:45:d7:da:c1:a4:24:44:fd:65:34:fd:7e:ec:
         ba:50:60:3b:cc:e3:82:31:a0:2e:1a:d5:d0:6a:26:4c:0b:0a:
         39:21:19:8e:82:87:54:38:56:bb:d0:26:a0:81:99:bd:b3:80:
         9c:8b:4f:89:cf:02:01:97:b4:fb:e3:9f:4d:82:d6:5a:22:fa:
         11:8f:f0:d5:07:26:e8:05:a0:45:36:47:dc:26:6c:db:70:80:
         d9:29:31:b8:e4:c1:c4:59:92:4d:cd:a7:0d:8d:48:3e:28:f0:
         aa:7f:56:d3:31:82:a3:4f:14:ac:d5:b9:18:cb:87:13:62:b8:
         35:36:f1:ef
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

$ openssl x509 -in Issuer.pem -noout -text
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:9c:2a:04:77:5c:d8:50:91:3a:06:a3:82:e0:d8:
                    50:48:bc:89:3f:f1:19:70:1a:88:46:7e:e0:8f:c5:
                    f1:89:ce:21:ee:5a:fe:61:0d:b7:32:44:89:a0:74:
                    0b:53:4f:55:a4:ce:82:62:95:ee:eb:59:5f:c6:e1:
                    05:80:12:c4:5e:94:3f:bc:5b:48:38:f4:53:f7:24:
                    e6:fb:91:e9:15:c4:cf:f4:53:0d:f4:4a:fc:9f:54:
                    de:7d:be:a0:6b:6f:87:c0:d0:50:1f:28:30:03:40:
                    da:08:73:51:6c:7f:ff:3a:3c:a7:37:06:8e:bd:4b:
                    11:04:eb:7d:24:de:e6:f9:fc:31:71:fb:94:d5:60:
                    f3:2e:4a:af:42:d2:cb:ea:c4:6a:1a:b2:cc:53:dd:
                    15:4b:8b:1f:c8:19:61:1f:cd:9d:a8:3e:63:2b:84:
                    35:69:65:84:c8:19:c5:46:22:f8:53:95:be:e3:80:
                    4a:10:c6:2a:ec:ba:97:20:11:c7:39:99:10:04:a0:
                    f0:61:7a:95:25:8c:4e:52:75:e2:b6:ed:08:ca:14:
                    fc:ce:22:6a:b3:4e:cf:46:03:97:97:03:7e:c0:b1:
                    de:7b:af:45:33:cf:ba:3e:71:b7:de:f4:25:25:c2:
                    0d:35:89:9d:9d:fb:0e:11:79:89:1e:37:c5:af:8e:
                    72:69
                Exponent: 65537 (0x10001)

Decode certificate’s signature with issuer’s RSA public key

1
2
3
4
5

$ python
>>> signature = 0x7ec562501a593688a7e3923ebde222e950f62a0d6f737e0fc101c7d96bc829c0bed3981085c40c71c56f38fb6da040cf2c1fda6a79e1e93424e64cb6edc7a5d1107f9429e1e7bb5cc5858153adcd160c65ce47d45b6461225c448213099928bd7fdffb3dfab15f5f7015d414bb89c930dd0ccd1eacd677cc6f2ee49d9021ae8b4feaf945d7dac1a42444fd6534fd7eecba50603bcce38231a02e1ad5d06a264c0b0a3921198e8287543856bbd026a08199bdb3809c8b4f89cf020197b4fbe39f4d82d65a22fa118ff0d50726e805a0453647dc266cdb7080d92931b8e4c1c459924dcda70d8d483e28f0aa7f56d33182a34f14acd5b918cb871362b83536f1ef
>>> modulus = 0x009c2a04775cd850913a06a382e0d85048bc893ff119701a88467ee08fc5f189ce21ee5afe610db7324489a0740b534f55a4ce826295eeeb595fc6e1058012c45e943fbc5b4838f453f724e6fb91e915c4cff4530df44afc9f54de7dbea06b6f87c0d0501f28300340da0873516c7fff3a3ca737068ebd4b1104eb7d24dee6f9fc3171fb94d560f32e4aaf42d2cbeac46a1ab2cc53dd154b8b1fc819611fcd9da83e632b8435696584c819c54622f85395bee3804a10c62aecba972011c739991004a0f0617a95258c4e5275e2b6ed08ca14fcce226ab34ecf46039797037ec0b1de7baf4533cfba3e71b7def42525c20d35899d9dfb0e1179891e37c5af8e7269
>>> print "%x" % pow( signature, 65537, modulus )
1ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff003021300906052b0e03021a05000414f97eff4a56c70bf2a2cdf0546b1efedcb2d8c870

The leading “1fff…” of the result is the standard RSA algorithm padding, following “3021300906052b0e03021a05000414” is the “ASN.1 BER SHA1 algorithm designator prefix”, so the actual decoded SHA1 hash value is the remaining “f97eff4a56c70bf2a2cdf0546b1efedcb2d8c870”.

Compare the decoded signature value with certificate’s hash

As defined in RFC 3280, X.509 certificate’s ANS.1 format is

1
2
3
4

Certificate  ::=  SEQUENCE  {
     tbsCertificate       TBSCertificate,
     signatureAlgorithm   AlgorithmIdentifier,
     signatureValue       BIT STRING  }

The decoded SHA1 hash value is tbsCertificate’s hash value, not the whols certificate’s hash value (the output of “openssl x509 -noout -in Google.pem -fingerprint -sha1”).

To extract tbsCertificate from the certificate, we need to convert it from PEM format to DER format (the binary format) first:

1

$ openssl x509 -in Google.pem -inform PEM -out Google.crt -outform DER

Then use “openssl asn1parse” to find the offset and size of tbsCertificate in the certificate:

1
2
3
4
5
6
7
8
9

$ openssl asn1parse -inform der -in Google.crt
    0:d=0  hl=4 l=1142 cons: SEQUENCE
    4:d=1  hl=4 l= 862 cons: SEQUENCE
    8:d=2  hl=2 l=   3 cons: cont [ 0 ]
   10:d=3  hl=2 l=   1 prim: INTEGER           :02
   13:d=2  hl=2 l=   8 prim: INTEGER           :38F6C8AEBBF216D2
   23:d=2  hl=2 l=  13 cons: SEQUENCE
   25:d=3  hl=2 l=   9 prim: OBJECT            :sha1WithRSAEncryption
...

Based on certificate’s ANS.1 definition, the second line represents tbsCertificate’s offset (in the binary format DER) 4, size 866( head length 4, body length 862). So it can be extracted by:

1

$ dd if=Google.crt of=Google.tbsCertificate skip=4 bs=1 count=866

And its SHA1 hash matches the decoded value:

1
2

$ sha1sum Google.tbsCertificate
f97eff4a56c70bf2a2cdf0546b1efedcb2d8c870  Google.tbsCertificate

Create My Own Repo to Manage Bundle of Git Repositories

| Comments

Create repo manifest:

  1. As gerrit server administrator, create below pojects in gerrit server
    1. container:for global access control, choose “Only serve as parent for other projects”, unselect “Create initial empty commit”
    2. container/repo_1/manifest:repo manifest, access right inherit from “container”, unselect “Only serve as parent for other projects”, select “Create initial empty commit”
    3. container/repo_1/git_one:child project inside the repo, access right inherit from “container”, unselect “Only serve as parent for other projects”, select “Create initial empty commit”.
    4. container/repo_1/git_two
  2. Create my own repo config:
    1. on a client working machine, checkout the manifest git first:
1

~$ git clone ssh://gerritreview.com:29418/container/repo_1/manifest

b. add a new file manifest.xml to the git as below, and merge it to master branch: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
        <remote  name="my_repo"
                fetch="ssh://gerritreview.com:29418/"
                review="https://gerritreview.com/gerrit/"
                />
        <default revision="my_working_branch"
                remote="my_repo"
                sync-j="4"
                />

        <project path="container/repo_1/git_one" name="container/repo_1/git_one" />
        <project path="container/repo_1/git_two" name="container/repo_1/git_two" />
</manifest>

3. On gerrit server, create a working “Branch Name” “my_working_branch” on these three projects, with “Initial Revision” set to “master” in https://gerritreview.com/gerrit/#/admin/projects/container/repo_1/${GIT_NAME},branches
4. On a client’s working machine, checkout the repo with below command: 

1
2

~$ repo init -u ssh://gerritreview.com:29418/container/repo_1/manifest -b my_working_branch -m manifest.xml --repo-url https://chromium.googlesource.com/external/repo.git --no-repo-verify
~$ repo sync -j 32

Setup the access control rules for this repo:

1. define thress group and their access right:
          developers: can submit CL, checkout code, review +/_ 1
          reviewers: can review +/- 2
          submitters: can merge CL to working branch
2. enforce these rules on gerrit server:
First delete all default access permission granted for “Registered Users” from https://gerritreview.com/gerrit/#/admin/projects/All-Projects,access
Then create below access rules for the projects under container:

Config Gerrit Server Behind Apache Https Reverse-proxy

| Comments

I want to setup a secure gerrit server for a small developer group within intranet, I choose Apache as its reverse-proxy server, and use HTTP as gerrit server’s auth type, becasue I only want a few selected people to see the server, so no LDAP.

Here’s the final web view from a registered developer:

Here’s the gerrit server config:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

[gerrit]
  basePath = git
  canonicalWebUrl = https://gerritreview.com/gerrit
[database]
  type = mysql
  hostname = localhost
  database = reviewdb
  username = gerrit2
  password = 12345678
[auth]
  type = HTTP
[sendemail]
  smtpServer = localhost
[container]
  user = gerrit2
  javaHome = /usr/lib/jvm/java-7-openjdk-amd64/jre
[sshd]
  listenAddress = *:29418
[httpd]
  listenUrl = proxy-https://localhost:8080/gerrit
[cache]
  directory = cache
[index]
  type = LUCENE

And the changes I made upon default Apache HTTPS site config:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

--- /etc/apache2/sites-available/default-ssl.conf    2014-01-07 05:23:42.000000000 -0800
+++ /etc/apache2/sites-available/000-default.conf 2015-03-25 14:41:20.867255345 -0700
@@ -130,6 +130,71 @@
      # MSIE 7 and newer should be able to use keepalive
      BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
 
+               ServerName gerritreview.com
+               ProxyRequests Off
+               ProxyVia Off
+               ProxyPreserveHost On
+               <Proxy *>
+                       Order deny,allow
+                       Allow from all
+               </Proxy>
+ 
+               <Location />
+                       AuthType Basic
+                       AuthName "Gerrit Code Review"
+                       Require valid-user
+                       AuthBasicProvider file
+                       AuthUserFile /etc/apache2/passwords
+               </Location>
+
+               AllowEncodedSlashes On
+               SSLProxyEngine On
+               SSLProxyVerify none
+               SSLProxyCheckPeerCN off
+               SSLProxyCheckPeerName off
+ 
+               ProxyPass /gerrit/ http://localhost:8080/gerrit/ nocanon
+               ProxyPassReverse /gerrit/ http://localhost:8080/gerrit/
+               # is this necessary?  
+               Header edit Location "^http:(.*)$" "https:$1"
+
  </VirtualHost>
 </IfModule>

After setup, this gerrit server was deployed in a kvm guest machine, connected to its kvm host through an isolated virtual bridge. Allowing bidirectional access to tcp port 29418 (gerrit ssh), 443 (HTTPS), 25 (sendmail), as below command:

1
2
3
4
5
6
7
8

#forward kvm host's incoming (from NIC eth0) tcp dst port 29418 to gerrit server vm.  
iptables -I FORWARD -i eth0 -p tcp -m state --state NEW,RELATED,ESTABLISHED -m tcp -d $VM_GUEST_IP/32 -dport 29418 -j ACCEPT
#any incoming packets from interface eth0, protocol tcp, dst port 29418 will be applied DNAT function (replace the dst addr from kvm host to $VM_GUEST_IP)
iptables -t nat -i eth0 -I PREROUTING -p tcp  --dport 29418 -j DNAT --to $VM_GUEST_IP:29418
#replace any outboud tcp/29418 packet from $VM_GUEST_IP with kvm host's addr, and push to host's NIC eth0
iptables -t nat -A POSTROUTING -p tcp -o eth0 -s $VM_GUEST_IP --sport 29418 -j MASQUERADE
#forward outgoing tcp/29418 connect from $VM_GUEST_IP to host's NIC eth0
iptables -I FORWARD -o eth0 -p tcp -m state --state NEW,RELATED,ESTABLISHED -m tcp -s $VM_GUEST_IP --sport 29418 -j ACCEPT

Also NAT rules to allow connection from the vm guest (gerrit server) to connect to a NTP server:

1
2
3

iptables -t nat -A POSTROUTING -p udp -s $VM_GUEST_IP/32 -d 174.137.132.100 -dport 123 -j MASQUERADE
iptables -I FORWARD -p udp -s $VM_GUEST_IP/32 -d 174.137.132.100/32 -dport 123 -j ACCEPT
iptables -I FORWARD -p udp -d $VM_GUEST_IP/32 -s 174.137.132.100/32 -dport 123 -j ACCEPT

Notes:

Decode Instruction Address in OOPS to C Code File:line

| Comments

The OOPS message:

1
2
3
4
5
6
7
8
9
10
11
12
13

[ 2405.090047] Unable to handle kernel paging request at virtual address 008b2005
[ 2405.097586] pgd = 80004000
[ 2405.100427] [008b2005] *pgd=00000000
[ 2405.104187] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[ 2405.109673] Modules linked in: mbtusbchar usbfwdnld fusion(O) gal3d amp_core(O)
[ 2405.117310] CPU: 1    Tainted: G        W  O  (3.4.50+ #11)
[ 2405.123072] PC is at module_put+0x44/0x8c
[ 2405.127219] LR is at cdev_put+0x24/0x28
[ 2405.131182] pc : [<8007521c>]    lr : [<800c97d0>]    psr: 20000013
[ 2406.111129] Backtrace:
[ 2406.113680] [<800751d8>] (module_put+0x0/0x8c) from [<800c97d0>] (cdev_put+0x24/0x28)
[ 2406.121764]  r4:45a8e010 r3:ffffffff
[ 2406.125491] [<800c97ac>] (cdev_put+0x0/0x28) from [<800c7410>] (fput+0x21c/0x22c)

Method A: Recompile vmlinux or failed module with debug info enabled if possible (CONFIG_DEBUG_INFO=y), then gdb can give out the C source code line of the failed instruction:

1
2

$ make $MY_DEFAULT_CONFIG_FILE -C $LINUX_ROOT_DIR CROSS_COMPILE=arm-linux-gnueabi- ARCH=arm
$ make $MY_UIMAGE_NAME -C $LINUX_ROOT_DIR CROSS_COMPILE=arm-linux-gnueabi- ARCH=arm CONFIG_DEBUG_INFO=y -j32

Recompile individual module if needed:

1

$ make modules -C $LINUX_ROOT_DIR CROSS_COMPILE=arm-linux-gnueabi- ARCH=arm CONFIG_DEBUG_INFO=y -j32

Use gdb to get C source file:line info from given instruction address:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

$ arm-linux-androideabi-gdb –f vmlinux
Reading symbols from linux/vmlinux...done.
(gdb) list *(module_put+0x44)
0x80073964 is in module_put (kernel/module.c:942).
937     void module_put(struct module *module)
938     {
939             if (module) {
940                     preempt_disable();
941                     smp_wmb(); /* see comment in module_refcount */
942                     __this_cpu_inc(module->refptr->decs);
943
944                     trace_module_put(module, _RET_IP_);
945                     /* Maybe they're waiting for us to drop reference? */
946                     if (unlikely(!module_is_live(module)))
(gdb) disas (module_put+0x44)
Dump of assembler code for function module_put:
   0x80073960 <+64>:    add     r3, r3, #4
   0x80073964 <+68>:    ldr     r1, [r3, r2]

Method B: If can not recompile the vmlinux/module, but the problematic vmlinux/module file is available, use objdump to get the assembly line, but no C source line available.

1

$ arm-linux-androideabi-objdump -dSl vmlinux >vmlinux.disasm

module_put+0x44 = 0x800751d8 + 0x44 = 0x8007521c:

1
2
3

800751d8 <module_put>:
module_put():
8007521c:       e7931002        ldr     r1, [r3, r2]

Compared with method A, method B get the same instruction line.

Appendix: only for comparison/reference: use objdump to decode an object file with debug info:

1

$ arm-linux-androideabi-objdump -dSlr kernel/module.o >kernel/module.disasm

Check if module_put+0x44 = 0xf28+0x44 = 0xf6c point to the same C source line and assembly line:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

 00000f28 <module_put>:
  module_put():
  linux/kernel/module.c:938
          return ret;
  }
  EXPORT_SYMBOL(try_module_get);

  void module_put(struct module *module)
  {
       f28:       e1a0c00d        mov     ip, sp
       f2c:       e92dd818        push    {r3, r4, fp, ip, lr, pc}
       f30:       e24cb004        sub     fp, ip, #4
  linux/kernel/module.c:939
          if (module) {
       f34:       e3500000        cmp     r0, #0
       f38:       089da818        ldmeq   sp, {r3, r4, fp, sp, pc}
  current_thread_info():
  linux/arch/arm/include/asm/thread_info.h:97
       f3c:       e1a0300d        mov     r3, sp
       f40:       e3c34d7f        bic     r4, r3, #8128   ; 0x1fc0
       f44:       e3c4403f        bic     r4, r4, #63     ; 0x3f
  module_put():
  linux/kernel/module.c:940
                  preempt_disable();
       f48:       e5943004        ldr     r3, [r4, #4]
       f4c:       e2833001        add     r3, r3, #1
       f50:       e5843004        str     r3, [r4, #4]
  linux/kernel/module.c:941
                  smp_wmb(); /* see comment in module_refcount */
       f54:       f57ff05f        dmb     sy
  linux/kernel/module.c:942
                  __this_cpu_inc(module->refptr->decs);
       f58:       e59f2050        ldr     r2, [pc, #80]   ; fb0 <module_put+0x88>
       f5c:       e5941014        ldr     r1, [r4, #20]
       f60:       e5903138        ldr     r3, [r0, #312]  ; 0x138
       f64:       e7922101        ldr     r2, [r2, r1, lsl #2]
       f68:       e2833004        add     r3, r3, #4
       f6c:       e7931002        ldr     r1, [r3, r2]
       f70:       e2811001        add     r1, r1, #1
       f74:       e7831002        str     r1, [r3, r2]
  linux/kernel/module.c:946

Atomic File Writing

| Comments

QA reported a bug, sometimes when power cycle target board right after pairing a bluetooth device, the whole bluetooth paring info lost. I found out the root cause is that the configure file of bluedroid lost all content in this case. Turned out the bluedroid configure file writing operation is not atomic. I came up with a revised file writing process, to guarantee it’s atomic:

  1. copy the configure file to a temporary file.
  2. write the update content to the temporary file.
  3. fsync the temporary file. (the step that bluedroid missed)
  4. rename the temporary file to configure file.

Because rename is atomic, and all steps before it are revertable (will not affect the configure file), so the whole process is atomic.

(atomic_write.c) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143

/*
 * Copyright (C) 2013 Yongbing Chen <yongbing.chen.wh@gmail.com>
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

#include <stdio.h>
#include <string.h>
#include <fcntl.h>
#include <errno.h>
#include <stdlib.h>

#define CHECK_RESULT(x) \
 do { \
     if (x != 0){ \
         printf("error %s\n", strerror(errno)); \
         return -1; \
     } \
 }while (0);

#ifndef TEMP_FAILURE_RETRY 
#define TEMP_FAILURE_RETRY(x) \
 ({ \
  int _result; \
  do _result = (int) (x); \
  while (_result == -1 && errno == EINTR); \
  _result; \
  })
#endif


static int copy_file(const char *src_file, const char *dst_file)
{
  int ret = -1;
  FILE *fp1, *fp2;
  fp1 = fopen(src_file,"r");
  if(fp1 == NULL){
      printf("open %s error %s\n",src_file, strerror(errno));
      return ret;
  }

  fp2 = fopen(dst_file,"w");
  if(fp2 == NULL){
      printf("open %s error %s\n", dst_file, strerror(errno)); fclose(fp1);
      return ret;
  }

  fseek(fp1, 0 , SEEK_END);
  int file_size = ftell(fp1);
  fseek(fp1, 0 , SEEK_SET);
  if (file_size == 0){
      fclose(fp1); fclose(fp2);
      return 0;
  }
  char *buffer = (char*)malloc(file_size);
  if (NULL == buffer){
      printf("error %s\n", strerror(errno)); fclose(fp1); fclose(fp2);
      return -1;
  }

  printf("copy len %d from %s to %s\n", file_size, src_file, dst_file);
  ret = fread(buffer, file_size, 1, fp1);
  if (ret != 1){
      printf("ret %d, error %s\n",ret, strerror(errno));
      goto error;
  }

  ret = fwrite(buffer, file_size, 1, fp2);
  if (ret != 1){
      printf("ret %d, error %s\n",ret, strerror(errno));
      goto error;
  }
  ret = 0;

error:
  fclose(fp1);
  fclose(fp2);
  free(buffer);
  return ret;
}


static int update_config_to_tmp_file(const char* curr_file, const char* file_name)
{
  int fd = open(file_name, O_CREAT | O_APPEND | O_RDWR, 0660);
  if(fd > 0){
      if(access(curr_file,  F_OK) == 0)
          CHECK_RESULT(copy_file(curr_file, file_name));
      int test_data = rand();
      printf("writing data %d to config file %s\n", test_data, file_name);
      int data_write = TEMP_FAILURE_RETRY(write(fd, &test_data, sizeof(test_data)));
      if (data_write < 0 || data_write != sizeof(test_data)){
          printf("%s failed len:%d %s",__func__, data_write, strerror(errno));
          return -1;
      }
  }
  else
      return -1;

  close(fd);
  return 0;

}

static int sync_conf_file(const char* file_name)
{
  int ret = -1;
  int fd = open(file_name, O_RDONLY);
  if(fd > 0){
      ret = fsync(fd);
  }
  close(fd);
  return ret;
}

int atomic_update_config_file()
{
  const char* file_name = "config";
  const char* file_name_new = "config.new";
  CHECK_RESULT(update_config_to_tmp_file(file_name, file_name_new));//open, write, close inside this operation.
  CHECK_RESULT(sync_conf_file(file_name_new));//open, fsync, close to achive a sync.
  CHECK_RESULT(rename(file_name_new, file_name));
  return 0;
}

int main(int argc, char **argv)
{
  int ret = atomic_update_config_file();
  if (ret != 0){
      printf("update config failed, keeping original one.\n");
  }
  return 0;
}

Reference:

Android Bluetooth: Pairing a HID Device

| Comments

  1. User started scanning from Bluetooth Settings UI, Android Bluetooth service responded to this request, calling bluedroid to start discovery.
  2. Bluedroid found nearby devices in discovery mode, reported them through device found callback.
  3. User selected one device from found devices, started to pairing it, Android Bluetooth service called bluedroid to creat bond with it.
  4. Bluedroid requested PIN code from end user (simple secure pairing mode has different procedure).
  5. Bluedroid started SDP process to find remote device’s UUID.
  6. After SDP finished, bluedroid called remote device property changed on UUID changed to notify Android.
  7. Android received the event, then HID profile service started to connect it. Without this SDP event, the pair will fail due to no further action over L2CAP anymore, see pairing failed by SDP for example.
  8. In HID profile connection process, bluedroid conducted another round of SDP, fetched HID descriptor from remote device, created hidraw and input device file for the new remote device, through uhid interface, the remote device was ready to use from that point.

Logs and call stack:

Full logs:

Misc Code Samples

| Comments

1 inotify: monitoring hidraw device file add/remove events:

(inotify-example.c) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201

/*
 * Copyright (C) 2005 The Android Open Source Project
 * Copyright (C) 2013 Yongbing Chen <yongbing.chen.wh@gmail.com>
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

#include <string.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/types.h>
#include <poll.h>
#include <sys/epoll.h>
#include <sys/inotify.h>
#include <pthread.h>
#include <unistd.h>

static const unsigned int EPOLL_ID_INOTIFY = 0x80000001;
#define MONITORING_DIR "/dev"
#define HIDRAW_DEV_FILE_PREFIX "hidraw"

static int init_epoll_fd()
{
  int epoll_fd = epoll_create(1);
  if (epoll_fd < 0){
      printf("epoll_crate failed %s",strerror(errno));
      return -1;
  }
  return epoll_fd;
}

static int init_inotify(char * dev_path, int epoll_fd)
{
  int inotify_fd = inotify_init();
  if(inotify_fd < 0){
      printf("inotify_init failed, errno %s",strerror(errno));
      return -1;
  }

  int result = inotify_add_watch(inotify_fd, dev_path, IN_DELETE | IN_CREATE);
  if(result < 0){
      printf("Could not register INotify for %s  errno %s", dev_path, strerror(errno));
      close(inotify_fd);
      return -1;
  }
  return inotify_fd;
}

static int add_inotify_to_epoll(int epoll_fd, int inotify_fd)
{
  struct epoll_event event_item;
  memset(&event_item, 0, sizeof(event_item));
  event_item.events = EPOLLIN;
  event_item.data.u32 = EPOLL_ID_INOTIFY;
  int result = epoll_ctl(epoll_fd, EPOLL_CTL_ADD, inotify_fd, &event_item);
  if (result != 0){
      printf("Could not add INotify to epoll instance.  errno %s", strerror(errno));
      return -1;
  }
  return 0;
}

static void process_add_event(char *hidraw_file_name)
{
  printf("%s, %s\n", __func__, hidraw_file_name);
  //may add the file to epoll here.
}

static void process_remove_event(char *hidraw_file_name)
{
  printf("%s, %s\n", __func__, hidraw_file_name);
  //may remove the file from epoll here.
}

//Cite from Android EventHub.cpp
static int read_inotify(int inotify_fd)
{
    int rc;
    char devname[128];
    char *filename;
    char event_buf[512];
    int event_size;
    int event_pos = 0;
    struct inotify_event *event;

    printf("read_inotify fd: %d\n", inotify_fd);
    rc = TEMP_FAILURE_RETRY(read(inotify_fd, event_buf, sizeof(event_buf)));
    if(rc < (int)sizeof(*event)) {
        printf("could not get event, %s\n", strerror(errno));
        return -1;
    }

    strcpy(devname, MONITORING_DIR);
    filename = devname + strlen(devname);
    *filename++ = '/';

    while(rc >= (int)sizeof(*event)) {
        event = (struct inotify_event *)(event_buf + event_pos);
        printf("%d: %08x \"%s\"\n", event->wd, event->mask, event->len ? event->name : "");
        if(event->len && !strncmp(event->name, HIDRAW_DEV_FILE_PREFIX, strlen(HIDRAW_DEV_FILE_PREFIX))) {
            strcpy(filename, event->name);
            if(event->mask & IN_CREATE) {
                process_add_event(devname);
            } else {
                process_remove_event(devname);
            }
        }
        event_size = sizeof(*event) + event->len;
        rc -= event_size;
        event_pos += event_size;
    }
    return 0;
}

static void process_inode_event(struct epoll_event *events,
      int event_counts, int inotify_fd)
{
  int i;
  for (i = 0; i < event_counts; i++){
      if (events->data.u32 == EPOLL_ID_INOTIFY){
          if (events->events & EPOLLIN) {
              read_inotify(inotify_fd);
          } else {
              printf("received unexpected epoll event 0x%08x for inotify.", events->events);
          }
      }
      events++;
  }
}

int main(int argc, char **argv)
{
  int epoll_fd = init_epoll_fd();
  if (epoll_fd < 0)
      return EXIT_FAILURE;

  int inotify_fd = init_inotify(MONITORING_DIR, epoll_fd);
  if (inotify_fd < 0){
      close(epoll_fd);
      return EXIT_FAILURE;
  }

  int rc = add_inotify_to_epoll(epoll_fd, inotify_fd);
  if (rc < 0){
      close(epoll_fd);close(inotify_fd);
      return EXIT_FAILURE;
  }

  static const int EPOLL_MAX_EVENTS = 16;
  struct epoll_event epoll_events[EPOLL_MAX_EVENTS];
  while(1){
      int rc = TEMP_FAILURE_RETRY(epoll_wait(epoll_fd, epoll_events, EPOLL_MAX_EVENTS, -1));
      if (rc <= 0) {
          printf("poll failed errno=%s\n", strerror(errno));
          return 1;
      } else {
          process_inode_event(epoll_events, rc, inotify_fd);
      }
  }
  close(inotify_fd);
  close(epoll_fd);

  return EXIT_SUCCESS;
 }

#ifdef __RUNNING_LOG
 130|shell@android:/data/app # ./test_inotify
[ 4998.540741] usb 1-1: USB disconnect, device number 3
read_inotify fd: 4
1: 00000200 "hidraw0"
Removing device '/dev/hidraw0' due to inotify event
process_remove_event, /dev/hidraw0
read_inotify fd: 4
1: 00000200 "hidraw1"
Removing device '/dev/hidraw1' due to inotify event
process_remove_event, /dev/hidraw1
[ 5002.247269] usb 1-1: new full-speed USB device number 4 using berlin-ehci
[ 5002.406589] usb 1-1: New USB device found, idVendor=046d, idProduct=c52b
[ 5002.413862] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 5002.421457] usb 1-1: Product: USB Receiver
[ 5002.425927] usb 1-1: Manufacturer: Logitech
read_inotify fd:[ 5002.443452] logitech-djreceiver 0003:046D:C52B.000B: hiddev0,hidraw0: USB HID v1.11 Device [Logitech USB Receiver] on usb-f7ed0000.usb-1/input2
 4
1: 00000100 "hidraw0"
process_add_event, /dev/hidraw0
[ 5002.464392] input: Logitech Unifying Device. Wireless PID:2012 as /devices/soc.0/f7ed0000.usb/usb1/1-1/1-1:1.2/0003:046D:C52B.000B/input/input4
read_inotify fd:[ 5002.479144] logitech-djdevice 0003:046D:C52B.000C: input,hidraw1: USB HID v1.11 Keyboard [Logitech Unifying Device. Wireless PID:2012] on usb-f7ed0000.usb-1:1
 4
1: 00000100 "hidraw1"
process_add_event, /dev/hidraw1
#endif

2 pthread timer_create: periodically generate an event, as a heartbeat:

(pthread_timer-example.c) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125

/*
 * Copyright (C) 2013 Yongbing Chen <yongbing.chen.wh@gmail.com>
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

#include <stdlib.h>
#include <errno.h>
#include <sys/types.h>
#include <pthread.h>
#include <time.h>


typedef struct wrapper_class{
  int is_alarm_fired;
  timer_t timerid;
}wrapper_class, *wrapper_class_t;


static void something_is_on_fire(wrapper_class_t p)
{
  printf("%s, is_alarm_fired %d\n", __func__, p->is_alarm_fired);
}

static void defuse_the_fire_alarm(wrapper_class_t p)
{
  printf("%s, is_alarm_fired %d\n", __func__, p->is_alarm_fired);
}

static void trigger_alarm(union sigval sigev_val)
{
  wrapper_class_t p = (wrapper_class_t)sigev_val.sival_ptr;
  //TODO: may need mutex to protect object p.
  if (p->is_alarm_fired){
      something_is_on_fire(p);
  }
}

static void start_timer_for_alarm(wrapper_class_t p)
{
  struct sigevent sev;
  memset(&sev, 0, sizeof( struct sigevent));
  sev.sigev_notify = SIGEV_THREAD;
  sev.sigev_notify_function = trigger_alarm;
  sev.sigev_value.sival_ptr = p;
  if (timer_create(CLOCK_REALTIME, &sev, &p->timerid) == -1){
      return;
  }

  printf("timer ID is 0x%x\n", p->timerid);

  struct itimerspec its;
  its.it_value.tv_sec = 10;
  its.it_value.tv_nsec = 0;
  its.it_interval.tv_sec = 10;
  its.it_interval.tv_nsec = 0;
  if (timer_settime(p->timerid, 0, &its, NULL) == -1){
      printf("timer_settime failed, err %s\n", strerror(errno));
      timer_delete(p->timerid); p->timerid = 0;
      return;
  }
}

static void event_handler(wrapper_class_t p)
{
  if (p->is_alarm_fired){
      something_is_on_fire(p);
      start_timer_for_alarm(p);
  }
  else{
      if (p->timerid != 0){
          timer_delete(p->timerid); p->timerid = 0;
      }
      defuse_the_fire_alarm(p);
  }
}

//Do not forget to delete this timer in wrapper class's destroy method!
//for abnormal exit path.
static void wrapper_class_deconstructor(wrapper_class_t p)
{
  if (p->timerid != 0){
      timer_delete(p->timerid); p->timerid = 0;
  }
  free(p);
}

int32_t main(int32_t argc, char** argv)
{
  wrapper_class_t p = (wrapper_class_t)calloc(1, sizeof(wrapper_class));
  if (p == NULL)
      return EXIT_FAILURE;
  
  p->is_alarm_fired = 1;
  event_handler(p);
  sleep(55);
  
  p->is_alarm_fired = 0;
  event_handler(p);
  wrapper_class_deconstructor(p);

  return EXIT_SUCCESS;
}

#ifdef __RUNNING_LOG
shell@android:/data/app # logwrapper ./test_pthread_timer
07-01 03:01:37.248 I/test_pthread_timer( 2945): something_is_on_fire, is_alarm_fired 1
07-01 03:01:37.248 I/test_pthread_timer( 2945): timer ID is 0x80000000
07-01 03:01:47.248 I/test_pthread_timer( 2945): something_is_on_fire, is_alarm_fired 1
07-01 03:01:57.248 I/test_pthread_timer( 2945): something_is_on_fire, is_alarm_fired 1
07-01 03:02:07.248 I/test_pthread_timer( 2945): something_is_on_fire, is_alarm_fired 1
07-01 03:02:17.248 I/test_pthread_timer( 2945): something_is_on_fire, is_alarm_fired 1
07-01 03:02:27.248 I/test_pthread_timer( 2945): something_is_on_fire, is_alarm_fired 1
07-01 03:02:32.248 I/test_pthread_timer( 2945): defuse_the_fire_alarm, is_alarm_fired 0
#endif

3 netlink NETLINK_KOBJECT_UEVENT: same purpose as #1:

(ueventd-example.c) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115

/*
 * Copyright (C) 2013 Yongbing Chen <yongbing.chen.wh@gmail.com>
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

#include <stdlib.h>
#include <errno.h>
#include <sys/socket.h>
#include <linux/netlink.h>
#include <sys/types.h>
#include <unistd.h>

static int32_t init_event_sock(void)
{
    struct sockaddr_nl netlink_addr;
    const int32_t buffersize = 128 * 1024;
    int32_t ret = 0;

    memset(&netlink_addr, 0x00, sizeof(struct sockaddr_nl));
    netlink_addr.nl_family = AF_NETLINK;
    netlink_addr.nl_pid = getpid();
    netlink_addr.nl_groups = 1;

    int32_t event_socket = socket(PF_NETLINK, SOCK_DGRAM, NETLINK_KOBJECT_UEVENT);
    if (event_socket == -1) {
        int32_t err = errno;
        printf("error getting socket: %s", strerror(errno));
        return -err;
    }

    /* set receive buffersize */
    setsockopt(event_socket, SOL_SOCKET, SO_RCVBUFFORCE, &buffersize, sizeof(buffersize));

    ret = bind(event_socket, (struct sockaddr *) &netlink_addr, sizeof(struct sockaddr_nl));
    if (ret < 0) {
        int32_t err = errno;
        printf("bind failed: %s", strerror(errno));
        close(event_socket);
        return -err;
    }

    return event_socket;
}




int32_t main(int32_t argc, char **argv)
{
  char event_sock_buf[1024] = {0};

  int32_t event_socket = init_event_sock();
  if (event_socket < 0){
      return EXIT_FAILURE;
  }

  while (1){
      int32_t len = TEMP_FAILURE_RETRY(recv(event_socket, &event_sock_buf, sizeof(event_sock_buf), 0));
      if (len < 0){
          printf("recv failed: %s", strerror(errno));
          return EXIT_FAILURE;
      }
      int32_t i = 0;
      while (i < len) {
          printf("next event in buf: %s\n",event_sock_buf+i);
          i += strlen(event_sock_buf+i)+1;
      }
  }
  close(event_socket);
  return EXIT_SUCCESS;
}


#ifdef __RUNNING_LOG
shell@android:/data/app # ./test_uevent
[ 4550.363457] usb 1-1: USB disconnect, device number 2
...
next event in buf: remove@/devices/soc.0/f7ed0000.usb/usb1/1-1/1-1:1.2/0003:046D:C52B.0003/hidraw/hidraw0
next event in buf: ACTION=remove
next event in buf: DEVPATH=/devices/soc.0/f7ed0000.usb/usb1/1-1/1-1:1.2/0003:046D:C52B.0003/hidraw/hidraw0
next event in buf: SUBSYSTEM=hidraw
next event in buf: MAJOR=249
next event in buf: MINOR=0
next event in buf: DEVNAME=hidraw0
next event in buf: SEQNUM=878
...

[ 4554.467274] usb 1-1: new full-speed USB device number 3 using berlin-ehci
[ 4554.626594] usb 1-1: New USB device found, idVendor=046d, idProduct=c52b
[ 4554.633604] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 4554.641231] usb 1-1: Product: USB Receiver
[ 4554.645650] usb 1-1: Manufacturer: Logitech
[ 4554.664080] logitech-djreceiver 0003:046D:C52B.0007: hiddev0,hidraw0: USB HID v1.11 Device [Logitech USB Receiver] on usb-f7ed0000.usb-1/input2
[ 4554.686310] input: Logitech Unifying Device. Wireless PID:2012 as /devices/soc.0/f7ed0000.usb/usb1/1-1/1-1:1.2/0003:046D:C52B.0007/input/input3
...
next event in buf: add@/devices/soc.0/f7ed0000.usb/usb1/1-1/1-1:1.2/0003:046D:C52B.0007/hidraw/hidraw0
next event in buf: ACTION=add
next event in buf: DEVPATH=/devices/soc.0/f7ed0000.usb/usb1/1-1/1-1:1.2/0003:046D:C52B.0007/hidraw/hidraw0
next event in buf: SUBSYSTEM=hidraw
next event in buf: MAJOR=249
next event in buf: MINOR=0
next event in buf: DEVNAME=hidraw0
next event in buf: SEQNUM=896
#endif

Bluedroid Bug Fixing

| Comments

I fixed thress tricky bugs in one week, that’s pretty efficient, even out of my own expectation ^-^

bug I: TiVo remote connected but not working.

QA reported a bug that TiVo remote control connected, but no key input captured by our host.

Compared with the normal case (some other HID device), I found below message missed for TiVo case after BT connect:

1
2

05-16 22:13:34.386 D/bt-btif ( 2146): UHID_START from uhid-dev
05-16 22:13:34.386 D/bt-btif ( 2146): UHID_OPEN from uhid-dev

That means the input device for this newly added HID device(TiVo remote) has never been opened by Android EventHub.cpp in failed case.

Then I checked the getevent result when TiVo remote connected, found out that there’s no input device created for it at all.

Tracing from where Bluedroid created a uhid device for a newly added HID device in bta_hh_co_send_hid_info() down to hid kernel code, turned out that the Vendor_id/Product_id for TiVo remote already registered in hid kernel part (point to an existing kernel driver linux/drivers/hid/hid-tivo.c, which is based on BlueZ hidp), so this is an issue of Bluedroid, affecting a group of device which has registered id in kernel as HID_BLUETOOTH_DEVICE.

This kind of devices suppose to be servered by their own driver, so the generic probe method hid_device_probe() will not be triggered when trying to create a uhid device from Bluedroid, thus the input device file will not be created by input_register_device() within this probe.

bug II: Lenovo BT mouse can not pair with our host.

The log shows below error when trying to pair Lenovo BT mouse:

1
2

05-16 21:07:41.451 W/bt-sdp  ( 2146): result :36005A0900000A000100010200013503191200090004350D350619010009000135031900010902053503191002090009350835061912000901000902000901000902010917EF0902020960020902030902450902042801090205090002
05-16 21:07:41.451 W/bt-sdp  ( 2146): SDP - Bad type: 0x02 or len: 4 in attr_rsp

Compared with air log, I found that one byte of the SDP message is different: In air log, I found :0900000A00010001020001, while in dump message in the error message, it’s 0900000A00010001090001

Narrow down this change from driver level to Bluedroid SDP, finally found out it’s a bug in btsnoop_capture(), a simple load/restore bug.

bug III: Can not unpair a remote HID device.

QA found one HID device can not be unpaired: if the device is connected when doing unpair, the device remain connected after unpair, and can not be paired again after unpair.

From the log, I found below error message when trying to re-piar:

1
2

05-20 21:09:19.567  2125  2158 W bt-btif : btif_hh_connect: Device  already added, attr_mask = 0x8005
05-20 21:09:19.567  2125  2158 E bt-btif : btif_hh_connect: Error, device  can only be reconnected from device side

Compared with another HID device which can successfully do unpair/pair, I found that the success case can correctly response to host’s BTA_HH_CTRL_VIRTUAL_CABLE_UNPLUG request as defined in HID spec 1.1, figure A.4.

In the failed case, the HID device will not initiate disconnect against our host after receiving the unplug request, and Bluedroid in our host failed to deal with this situation, as required by HID spec, to delete pair information after issue this requirement. Due to this remaining piece of pair information, next pair attempt will fail.

Dump_stack in Android Native C Code

| Comments

1 Wrapper Android C++ method into a C function:

1
2
3
4
5
6
7
8
9

external/bluetooth/bluedroid$ cat bta/sys/dump_stack.cpp
#include <utils/CallStack.h>
using namespace android;
extern "C" void dump_stack_android(void)
{
        CallStack stack;
        stack.update();
        stack.dump();
}

2 Call this C function from target place:

1
2
3
4
5
6
7

diff --git a/bta/dm/bta_dm_act.c b/bta/dm/bta_dm_act.c
+extern void dump_stack_android(void);
 static void bta_dm_adjust_roles(BOOLEAN delay_role_switch)
 {
+    dump_stack_android();
     if(bta_dm_cb.device_list.count)
     {

3 Add library libutils as dependency in LOCAL_SHARED_LIBRARIES,

1
2
3
4
5
6
7
8

diff --git a/main/Android.mk b/main/Android.mk
 LOCAL_SRC_FILES+= \
 +       ../bta/sys/dump_stack.cpp \
         ../udrv/ulinux/uipc.c

 LOCAL_SHARED_LIBRARIES := \
     libcutils \
+    libutils \

I met a ld error as:

1

error: undefined reference to 'android::CallStack::CallStack()'

It turned out that I added the libutils dependency into a static library libbt-brcm_bta, which can not solve this dynamic symbol at link time

Move the dependency to dynamic library bluetooth.default.so solved the problem.

The final result on running board is:

1
2
3
4
5
6
7
8
9
10
11
12

root@android:/ # logcat -v time |grep CallStack&
05-09 21:01:57.666 D/CallStack( 2133): (null)#00  pc 0005c444  /system/lib/hw/bluetooth.default.so (dump_stack_android+15)
05-09 21:01:57.666 D/CallStack( 2133): (null)#01  pc 0004ddb2  /system/lib/hw/bluetooth.default.so
05-09 21:01:57.666 D/CallStack( 2133): (null)#02  pc 0004c310  /system/lib/hw/bluetooth.default.so (bta_sys_conn_close+27)
05-09 21:01:57.666 D/CallStack( 2133): (null)#03  pc 000577d4  /system/lib/hw/bluetooth.default.so (bta_av_str_closed+115)
05-09 21:01:57.666 D/CallStack( 2133): (null)#04  pc 0004705e  /system/lib/hw/bluetooth.default.so (bta_av_ssm_execute+269)
05-09 21:01:57.666 D/CallStack( 2133): (null)#05  pc 00046f1c  /system/lib/hw/bluetooth.default.so (bta_av_hdl_event+159)
05-09 21:01:57.666 D/CallStack( 2133): (null)#06  pc 0004bf02  /system/lib/hw/bluetooth.default.so (bta_sys_event+49)
05-09 21:01:57.666 D/CallStack( 2133): (null)#07  pc 00074b70  /system/lib/hw/bluetooth.default.so (btu_task+559)
05-09 21:01:57.666 D/CallStack( 2133): (null)#08  pc 00042784  /system/lib/hw/bluetooth.default.so (gki_task_entry+91)
05-09 21:01:57.666 D/CallStack( 2133): (null)#09  pc 0000e3d8  /system/lib/libc.so (__thread_entry+72)
05-09 21:01:57.666 D/CallStack( 2133): (null)#10  pc 0000dac4  /system/lib/libc.so (pthread_create+160)

This result can be verified as the same as addr2line:

1
2

$ arm-eabi-addr2line -e ../../../out/target/product/bg2ct_dmp_emmc/symbols/system/lib/hw/bluetooth.default.so 0004c310
external/bluetooth/bluedroid/bta/./sys/bta_sys_conn.c:236

Update: Peek stack of a running process

Android debuggerd can be used to dump a running process’s stack:

1
2
3
4
5
6
7
8
9
10
11
12
13


ALOGD("peeking stack of process %d\n", pid);
kill(pid, SIGSTOP);
ptrace(PTRACE_ATTACH, pid, 0,0);
char *tombstone_path = engrave_tombstone(pid,
      pid,
      0/*no signal*/,
      true /*dump_sibling_threads*/,
      false /*not quiet*/,
      &detach_failed,
      &total_sleep_time_usec);
ptrace(PTRACE_DETACH, pid, 0, 0);
kill(pid, SIGCONT);

The stack of main thread of the process will shown in logcat, and all others will be in the tombstone file. The target process will resume to execution right after the dump stack finished. This is useful when debugging some real time issues.

Dissect Bluedroid From A2DP: Part v: Key Components

| Comments

1 Interface and implementation

bt_interface_t: Android defined, Bluedroid implemented as bluetoothInterface in external/bluetooth/bluedroid/btif/src/bluetooth.c: System control BT adapter.

btav_interface_t: Android defined, Bluedroid implemented as bt_av_interface in external/bluetooth/bluedroid/btif/src/btif_av.c: System control A2DP service.

audio_hw_device and audio_stream_out: Android defined, Bluedroid implemented in external/bluetooth/bluedroid/audio_a2dp_hw/audio_a2dp_hw.c: AudioFlinger use A2DP client as audio output device.

bt_vendor_interface_t: Bluedroid defined, BRCM implemented as BLUETOOTH_VENDOR_LIB_INTERFACE in device/common/libbt/src/bt_vendor_brcm.c: Bluedroid talk to BT char device driver, internal usage only.

tHCI_IF: Bluedroid defined, Bluedroid implemented as hci_h4_func_table in external/bluetooth/bluedroid/hci/src/hci_h4.c: Bluedroid HCI interface (data/cmd/evt in/out), internal usage only.

bt_hc_interface_t: Bluedroid defined, Bluedroid implemented as bluetoothHCLibInterface in external/bluetooth/bluedroid/hci/src/bt_hci_bdroid.c: Wrapper of tHCI_IF, has bt_hc_worker_thread to serialize downcoming HCI commands and read upcoming data/evt from HCI device.

L2CAP layer API, in external/bluetooth/bluedroid/stack/include/l2c_api.h, internal usage only.

LMP API, in external/bluetooth/bluedroid/stack/include/btm_api.h, internal usage only.

2 Tasks/Roles/Layers

btif_task, managing all messages being passed Android Bluetooth HAL and BTA.

btu_task, the main task of the Bluetooth Upper Layers unit, routing in/out BT cmd/event/data, processing timeout events.

bt_hc_worker_thread, HCI worker thread, all HCI traffic come through this thread.

userial_read_thread, monitoring incoming packets from BT char device driver, transfering these to bt_hc_worker_thread.

btif_media_task, task for A2DP SBC encoder.

uipc_read_task, A2DP server thread, receive audio input data from A2DP client, feed into btif_media_task.

UIPC/A2DP_CTRL_PATH/A2DP_DATA_PATH, socket based IPC, for A2DP client connect/control to A2DP server.

AVDT_CHAN_SIG/AVDT_CHAN_MEDIA/AVDT_CHAN_REPORT, A2DP channels, communicate with remote device.

Serial Finished.

Reference: