Bluetooth synchonisation issue - firmware versus mobile App - workaround

boubourunner
boubourunner Registered Users Posts: 13
Master Traveler
edited January 24 in TomTom Sports
Hey,

I was expected a new firmware to get my wach sync properly.
(1.7.60, 1.7.62, 1.7.64 , none of them was able to sync QuickGPSFix ... only activity upload works).

I know now that in my cases watch firmware was not the issue. (for this particular case)

go there and read this: sync issue workaround

(the other topic has been moved to Sports [Connect][Apps][Website] but more often people with sync issue comes here to request a firmware update)

Stephane

Comments

  • tfarabaugh
    tfarabaugh Posts: 16,936
    Superuser
    You should be able to fix the QuickFix issue for your watch by following these steps (make sure you have installed the latest version of Connect if prompted):

    1) Plug the watch into the computer and in MySports Connect, click the Gear Icon
    2) Press CTRL (Windows) or CMD (MAC) on your keyboard while you hover your mouse over the reset button.
    3) You'll see the reset button turns into a "Turn Off" button.
    4) Click it, disconnect the watch and the screen will go dead.
    5) Reconnect the watch and it will wake up and quick fix should now update as usual.

    I hope this helped answer your question. If so, please mark it as a solution so others can look for it if they have the same question.
  • boubourunner
    boubourunner Registered Users Posts: 13
    Master Traveler
    Thanks to answser but it seems you didn't follow the link.
    (I already did this procedure on following firmware: 1.7.60, 1.7.62, 1.7.64)

    Look at that (answer are inside):
    Debug my synchronisation issue: story_link
  • tfarabaugh
    tfarabaugh Posts: 16,936
    Superuser
    Thanks to answser but it seems you didn't follow the link.
    (I already did this procedure on following firmware: 1.7.60, 1.7.62, 1.7.64)

    Look at that (answer are inside):
    Debug my synchronisation issue: story_link
    I actually did not click your link intentionally as you are not supposed to link to outside websites on the forums. I am not interested in being spammed (not saying yours is but there is no way of knowing that in advance).
  • boubourunner
    boubourunner Registered Users Posts: 13
    Master Traveler
    Let start to understand what's wrong

    I manage to root my phone (by the way install the lineage os ROM that is great).
    After activating developper menu it's possible to log HCI for bluetooth.

    wireshark is a great help to decode btsnoop. really !

    As explained here (https://github.com/dlenski/ttblue/blob/master/tt_bluetooth.md), file write to TomTom watch are done
    by writing 20 bytes by 20 bytes on TRANSFERT characteristic (handle=0x78 for Runner 3).

    from wireshark I can see:

    ::

    Bluetooth Attribute Protocol
    Opcode: Write Command (0x52)
    Handle: 0x0078 (Unknown)
    Value: 6f0000001f1b574800000800cf9e910208956e00

    Bluetooth Attribute Protocol
    Opcode: Write Command (0x52)
    Handle: 0x0078 (Unknown)
    Value: 6f0000001f1b574800000800cf9e910208956e00


    And what I discover (and paste here) is that when the TomTom App goes wrong it may send many times the same datas to the watch: that's not right !
    The watch detect the anomaly thanks to the CRC16 checksum check.
    And the watch informs the error with:

    ::

    Bluetooth Attribute Protocol
    Opcode: Handle Value Notification (0x1b)
    Handle: 0x007b (Unknown: Unknown)
    Value: ffffffff

    The Value for this characteristic should be a counter starting by 0 and increasing by one every 256*20 bytes ! (and ffffffff indicates an error)
    After this the watch Disconnect !

    It appears that this error is caused by the android MySport application itself and not the watch firmware.

    Let's look at the application logs on my rooted phone and logcat:

    ::

    D MessageProxy: Sending message:WriteCharacteristicMessage with uuid:170d0d33-4213-11e3-aa6e-0800200c9a66
    W System.err: java.lang.Exception: Attempting to send a message before the current message is complete.WriteCharacteristicMessage with uuid:170d0d33-4213-11e3-aa6e-0800200c9a66
    W System.err: at com.tomtom.ble.service.messaging.MessageProxy.sendMessage(MessageProxy.java:37)
    W System.err: at com.tomtom.ble.BleDevice.writeCharacteristic(BleDevice.java:440)
    W System.err: at com.tomtom.ble.device.WatchDevice.writeCharacteristic(WatchDevice.java:2024)
    W System.err: at com.tomtom.ble.BleService.writeToCharacteristic(BleService.java:89)
    W System.err: at com.tomtom.ble.service.AbstractFileTransferGattService.sendNextPacketAfterDelay(AbstractFileTransferGattService.java:1026)
    W System.err: at com.tomtom.ble.service.AbstractFileTransferGattService.access$2900(AbstractFileTransferGattService.java:48)
    W System.err: at com.tomtom.ble.service.AbstractFileTransferGattService$11.run(AbstractFileTransferGattService.java:942)
    W System.err: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:428)
    W System.err: at java.util.concurrent.FutureTask.run(FutureTask.java:237)
    W System.err: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
    W System.err: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
    W System.err: at java.lang.Thread.run(Thread.java:761)

    The Exception speaks by itself: "Attempting to send a message before the current message is complete.WriteCharacteristicMessage"
    From this it seems that TomTom App Mysport does not use well the android callback onCharacteristicWrite.
    It seems that this is the root cause of the sync error (if yes it should be easy to fix)
    Why it's like this ? A poor design of the application ? An attempt to speed up the communication even if it breaks stability ?

    In order to understand more I decide to write my own application as TomTom doesn't provide the source code of its application.
  • boubourunner
    boubourunner Registered Users Posts: 13
    Master Traveler
    The Story - Why I develop my own Sync Application
    =================================================

    The good about the TomTom runner watch:
    - Personnalised workout: great, acts like a personnal coach.
    - Trail navigation feature: great to not get lost and discover new trails ! impressive feature ! (usualy only available on very expensive watch)
    - Swimming pool count turn: not tried it yet but great feature
    - Great screen visibility, even better on sun. Impressive ! (I prefer this over a color screen)
    - Joystic navigation: well done menu system
    - Great usability overall: simple and feature full. Many features without complexity.

    **BUT**, my TomTom Runner 3 won't sync properly over bluetooth from the tomTom MySport android App. (at least on my mobile phone)
    I was not able to get the right explaination from TomTom support so I decide to understand by myself what was going on.

    First Big thanks for previous effort on following project (I would never start this whithout this help!):
    - ttblue (sync over bluetooth from a computer) : https://github.com/dlenski/ttblue
    - ttwatch (sync over usb and process ttbin): https://github.com/ryanbinns/ttwatch
    - TomTomWatch java desktop application: https://github.com/scubajorgen/TomTomWatch
    - wireshark
    - android HCI btsnoop and logcat

    And best of it:
    - ttblue with bluetooth protocol reserse engeniering: https://github.com/dlenski/ttblue/blob/master/tt_bluetooth.md => great Job ! Thanks !

    tt_bluetooth.md contains all the informations I needed to start a Proof Of Concept for an android application.
    ttblue provide even example for a bluetooth implementation of a desktop tool.




    Lets start a Proof of Concept: sync QuickGPS between my phone and my runner 3 watch

    ttblue and ttwatch projects gives me confidence about feasibility without spending too much time.

    My choice is done I will write a python android application thanks to "kivy" and "able".

    Python is great to develop fast and it still give direct access to android SDK. (I felt better at this step to be able to access to android Apis without abstraction that might be buggy)
    "able" should give me a very low abstraction to native andoid bluetooth API. (https://github.com/b3b/able)

    This choice may seems strange for an android developer but I had no experience for that and feel at home with python.

    step 1) discover device, connect gatt and discovers gatt services. (able example did that):
    """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

    "able" project comes with an example https://github.com/b3b/able/tree/master/examples/alert
    This example seems a very good starting point for my projet (to be honest that's why I start from here).

    After a very few modifications I could get :

    ::

    : ---- characteristic: 00002a01-0000-1000-8000-00805f9b34fb --- handle: 0x18 --- permissions: 0 --- properies: 2 --- WriteType: 2
    : ---- characteristic: 00002a00-0000-1000-8000-00805f9b34fb --- handle: 0x16 --- permissions: 0 --- properies: 2 --- WriteType: 2
    : service: d735f430-486d-11e4-916c-0800200c9a66
    : ---- characteristic: d735f432-486d-11e4-916c-0800200c9a66 --- handle: 0x95 --- permissions: 0 --- properies: 26 --- WriteType: 2
    : ---- characteristic: d735f431-486d-11e4-916c-0800200c9a66 --- handle: 0x92 --- permissions: 0 --- properies: 26 --- WriteType: 2
    : service: b993bf91-81e1-11e4-b4a9-0800200c9a66
    : ---- characteristic: 47ec27b0-5c56-11e5-a837-0800200c9a66 --- handle: 0x87 --- permissions: 0 --- properies: 22 --- WriteType: 1
    : ---- characteristic: b993bf93-81e1-11e4-b4a9-0800200c9a66 --- handle: 0x85 --- permissions: 0 --- properies: 12 --- WriteType: 1
    : ---- characteristic: b993bf92-81e1-11e4-b4a9-0800200c9a66 --- handle: 0x82 --- permissions: 0 --- properies: 28 --- WriteType: 1
    : ---- characteristic: c6945cb0-4ab4-11e7-9598-0800200c9a66 --- handle: 0x8a --- permissions: 0 --- properies: 2 --- WriteType: 2
    : service: b993bf90-81e1-11e4-b4a9-0800200c9a66
    : ---- characteristic: 170d0d34-4213-11e3-aa6e-0800200c9a66 --- handle: 0x7b --- permissions: 0 --- properies: 22 --- WriteType: 1
    : ---- characteristic: 170d0d31-4213-11e3-aa6e-0800200c9a66 --- handle: 0x72 --- permissions: 0 --- properies: 26 --- WriteType: 2
    : ---- characteristic: 170d0d33-4213-11e3-aa6e-0800200c9a66 --- handle: 0x78 --- permissions: 0 --- properies: 22 --- WriteType: 1
    : ---- characteristic: 170d0d32-4213-11e3-aa6e-0800200c9a66 --- handle: 0x75 --- permissions: 0 --- properies: 22 --- WriteType: 1
    : service: 00001801-0000-1000-8000-00805f9b34fb
    : service: 0000180a-0000-1000-8000-00805f9b34fb
    : ---- characteristic: 00002a24-0000-1000-8000-00805f9b34fb --- handle: 0x44 --- permissions: 0 --- properies: 2 --- WriteType: 2
    : ---- characteristic: 00002a26-0000-1000-8000-00805f9b34fb --- handle: 0x48 --- permissions: 0 --- properies: 2 --- WriteType: 2
    : ---- characteristic: 00002a23-0000-1000-8000-00805f9b34fb --- handle: 0x42 --- permissions: 0 --- properies: 2 --- WriteType: 2
    : ---- characteristic: 00002a2a-0000-1000-8000-00805f9b34fb --- handle: 0x50 --- permissions: 0 --- properies: 2 --- WriteType: 2
    : ---- characteristic: 00002a28-0000-1000-8000-00805f9b34fb --- handle: 0x4c --- permissions: 0 --- properies: 2 --- WriteType: 2
    : ---- characteristic: 00002a27-0000-1000-8000-00805f9b34fb --- handle: 0x4a --- permissions: 0 --- properies: 2 --- WriteType: 2
    : ---- characteristic: 00002a25-0000-1000-8000-00805f9b34fb --- handle: 0x46 --- permissions: 0 --- properies: 2 --- WriteType: 2
    : ---- characteristic: 00002a29-0000-1000-8000-00805f9b34fb --- handle: 0x4e --- permissions: 0 --- properies: 2 --- WriteType: 2
    : service: 0000180f-0000-1000-8000-00805f9b34fb
    : ---- characteristic: 00002a19-0000-1000-8000-00805f9b34fb --- handle: 0x62 --- permissions: 0 --- properies: 18 --- WriteType: 2



    step 2) add hidden services
    """""""""""""""""""""""""""

    TomTom watch doesn't declare publicly all it's services.
    After android discoverServices results, they are some essential services are missing.

    I spent some times to find how to manually add characteristics and finally Y found a "Hiden" android API (by the way, it's good to known that android documentation hides some APIs):

    The following BluetoothGattCharacteristic constructor enable to give the "handle":
    ::

    BluetoothGattCharacteristic(uuid_obj, handle, properties, permissions)

    Then add the characteristic to a service (pick one):
    ::

    my_service.addCharacteristic(new_characteristic)

    Great I can now communicate to all needed handle !


    step 3) crc16
    """""""""""""

    In order to read and write file to the watch it's necessary to compute CRC16 MODBUS checksum.

    After some research I found all the necessary information here:

    https://www.digi.com/resources/documentation/digidocs/90001537/references/r_python_crc16_modbus.htm

    That's great, thanks digi to provide a python sample code that do it.

    After some unit test I'm confident that my python application can manage CRC16 MODBUS checksum.

    step 4) tomtom watch authentication
    """""""""""""""""""""""""""""""""""

    I used the same password as in the official App.
    i follow the this step and the watch acquit as expected:
    ::

    83 < - 01 00
    85 < -- [magic bytes (8)]
    73 < - 01 00
    82 < -- [6-digit code as 32-bit LE integer]
    82 - > 01 (if successful)

    01 received on 0x82 handle means SUCCESS ! sounds good.

    step 5) download and write QuickGPSFix
    """"""""""""""""""""""""""""""""""""""

    Yes the write file operation start :) and without errors until it's DISCONNECT :((((
    The watch DISCONNECT for unkonwn reason and the transfer is only about 20% .....

    hard time: I'm close but it doesn't work !

    step 6) 30s DISCONNECT
    """"""""""""""""""""""

    What a pity, my implementation always fails with a DISCONNECT but I can see that this is not the same pb as the TomTom official app.
    In my case the watch DISCONNECT 30s after the GATT CONNECT (in case of Tomtom App, it was a bad checksum check from the watch).
    But why ? I have no experience with bluetooth LE device and I had no idea about what was going on and why I could not get more than 30s of connection ?

    After trying many things (reverse engineer more communication with the official app).
    I discover that if I create "bond" then the connection will not ends after 30s.
    But it was still not alaways working ... After many tries I understood and found that the initial state should always be BOND_NONE.
    I the initial state is BOND_NONE and i createBond then the connection will not end after this 30s timeout.

    at the end, my app do that:
    ::

    - ensure state is BOND_NONE (removeBond if needed, removeBond is again an hidden API from android SDK)
    - scan for watch
    - createBond and wait state BOND_BONDED
    - discoverServices
    - manually add missing services
    - start watch communication

    THIS TIMES it WORKS !!!! The POC is over my phone can work with the TomTom runner 3 watch !

    So TomTom support, I have now my anwser:
    The incompatibility between my phone and the watch is not a hardware problem nor a watch firmware problem.
    The incompatibility seems a poor design for bluetooth ble communication within the TomTom Mysport App.
    What a pity, because the rest of the application and the watch is great.

    For me what an happy end, I know that I will be able to use my watch the way I was wishing for.

    step 7) clean up and write story
    """"""""""""""""""""""""""""""""

    Why not do a little more work to make my POC useful for me and may be other:
    I start removing all the hardoded things from my code (passord, device_id ...).
    Then I add some option to the app to make it usable for someone else.

    step 8) add upload ttbin
    """"""""""""""""""""""""

    At this step my POC could be over but I decide to add a feature: read activities ttbin from the watch.
    (not a huge effrort as I had already read and write file capabilities.)
    As it is just a POC, I decide to upload without deleting the activity (In the short term I will continue to use the official app for that).

    step 9) I'm in good mood, do a little more
    """"""""""""""""""""""""""""""""""""""""""

    - Add a pairing feature to the App (enable to generate and use a new password. No more need to get the password from the offitial App).
    - Add an option to delete file after upload to make upload a real fetaure
    - Add an Help View that describes how to use the ttdroid App !!!

    This time I don't want to add more features as it is just a POC. The last modification I plan to do are:

    - correct some typos on text
    - add a popup for two issues that happens in rare cases:
    - error: no way to call "discoverServices"
    - error: in Get ttbin the watch send an inexpected data on 0x82 handle

    .. NOTE:: EDIT: should be fixed see step 10)


    step 10) Fix encoutered stability pb
    """"""""""""""""""""""""""""""""""""

    3 fixes:
    - close_gatt was missing after "Get ttbin upload track". (this causes instability)
    - removeBond at the end: cleaner and avoid to do it on next startup
    - cancelDiscovery on init: they was conflict in case android was doing a service discovery.


    *After those fixes the application is very stable for me. (tested only on my phone)*

    Next
    ----

    Not sure ?
    It will also depends how the TomTom community react ?
    It depends also if TomTom company want's some kind of partnership with me.

    As I want to learn react (and may be react-native) I may start from scratch a new application that could:
    - sync QuickGPS
    - upload activity
    - download trails (need to reserve enginneer the format)
    - dowload personnalised workout (need to reserve enginneer the format)
    - update watch time
    - upload strava
    - upload runalyse
    - ???
  • djlegge
    djlegge Registered Users Posts: 14
    Apprentice Traveler
    Nice work @boubourunner !
    I often suspected the problems were not just with the watch firmware because frequently I can encourage a successful sync by force-quitting the Android app and restarting it.

    I have read your story and downloaded the app - I hope to find time to have a play with it myself. The TT app is pretty good when it syncs, but it's always good to have alternatives - especially open source ones. This could especially be useful if TT stop supporting the watch in a few years and turn their servers off.

    Please keep us up to date here - posting links to external websites is fine ! Others do this all the time :relaxed:

    I know it's just a proof of concept and a bit of fun but I'd be interested to see the source code if you could find time to put it on github ?
  • boubourunner
    boubourunner Registered Users Posts: 13
    Master Traveler
    @djlegge thanks,

    I agree with you, it could be nice to have an open source alternative.
    I'll share my work when it will be ready. I'll let you know and don't want to keep it just for my personal usage.