tachograph-reader
tachograph-reader copied to clipboard
jugglingcats
Gen 2, Summarize informations
Hello
I want to summarize and consult all information which I need to write update to gen2.
-The structures of bytes for the segments is the same like in gen1. -In the segment is only one change, in new generation type can be also 0x02(content) and 0x03(sygnature). -On the one card might be also types of gen1 and gen2
On the beggining I thought that I can compare length of data "application_Identification" becouse it is exacly 10 byte in gen1(pic.2) and 15 byte in gen2(pic.3) . After select generation, choosen config. But I found information that in the card can be mixed information (gen1 and gen2 segments).
*"segment" mean one block of data like "application_Identification"
*@Dilemma725 wrote a program which to be able to check wchich generation card is. In his code he checking first and second byte of all card byte data. -if first 0x76 and second 0x06 - G1 -if first 0x00 and second 0x02 - G2 Why? First block of data is the same for both generation(pic1)
My question is: Did the best way do update this application to generation 2 is getting each segment one by one and checking his type?
Originally posted by @Phoryn in https://github.com/jugglingcats/tachograph-reader/issues/42#issuecomment-1195802860
Hi,
the drivercards (gen1) I saw until now starts with 0x76 0x01.
If you want to restart this project I think that we have to identity the type of the data first (Drivercard or Tacho then Gen x) and load the appropriate .config
I also make some research on and move it .net Core and make some clean ups.
Hi, I probably need to implement partial support for the version 2 of the specification. Can someone share with me at least a few gen2 .ddd files? I'm having trouble finding them. I promise that I will not publish them anywhere and will use them only for the development, which I might contribute back if it gets somewhere.
@jugglingcats @davispuh @rimutaka @diogocp
I don't work at company that works with Tachographs anymore. Pretty sure they have gen2 files but it's not possible to share them because they contain customer data and GDPR all that.
*@Dilemma725 wrote a program which to be able to check wchich generation card is. In his code he checking first and second byte of all card byte data. -if first 0x76 and second 0x06 - G1 -if first 0x00 and second 0x02 - G2 Why? First block of data is the same for both generation(pic1)
76 01/21, 76 02/22, 76 03/23 etc... are identifiers of data blocks in the ddd file from the tachograph (01 - 1gen, 21 - 2 gen).
In the ddd file from the driver card, it may start with 00 02. This is the EF ICC identifier. This data block is not mandatory, so it doesn't have to be at the beginning of the file. Also, the order of the blocks is not mandatory.
What file is the file you used to get those pictures? We as a company are also busy trying to decipher gen 2, and when I figure it out I plan on implementing it on the Jugglingcat lib so nobody ever has to do this work again hopefully for the coming years.
@OscarKro the images from @Phoryn will be from the specification. I uploaded the gen1 spec to this repo. If someone can link me to the gen2 spec am happy to upload it too. I'm not working on this project any more but happy to facilitate and can add collaborators if/when appropriate!
Yeah but thats the problem. I cant find a similar spec like on the first generation anywhere. I used ECE/TRANS/SC.1/2006 for the first generation, and then appendix VII gives a pretty clear documentation on what the file looks like when downloaded. But there isnt a similar document to be found for the second generation as far as i can tell
Here is sample data. Be advised that the certificates used are test certificates, so they will not properly validate. The dataset contains driver cards and transfers from a vehicle unit. None of the data contain real persons, so there are no privacy concerns. SampleData.zip
Wow thanks man! great help. To give some context on what i'm doing: I'm currently working for a small company where we create devices to remote download the tachographs in trucks. I've been asked to check how we can validate the downloaded files. We were using the @jugglingcats lib for testing and finding out how it all comes together. It worked great for GEN 1. Now i've been asked to look into GEN 2. For research purposes i'm now creating some simple python code to pull the data from the gen 1 and 2 files, gen 1 works, but now still gen 2. When that is done, I need to find out how to validate it all, but thats the real part I can't make heads or tails of. The getting data from the GEN 2 file is already succeeding bit by bit. I find the record arrays actually pretty handy dandy.
Well, you can run through all of the data structures and verify plausibility, to one degree or another, but generally it is enough to verify the signatures to have full confidence in the download not having been corrupted or tampered with.
Yeah, in the Netherlands we say "je slaat de spijker op zijn kop". Which roughly translates to "you hit the nail on its head with your comment". Verifying the signatures is exactly the part I don't understand. But I'll get to it someday, and when I do I want to update the @jugglingcats library so nobody ever has to do this sheit again.
Yeah but thats the problem. I cant find a similar spec like on the first generation anywhere. I used ECE/TRANS/SC.1/2006 for the first generation, and then appendix VII gives a pretty clear documentation on what the file looks like when downloaded. But there isnt a similar document to be found for the second generation as far as i can tell
See #42, there I linked latest doc at the time - https://eur-lex.europa.eu/eli/reg_impl/2016/799/2018-04-17
It's not exactly that readable but info is there you just need decypher it in your mind :joy: back then I went over it many times and got pretty good understanding but it was many many years ago.
Looks like this https://eur-lex.europa.eu/eli/reg_impl/2016/799/2023-08-21 (interestingly date in future :D) is the most newest one now.
@mpi-wl
Is there any chance that you will share some information how to read gen2 files? I mean, I can read Gen1 files based on that information.
But when I'm trying to read a gen 2 files (that ones which starts with 76 21) I cannot find any data.
@mpi-wl Is there any chance that you will share some information how to read gen2 files? I mean, I can read Gen1 files based on that information. But when I'm trying to read a gen 2 files (that ones which starts with 76 21) I cannot find any data.
0x7621 is "Transfer data overview structure generation 2, version 1 (TREP 21 Hex)" - here is a direct link to the information: CELEX:02016R0799-20230821 (page 343 if it doesn't open on its own)
It will look something like this:
Hi @graealex, thanks for your response.
I must have missed this piece in the documentation, which isn't difficult since it is 600 pages long lol. I don't know if I understand this correctly, so there are not only Gen1 and Gen2 files but Gen1, Gen2V1, and Gen2V2?
My problem is that I can't read Gen 2 files manually. I have a couple of files, for example, gen 1 which starts with 0x00 and 0x02. This type of file seems quite understandable to me. When I wanted to find DriverCardApplicationIdentification, i just take a look on documentation. It says that Application_Identifications starts at 0501. So I open that file in some hexViewer and read that manually like here.
But when I've opened the second file which starts with 0x76 and 0x21 nothing make sense to me. In documentation DriverCardApplicationIdentification also starts at 0501 but with SFID 1. There is information that SFID is given in decimal, so I've transferred it to the hex, and cannot find many bytes 0501 but without that SFID or any other informations which make sense.
Am I doing something wrong? On what should I focus on?
Files with sections starting with 76 01 or 21, 76 02 or 22, 76 03 or 23... etc. are tachograph readings. These files have a different structure than the driver card. 05 01 is the Application_Identifications section in the driver card.
As @mpi-wl correctly wrote, 0x7621 indicates that this is a download directly from a VU (vehicle unit - the actual device memory). All these files come as .DDD files, but can be just a card or from a VU.
And yes, there is Gen 1, Gen 2 V1 and Gen 2 V2. And for VU downloads, the format has actually changed quite a bit between Gen 1 and Gen 2, notably because every block is now a record array, and never just the block itself.
I have written all the code for the company I work for which parses all three gens into usefull data objects so we can just extract everything. I happen to know a little bit about it now, but I don't really understand your question? Yes there are multiple generations, mainly gen 1 and gen 2, and a latter gen 2.1 where gen 2.1 mostly just adds GNSS data to different fields. Could you summarize what your question is?
I'm ashamed to admit but I was sure that the files with the extension .ddd are only files from the driver's card.
This completely changes the form of things, because I tried to read files that began with 0x76 and 0x21 in the same way as the driver's card.
For a few days, I'm trying to understand the structure of these files, file types, manual reading, etc. Thank guys, this information is very helpful for me. Trying to build a parser, like many people before me, from what I can see in this repo.
@Camillus83 There is no shame in not knowing that, because it's not like any of the tachograph stuff is particularly well documented. Generally there is a download protocol ("Appendix 7 DATA DOWNLOADING PROTOCOLS" in the EU spec) where a device requests blocks from the VU to download, and one such answer is the 0x7621 "Transfer data overview structure generation 2, version 1 (TREP 21 Hex)" block. These answer blocks then get written into a DDD file.
Ah okay, No, as said before, Both the driver card or the tachograph mass memory can be downloaded via the tachopgrahs interface. Both are called 'ddd' files. But they are completely different. The Driver card looks exactly like the structure described in the documents. And is basically just all card data in a specified order. LIke any card file system. Gen 1 memory files are very different from Gen 2. Where gen 1 is just bytes in a specified order and gen 2 consists of so called 'record arrays' where each record array first specifies the type, length and amount of fields giving you the chance to see the length of a section before actually looking at the data. The data then is about the same as it is in Gen 1. But there are some differents between all data in gen 1 and gen 2, so be mindfull of that. Gen 2 mostly has more data for specific fields then gen1. Gen 2.1 is about the same as gen 2 except for added GNSS data. If you need more detailed info I can help.
Someday I will upload all my code giving everyone an easy to use library to parse gen 1, 2 and 2.1 but for now im under NDA. I have to rebuild it in a different way to do this, but yeah. Time and all.
@graealex @OscarKro
As I said before, thank you for the explanation. Now I am able to read the files.
It seems that the generation changes and the rebuilding of the structure made a lot of mess in the existing solutions and in the creation of a compatibility solution with all generations. Going back to work on it, most likely I will have more questions later. So don't be surprised by notifications 😸
Hi Guys, it's again me.
After couple of weeks I've finally understood and built parser for Driver Cards. Now I started working on VU files, and again have some problems and misunderstandings. I've started with VuOverview Gen1 and everything was clear and fine.
First main issue was when I was trying to read VuActivities block from VU Gen1 file. Reading the documentation we can deduce that VuActivities Block is built by
- TimeReal (4 bytes)
- OdometerValueMidnight (3 bytes)
- VuCardIWData (x bytes)
- etc.
Block structure from my file looks like: 76 02 63 D1 C2 7F 05 7F 45 00 01 XX XX XX Where:
- 7602 is block ID
- 63 D1 C2 7F is TimeReal
- 05 7F 45 is OdometerValueMidnight
- 00 01 is noOfIWRecords
But when I've opened that file in HexEditor I saw that there are many more blocks which looks exactly the same like VuCardIWRecord and have the same prefix as 00 01.
It seems to me that I don't understand or wrongly reads file, because later other blocks after parsing have meaningless data.
well I would have to look it up in the code and documentation again, its been a while for me. But vehicle unit card iw record stands for "insertion and withdrawal" record if i remember correctly. And those can be multiple. hense the number of iw records uint16_t there.
You cant deduce the data of gen 2 files by just looking at the hex. You will see that all data records look more or less the same, as the whole idea of gen 2 and up files are made using the same first few bytes, indicating the type, length of data and amount of data blocks. Its only after the first few bytes that it is different. So if you just look at the hex and say 'hey, i see a lot the same', yeah that's the idea.
@OscarKro
According to the first part of message, the data blocks are built with noOfRecords, then recordsData(noOfRecords*recordData). That's why I'm surprised that noOfRecords says there is only 1 record like this, but on hexEditor I can see that next is almost identical structure, probably only values of TimeReal bytes are different.
According to the second part of your message, in DriverCard i've recognized generations based on the size etc. But In VU it's probably kind easier. 76 01 - Activites Gen1 76 21 - Activities Gen 2 etc.
Can you share the data you are trying to parse, or is it a privacy concern?