FT8 decoding #3
Comments
|
Hi Pedro LU7DID, Make sure you have well aligned the WAV files with the 15 second time mark. The decoder searches through time offsets within about a second, but still you have to be quite exact on the 15 second frame. The audio length should be 15 seconds. The decoder should work with any sample rate that is integer multiple of 6.25 Hz, but I have only tested 12000 Hz and 6400 Hz. Do your own WAV files decode with WSJT-X (there's an option File->Open to decode from WAV)? I have been thinking of putting up some test files indeed that I have recorded from WebSDR. I thought perhaps it's not nice to put up information containing callsigns etc, but I guess it's public information anyway, so I will do that soon. |
|
Thank you! The files I'd tried to use are the ones posted along with the ft8basic package, which when used with that package decodes fairly well. I didn't tested them with WSJT-X program no used samples from it with your program, but few samples that works ok for you would be a nice "smoke test" to ensure that everything is installed and set properly with your package. Thanks! 73 de Pedro, LU7DID |
|
If you mean https://github.com/rtmrtmrtmrtm/basicft8, then the samples there are still version 1 of the FT8 protocol, which is already phased out (it was used in WSJT-X versions prior to 2.0.0). The basicft8 repository has not been updated to the latest version of FT8. However, author's other repository https://github.com/rtmrtmrtmrtm/weakmon has been updated, though it does not contain any samples. decode_ft8 does not support FT8 protocol version 1. There are routines however in the code for the old protocol if anyone is curious. |
|
I have added some test WAV files recorded from WebSDR in Europe. You can also find the messages as decoded by WSJT-X in a corresponding TXT file. I will later add a test utility that can automatically score the results against WSJT-X. So far it's decoding about 70-80% of what WSJT-X can decode, and even some occasional messages that WSJT-X misses. |
|
Hi, You're right! The files points to the "old" FT8 format. TU to add your own set, it would be very helpful. If I'm not mistaken what is left in your decoder to enable it as a receiver is to allow input thru stdin and to have some way to sync itself with the FT8 15 secs windows, right? Having these two plus some implementation of the rtl-sdr receiver, plus some data manipulation with csdr plus ntp running plus and you should have a fully operational FT8 receiving chain. 73 de Pedro, LU7DID |
|
Hi, I'm getting segment violation when executing decode_ft8 with any of the sample .wav files, both from wsjtx or your test suit; cloned and rebuild the package and still does that. |
|
That's strange indeed. What OS do you have? I have checked on Mac OSX, Ubuntu 14.04 and Mint 18.3, and I have no errors. |
|
My mistake sudo ./decode_ft8 ./tests/websdr_test13.wav 73 de Pedro, LU7DID |
|
Hi, Following on previous posts I would like to share that I'd processed some few thousand .wav files from the repository left by the wsjt-x program in my main station (FT2000+4 El Yagui+PC Intel i7 4x4 GHz+) Pending to complete a more rigorous analysis of the results I can say that the ratio of successful decoding by the decode_ft8 program is close to the 50% of the cases; which I found remarkable given the vast computing power difference between the machine where wsjt-x runs and the raspberry pi Model 3 where the evaluation where performed. Computing time averages 1.2 secs per file; however when excluding files where no decoding where made the number goes up to close to 2 secs per file, again on a raspberry pi environment. I found, as said, both results good and remarkable. I believe that if the development roadmap includes to feed from live audio for a true receiver somewhere in the near future some optimization in the processing time needs to be made as the average time taken exceeds the 1.52 secs (theoretical, best case scenario) before the new window starts. However a quick overview of the code shows several spots where optimizations can be made with reasonably small effort. Really very good implementation work so far. 73 de Pedro, LU7DID |
|
Hi Pedro, thank you for the excellent report!
|
|
Also, note that the decode time can be adjusted by changing the number of candidates (kMax_candidates) and LDPC iterations (kLDPC_iterations). Perhaps they can even be lowered to speed things up without losing quality. It's subject to investigation, and your data is actually very helpful, so if you're interested and have time for that, I would be happy to learn what you can observe by changing the decode parameters. |
|
Sure do these parameters would influence the data. Also, if I'm not mistaken an optimization is pending to sort the candidates by order or likelihood. I didn´t observed in the code a hard time limit, which would be very handy. Although evaluations can be made out of .wav files nothing is better than real signals, so the ability to get a live stream would be very beneficial. I'd forked the package but didn't attempt any modification as I'm not sure what part of the code are you working with or plan to work with. |
Interested in your package, eager to contribute!
I ran this, but with far fewer cases. However, wsjt-x in the RBPI saves only files with some decoding, so it's easy to count the number of hits (actual files where something was decoded/total number of files).
I think light is good as a first approach, but as mentioned in the previous response, process only .wav files limits the usefulness of the package; externalizing some of the processing chain would be a good option IMHO so perhaps just receiving the stream of S15-LE mono thru standard input should be a very flexible approach.
Yes
Assumed that possibility would be an outlier (2x 3x std dev off) so didn't actually counted them. 73 de Pedro, LU7DID |
i think i am getting there. At least the encode is done. But to realise that, i had to use one of my video buffers(about 1.7 MB) in external SDRAM. Haven’t tried decode or real time decode/yet(very little free time atm). |
|
Dear Karliss, I am happy to report that after a lot of gnashing of teeth I have gotten your decode software to work on an STM32f769 Disco board.
Thanks for sharing your work. Charley, W5BAA |
|
Hi Charley, Did you made it work right out of a receiver audio source or processing .WAV files? 73 de Pedro, LU7DID |
|
Pedro, I am interfacing the STM32F769 board with an SDR Tranceiver (RS-HFIQ) at baseband IQ level. Regards, Charley |
|
Hi guys, first of all amazing work W5BAA! Second, this makes me feel lazy, although I have just been busy with other things. |
|
Karlis, Good to hear from you and thanks for the kind words. I have been doing SDR projects on STM32 processors for several years now. I was the chief architect for the STM32_SDR project which you may find on Github. I have had success in porting the STM32_SDR code over to the STM32F746 Disco board. When I started the FT8 work I started out on the STM32F746 but I soon ran into RAM limits so I changed over to the STM32F769 Disco board which has 512 kb of RAM. At present my code is not ready for sharing since to do so will invite a lot of questions and suggestions for improvement from the armchair experts. I do intend to share the code once I have got it working if a fairly robust manner. My intent is to have a self contained FT8 only SDR tranceiver that does SIMPLE FT8 Beacon plus basic searching for and answering another station's CQ. I was able to convert your code to simple C and to break up the processing where I do an extract of the power spectrum every 160 msec rather than try to process 15 seconds of time series in one gulp. The audio sample rate is 8000 samples per second and the basic FFT size is 2560. For time synchronization I have included a UBX GPS receiver. Now that I have the decoding working I an now getting your FT8 encoder code to play. I think you have done a great job of getting a good FT8 decoding suite. I am glad we have made contact. Perhaps as I get stuff to work I can ask a few questions as I get more insight on the FT8 protocol. In particular I think the addition of an Automatic Gain Control on receive may help the decoding but it may require some concentrated thought. Regards, Charley |
|
Hi Charley & Karlis, I'm struggling to reduce the platform to operate FT8 down to a Raspberry Pi Zero. It's way too small to run WSJT-X. I'm able to successfully run PiFT8 as a beacon and a very cumbersome way to run contacts, also I'm able to use a direct conversion receiver to successfully receive using WSJT-X in another PC. FT8_lib working out of a receiver would fit that part of the project. I do think there are multiple opportunities to work cooperatively even way before the project is mature enough to be presentable. 73 de Pedro, LU7DID |
|
Karlis and Pedro, I have has some success in getting the encode stuff to work. I first tried using the statement of int rc = pack77(message, packed); I was not able to get decode using my Rapberry Pi box. But, when I used the statement packtext77(message, packed); I was able to get decodes on the Paspberry Pi. So, then I tried my SDR rig on an antenna and actually had someone replay to my CQ. I am pleased. Regards, Charley |
|
Karlis and Pedro, I continue to make progress with my FT8 project on the STM32f769 platform. Here is a link to a video that I made today, https://www.dropbox.com/preview/Photos/FT8_SDR/W5BAA%20FT8%20SDR%20Tranceiver.mp4?role=personal I also placed the video on the SoftRock40 io group files folder. https://groups.io/g/softrock40/files/W5BAA%20FT8%20SDR%20STM32f769 Regards, Charley |
|
Pedro, I apologize for being so late in replying. You comment email ended up in my spam folder. The STM769 board is processing the baseband IQ signals from an SDR rig in real time. I am quite pleased with the FT8 decoding. I now have the unit capturing the messages of stations that reply to my CQ. I am logging the messages on an SD card installed on the STM769 board. Here is an extract from stations who called me this morning. I continue to make progress. W5BAA F8DZU -14 W5BAA KF8G EN91 W5BAA KF8G EN91 W5BAA KF8G EN91 W5BAA KF8G EN91 W5BAA KG4ORQ EL97 W5BAA KG4ORQ EL97 W5BAA KG4ORQ EL97 W5BAA KG4ORQ EL97 W5BAA VA3EKG EN82 W5BAA VA3EKG EN82 |
|
Great progress, Charley! It also inspires me to keep updating my code. I'm interested to cooperate - I might be able to help with some ideas of improvements. Also curious how performant is the decoder at the moment - what settings do you use and how long does it take to decode? There's a tradeoff between accuracy (number of decodes) and time spent, of course. Also few of the advanced techniques used by the original WSJT-X are missing in my code, but on the other hand, I added a very cruical DSP trick that improved accuracy. |
|
Karlis,
Good to hear from you.
I continue to be pleased with the decoding and coding.
Right now I am decoding up to eight messages and have tested decoding up
to ten. I am limiting the number of decoded messages due to limited
screen space.
I have set kLDPC_iterations = 20 and I am using the bp_decode(log174,
kLDPC_iterations, plain, &n_errors); option.
I have tried the ldpc_decode(log174, kLDPC_iterations, plain, &n_errors)
option; and it works but seems to take longer as you predicted.
I have not tried to make any processing time measurements yet. I hope to
do so in future.
However, even with decoding up to ten messages the decoding is done
without problem before the end of the 15 second receive time period.
So, it is feasible that we can make a CQ transmit call, decode the
replies and set a call back to a station replying to the CQ in the very
next FT8 time period.
I agree that doing the FFT at twice the sample rate and for twice the
period really makes a difference in decoding performance.
And, one of the key factors for getting this stuff to work on a small
processor is handling the FFT results as an eight bit number. This works
well since when dealing with RF signals it is best to get the receive
signal down into decibels.
I have noted that when searching for Costas Arrays and calculating a
candidate score that the score seems to be proportional to signal
strength. So, in future I might look at using the candidate score to
estimate received signal level.
I am convinced that I am decoding signals that are in the range of -110
to -120 dbm.
Yes, it will be nice to cooperate on future development. Have you been
able to join the SoftRock40 io group and access files on the this
website??
Regards,
Charley
On 2019-11-05 05:41, Kārlis Goba wrote:
Great progress, Charley! It also inspires me to keep updating my code. I'm interested to cooperate - I might be able to help with some ideas of improvements. Also curious how performant is the decoder at the moment - what settings do you use and how long does it take to decode? There's a tradeoff between accuracy (number of decodes) and time spent, of course. Also few of the advanced techniques used by the original WSJT-X are missing in my code, but on the other hand, I added a very cruical DSP trick that improved accuracy.
--
You are receiving this because you commented.
Reply to this email directly, view it on GitHub [1], or unsubscribe [2].
Links:
------
[1]
#3?email_source=notifications&email_token=AAMELXXJT3KTGAWFTDUVP5LQSEBSBA5CNFSM4GPPV5FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDBVBUY#issuecomment-549671123
[2]
https://github.com/notifications/unsubscribe-auth/AAMELXRC5JTTDR6YRVLDUP3QSEBSBANCNFSM4GPPV5FA
|
|
Karlis, Good to hear from you. I continue to be pleased with the decoding and coding. Right now I am decoding up to eight messages and have tested decoding up to ten. I am limiting the number of decoded messages due to limited screen space. I have set kLDPC_iterations = 20 and I am using the bp_decode(log174, kLDPC_iterations, plain, &n_errors); option. I have tried the ldpc_decode(log174, kLDPC_iterations, plain, &n_errors) option; and it works but seems to take longer as you predicted. I have not tried to make any processing time measurements yet. I hope to do so in future. However, even with decoding up to ten messages the decoding is done without problem before the end of the 15 second receive time period. So, it is feasible that we can make a CQ transmit call, decode the replies and set a call back to a station replying to the CQ in the very next FT8 time period. I agree that doing the FFT at twice the sample rate and for twice the period really makes a difference in decoding performance. And, one of the key factors for getting this stuff to work on a small processor is handling the FFT results as an eight bit number. This works well since when dealing with RF signals it is best to get the receive signal down into decibels. I have noted that when searching for Costas Arrays and calculating a candidate score that the score seems to be proportional to signal strength. So, in future I might look at using the candidate score to estimate received signal level. I am convinced that I am decoding signals that are in the range of -110 to -120 dbm. Yes, it will be nice to cooperate on future development. Have you been able to join the SoftRock40 io group and access files on the this website?? Regards, Charley |
|
Gents. I have created a private github repository and established Karlis and Pedro as collaborators. You should receive messages from github inviting you to collaborate. I placed the files associated with the FT8 decoding. If you wish, please take a peek at how I integrated Karlis's work into a realtime audio feed. I had wanted to establish the entire project in github from the STM32CubeIde but I could not figure out how to do so this morning. |
|
Pedro, Here is a link to another video that I made yesterday. https://www.youtube.com/watch?v=D4lKBUj9MAw Regards, Charley |
|
Hi guys, Just to report that quarantine in between I was able to pump some hours into the long delayed FT8 transceiver based on a raspberry pi using the RPiTX package and now ft8_lib. Pending some tweaking and other optimizations I've a version running using the library right out of the air, receiver SDR is a rtl-sdr dongle. Software handling rtl_fm modified to sample direct 2/Q and a front end to manage the samples prior to feed to the processing chain present at decode_ft8.cpp. Works reasonably well so far. Curious on suggestions to implement the SNR computation which is pending. Pedro LU7DID |
|
Karlis, Building on the above. What is your suggestion on the best algorithm to compute the SNR of each candidate signal? I understand the average power already computed can be used as a noise floor indeed. But it’s somewhat confusing how to link the candidate, with the power with the signal itself. Probably you will have a vantage point of view on the best way to accomplish that. |
I'd made some modifications to decode_FT8.cpp to incorporate into my code in order to fit into the Raspberry Pi, also introduced few of the pending optimizations (sort candidates, time limit, etc). Still struggling to figure out how to compute the SNR as I'm having some troubles to match how WSJTX computes it and the code structure. 73 de Pedro, LU7DID |
|
Getting the following output when decoding tests/20m_busy/test_01.wav after ffmpeg -i clone on macOS: The original file works fine, but the converted file (which should be a near duplicate) produces the segmentation fault. Thanks for the amazing work! 73 de K4BEN |
Hello,
I've been not successful to decode FT8 using .wav files taken from elsewhere. Could U place some .wav samples you are sure that works or that you use as part of your test as part of the package?
73 de Pedro, LU7DID
The text was updated successfully, but these errors were encountered: