feat: camera pass-through mode

[Schmetzler]

I added a pass-through mode for ALVR... recently only tested in the lobby, but it should work while streaming (or should it?).

You can trigger the pass-through by double tapping on the side of your headset... this is a little experimental and needs some practice you should not hit too hard and also not to soft and you must start in a resting position.
Basically it detects 2 maxima of the derivative of the accelerator sensor signal on a predefined delay.
The algorithm is somewhat described in the comments of onSensorChanged
Or you can switch to passthrough in the settings menu (which is not really usable when the device is in the headset). Maybe it can be set to a predefined button (any suggestions?)

I also added a preference Activity, where you can tinker with the values for the pass-through or disable auto detection if it interferes while playing.

There are 2 more options in the Settings menu, the Passthrough fraction (the fraction of the view that should be filled with the camera image ... this is needed as some cameras hava a really narrow FOV so the passthrough would look kinda too close) and the set Recording Hint, if set to false this can increase the FOV but usually also decreases the FPS which is kinda hard to look at, but maybe this is a no issue on newer smartphones.

[ShootingKing-AM]

seems like there are some lint issues, you can auto fix them by going though the documentation https://github.com/alvr-org/PhoneVR?tab=readme-ov-file#development

[Schmetzler]

Ah I see... Sorry... the spotless apply thing was checked always before android studio build the apk, but it seems it didn't check the c++ part.

Ok I will update I guess tomorrow or the day after tomorrow.

[ShootingKing-AM]

Ah I see... Sorry... the spotless apply thing was checked always before android studio build the apk, but it seems it didn't check the c++ part.

Ok I will update I guess tomorrow or the day after tomorrow.

Yeah, sure. I tried this and the passthough works really well, really good work !!

• The only experience related issue is the side double tap. Firstly, brilliant idea :D I got it down via practice too, but can there be a little intimation to user somehow that he is not tapping hardenough or fast enough, like maybe a small Accelerometer graph over the glView, or maybe someother way ?
• All GL functions returns errors, so we need a little more error handling. You can use this https://github.com/alvr-org/ALVR/blob/master/alvr/client_core/cpp/utils.h from ALVR. And then enclose all Gl* Functions in Gl() like in eg. https://github.com/alvr-org/ALVR/blob/28dc9205fde94e1834c7a8c5f28882f75dffb391/alvr/client_core/cpp/graphics.cpp#L337, that will take care of error handling of GL* functions. Also do check for errors in other functions if you missed :)
• Do change the log level of acclSense values log.d to verbose or trace since they are its log is very frequent
• Also use camelcase variable naming thoughout the code, its somewhere snake_case and sometimes camelCase. If the variable is member of the current class it has to be appended with "m" like mLowerBound, if its a variable only in the present context then simply lowerBound is good
• If possible you can seprate the passthough code from main files (separate from alvr_main.cpp and from ALVRActivity.java), because they are huge no of lines in their own right
• A little more documentation and some meaningful variable naming would be awesome, its hard to understand the code logic espicially in onSensorChanged() (for double tap detection)

Rest everything looks good

[Schmetzler]

Ok so, lets see what the steps are:

• User feedback (why it is not working) ... I guess this one is really difficult as I do not know in advance that someone want to do the double tapping.. but maybe the graph thing can be done (but don't know... this is kinda hard I think)... Maybe we could make a "training mode" for the correct thresholds.
• Wrap GL functions
• Change Log-Level (using Log.v now)
• camelCase for all variables, and mVariable for member variables
• separate passthrough code from ALVRActivity.java
• separate passthrough code from alvr_main.cpp
• more documentation (and meaningful variable names) (well for the double tap detection I guess an image would make things a little clearer... I will prepare something.

Another question... should I squash the commits together to one when I am finished here?

[ShootingKing-AM]

• Training mode sounds awesome and easier.
• Forget about squashing, i will take care of that :)

Edit: Also if you want more immediate response you can join Discord and communicate in #dev channel :3

[ShootingKing-AM]

Just to make myself clear, Training mode is nothing fancy, just an Activty showing realtime graphs of 3axis of accelSensor so that they(users) get used to it. You can use an existing libary for graphs like https://github.com/PhilJay/MPAndroidChart-Realm (other libs here https://github.com/lucasrafagnin/android-charts)

If you are looking for some button, there is a suggestion here: #77

[Schmetzler]

So here some points to understand the algorithm:

• First we gather accelerometer data from at least 600ms (this is around 9-10 datapoints)
• We calculate the absolute derivative for each axis ($|x_i - x_{i+1}|$)
• We calculate the sumMean on current position of the derivative:
  • mean for each axis
  • sum of these means

For each axis:

• we split the accelerometer derivative into 2 windows
• we calculate the maximum on each window
• All of the following must be true to set it as a possible trigger:
  • are the maxima minus the respective window edges higher than lower bound
  • are the maxima and the sumMean between lower and upper bound
  • was sumMean on first position of window below max(lower bound - 0.3, 0.5)
  • is the maximum of second half not on the edge of second half

(Beware It is not important on which axis this applies... but that is also the reason why I use sumMean to prevent it to trigger when something like this happens in one axis, but the other axis have more "movement")

• After possible trigger found, check whether the sumMean drops below lower bound during another 600ms

The current implementation needs a steady phase before the double tap and a steady phase after the double tap

[ShootingKing-AM]

Nice explanation. Do add this documentation into the code :3

[ShootingKing-AM]

ps: in #334 i added utils.h and modified it according to phonevr, i handled errors in existing alvr_main.cpp GL* calls which i forgot earlier. You may want to use that, it will be merged when tests are done.

[Schmetzler]

Well it seems like I have to cleanup my git history somehow... as rebasing needs to many merges for unneeded commits. I am not really well versed in this kind of stuff... so I think I will make a branch, rebase there (if this is possible) do some commit cherry picking and hope everything works as suspected... and after that I think I must rewrite history of the master branch so that it can be pulled from here.

Sounds that reasonable or is my thinking wrong?

[ShootingKing-AM]

just update/merge the master into your branch if rebasing your is hard, if you are using Github Desktop it will prompt you to resolve conflicts while merging.

Select Update from Master and it will show the below panel to fix conflicts

You can resolve those conflits easily there. No need to rewrite history on this pr, you can make it as messy as it can be dont worry about it.

Or should i rebase it for you ?

[Schmetzler]

Well I did some cherry-picking, rebasing force-pushing and so on...^^
On another branch I started to implement a visualisation, which currently works, but uses HelloChart (which is not available in the repos). So currently I use a downloaded jar file... Maybe I will use the thing you suggested, but I was a little worried about apk size, if this chart library just blows up the size for something that is not so important. (But maybe the mentioned chart library is also neglectable small... I do not know)

[ShootingKing-AM]

Quantify its size, i dont think android libraries would be that heavy. Still some linting errors :3

[ShootingKing-AM]

/ok-to-test sha=16e98a2

[Schmetzler]

I have a question... What am I missing? I don't get the last CI Test... Do I need to add some tests?

[ShootingKing-AM]

Just reopened and closed the PR so that the tests will get retriggered, now.

If i remember there were some errors (maybe linting errors). Were you able to test at local properly ?

[Schmetzler]

I don't get the Test APK error as it fails on Screenshot uploading... but I haven't even implemented a test (well... I know... it is not good design)... Maybe something with the CI Action is wrong?

[ShootingKing-AM]

/ok-to-test sha=a6ebf88

[github-actions]

📷 Screenshots of tests:

📱 Pixel_2(AVD) - 11

🔧 ALVRActivityTest_saveDeviceScreenBitmap

🔧 ALVRActivityTest_saveDeviceScreenBitmap_LookingDown

🔧 ALVRActivityTest_saveDeviceScreenBitmap_LookingFrontDown

🔧 ALVRActivityTest_saveDeviceScreenBitmap_Popup

🔧 InitActivityTest_saveDeviceScreenBitmap

🔧 MainActivityTest_saveDeviceScreenBitmap

🔧 SettingsActivityTest_saveDeviceScreenBitmap

🔧 view-op-error-1

For commit a6ebf88

[ShootingKing-AM]

I don't get the Test APK error as it fails on Screenshot uploading... but I haven't even implemented a test (well... I know... it is not good design)... Maybe something with the CI Action is wrong?

CI Actions are atlast fixed.

[ShootingKing-AM]

@Schmetzler Anything left on your end to do here ?

[Schmetzler]

If I find the time... I would like to look into the variable names, add some comments and maybe add some test cases... But I am not very well versed in android development (and also testing) anyway... I can imagine to add a test case where I put some predefined data into the algorithm and check if it triggers the passthrough mode... But I think it would take me at least a week (I need to find time, and I resetted my OS so I need to setup the development environment again)

[ShootingKing-AM]

its okay, take your time. Let me know if you are stuck somewhere : )

[Schmetzler]

Sorry I cannot afford to work on it anymore. Get everything to build is just too much of a hassle for me and stresses me... You may merge it the way it is (which is working and functional) but I won't change any naming or add comments without the possibility to check if everything works. And even for the testing it seems that there is no possibility to mock the sensordata.

If the code is too complicated I am willing to explain.

Hope you understand and find the code usefull nevertheless.

Thanks for your patience

[ShootingKing-AM]

Its okay, no need to worry too much, i will look into this : )