Announcement

Collapse
No announcement yet.

Lag at West (and South) Super Regional - anything we can do about it?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Lag at West (and South) Super Regional - anything we can do about it?

    We would really like to hear some experts comments on the lag issues that many teams seemed to suffer at the West and South Super regionals.

    We were at West, Despite the FTA's HEROIC efforts at channel changing before matches, there seemed to be a large number if issues related to this. We felt that 50% of our matches were impacted, it seemed to be a widespread problem, not just contained to our team. 100ms ping times were considered good by the drive team, often spiking ~1000ms during matches. ( Moto G4 Plays, 3.5 FTC SDK, Android 7.0.1)

    This is the first event since the NXT that we've encountered this difficulty and wondering why, what might have changed?, and what we (everyone) can do to avoid future problems. We'll throw out a few ideas our team has tossed around, but we're no experts, so we'd love some feedback discussion.

    - our State Event was ~50 teams vs 72 at West, have we just reached the S/N tipping point? (Worlds last year didn't have problems like this)

    - Has anything changed in the SDK involving WiFi direct?

    - Do certain phones 'channel hop'? Motorola maybe? Is there a way to keep that from happening?

    - West had open public WiFi, (anyone know about South?) could that have just produced too much noise ? Presumably all the spectator phones were mostly using cellular data to check scores and stream, but?

    - Is there anyway possible to use the 5ghz spectrum for WiFi direct connections? That seems like it would be quite an advantage.

    Any other comments or 'best practices' that would help teams would be greatly appreciated.

  • #2
    Is there anyway possible to use the 5ghz spectrum for WiFi direct connections? That seems like it would be quite an advantage.
    According to https://www.wi-fi.org/discover-wi-fi/wi-fi-direct :

    Does the specification underlying the Wi-Fi Direct certification program work on both frequency bands?

    Yes, the specification underlying the Wi-Fi Direct certification program supports operation in both 2.4 GHz and 5 GHz. Devices operating in the 2.4 GHz frequency band only and devices operating in both the 2.4 GHz and 5 GHz frequency bands can be certified under the Wi-Fi Direct program. Not all Wi-Fi Direct-certified devices support both frequency bands, however, so you should check which bands your devices support.
    So the Wi-Fi Direct specification allows 5 GHz, but I could find very little information about whether Android supports Wi-Fi Direct with 5 GHz. The closest I got was https://stackoverflow.com/questions/...frequency-band :

    I tested create a wifi direct group on the Yoga tablet 2 pro (Android 5.0) and had the same issue. It's an old question but I can't find an any reference to it anywhere else and this is still the top Google result for this topic.

    If the wifi direct group is created when the device is connected to a 5Ghz access point, then that wifi direct group will also be 5Ghz. If the group is created when there is no wifi connection active or a 2.4Ghz connection active, then the wifi direct group is 2.4Ghz.

    I'm not sure what happens if the connection changes. E.g. you could try disconnecting the wifi from the access point, establishing the wifi direct group, and then reconnecting to the access point.
    So I think it's worth a shot to try these steps (at your practice space, not at a competition! *):
    1. Install the Wi-Fi Analyzer app from the Google Play Store onto a non-robotics phone that can monitor both the 5 GHz and 2.4 GHz bands. **
    2. Use a 5 GHz-capable access point to set up an open SSID on the 5 GHz band. Make sure the access point has no SSIDs active on the 2.4 GHz band.
    3. From the Robot Controller phone, connect to this SSID using the Settings -> Wi-Fi applet.
    4. Start the FTC Robot Controller app on the Robot Controller phone. Use the Self-Inspect activity to clear all Wi-Fi Direct groups.
    5. Start the FTC Driver Station app on the Driver Station phone. Use Self-Inspect to clear all Wi-Fi Direct groups.
    6. Reboot the Robot Controller and Driver Station phones.
    7. Make sure the Robot Controller phone reconnects to the 5 GHz SSID using Settings -> Wi-Fi.
    8. Start the FTC Robot Controller app on the Robot Controller phone.
    9. Use the Wi-Fi Analyzer app on the non-robotics phone to verify that the Robot Controller phone is broadcasting a 5 GHz SSID of the form DIRECT-xx-4220-A-RC (assuming your robot phone is named 4220-A-RC).
    10. Start the Driver Station app on the Driver Station phone and use the app to pair with the Robot Controller. Make sure the Driver Station phone can use and maintain the connection (by, say, listing the op modes).
    11. Disable the 5 GHz SSID at the access point.
    12. Reboot the Robot Controller and Driver Station phones.
    13. Start both the Robot Controller and Driver Station apps and make sure they can reconnect.
    14. Use the Wi-Fi Analyzer app on the non-robotics phone to verify that the Robot Controller phone is still broadcasting the DIRECT-xx-4220-A-RC SSID on the 5 GHz band.
    If you get through these steps, then you should feel pretty confident that the phones will use 5 GHz.

    * Seriously, don't do this at a competition. The Judging and Inspection page for the East Super-Regionals says (and World's may be similar):

    You will be assigned a Wi-Fi channel to operate your Robot Controller on. The inspectors will configure your Robot Controller for your assigned Wi-Fi channel. You Robot Controller must remain on your assigned Wi-Fi channel for the duration of the event. Operating your Robot Controller on a Wi-Fi channel other than the assigned channel, unless specifically directed by an FTA, CSA, WTA, or the event Technical Director, is a violation of Tournament Rule <T6>.
    ** If you don't have a spare phone, but you have a Windows laptop with a dual-band adapter, you can use that instead. In step 9, start the Programming mode on the Robot Controller app. Then use the laptop to connect to the Programming interface via the Robot Controller's DIRECT-xx-4220-A-RC SSID. Then run this command:

    Code:
       C:\>netsh wlan show interfaces | findstr Channel
    If the result is higher than 11, then the SSID is using 5 GHz; if it's 11 or less, then it's using the 2.4 GHz band.
    John McDonnell
    Volunteer, former mentor

    Comment


    • #3
      At ESR many teams had similar problems and from what I found it was due to all the teams being put on 1 channel and not being spread out. During inspection, every team was put on channel 6. Many teams have multiple sets of phones so lets say that in the pits after inspection there were about 72*2 or 144 wifi networks on channel 6 due to the wifi direct connections. That is way too many networks and the ping was around 400-1000. Our team switched our channel to 11 which is the network they were posting the scores on and found significantly less ping time closer to 100 as a max and when we switched to channel 1, the max was about 80ms. As the day went on, I think more and more teams swapped channels making 6 less congested so everybody's robot worked more consistently. I highly recommend getting a wifi analyzer to monitor which channel your phone is on since sometimes when you press change channel, it doesn't actually change the channel.

      Comment


      • #4
        At the North Regional, they did not force everyone onto a certain channel and I think that was good. However, we had a huge difference between our Nexus 5 phones and our ZTE phones. The Nexus 5's did not work well and the ZTE phones in the exact same environement, worked great.

        So we have been doing some testing regarding this issue at least as it regards the Nexus 5. We had terrible ping problems and lag at the North Super Regional. An FTA tried valiantly to get it to work, but no go. They tried (or thought they tried) all channels with no improvement and we were having problems both at the practice field and in the arena. The second day and before our last match, we finally gave up with the Nexus 5 and changed back to our old ZTE phones. Instantly, problem solved and we scored a 568 in the final match.

        So with a little testing here's what we learned about the Nexus 5 used as a robot controller phone. We used Wifi Analyzer by VREM:
        - Sometimes, when the conditions are bad for 2.4GHz, the phones will choose 5GHz when starting the Robot Controller app for the first time. At least in our environment, this is sometimes very bad. Even though there is little 5GHz traffic, ping is sometimes consistently higher than 200. Other times it's great.
        - Once the app has been started for the first time, the wifi channel can't be changed except as detailed below.
        - In the Robot Controller app, the "Advanced" -> "Change Wifi Channel" setting does not work. It will not change the channel (except as described below).
        - The phones can choose 5GHz even when you have the Android setting: Wifi -> Advanced -> "Wif-Fi frequency band" -> "2.4GHz only" on both phones. You have no choice in the matter and the FTC Robot Controller app must not even be aware that it's not in control of the channel selection.
        - There are only 3 ways we could find to get a new channel:
        1. Reboot the phones and start the Robot Controller app. In our experience, our phone randomly picked among 4 channels: 1, 6, 11, and 161 (yes 5GHz)
        2. Exit the Robot Controller app and cycle wifi off and then on. Again it will randomly pick from among the same 4 channels.
        3. To choose your own channel, do the following procedure in the Robot Controller app:
        a. "Settings" -> "Advanced" -> "Change Wifi Channel"
        b. Select a channel and then hit the "Wifi" button at the bottom of the screen
        c. Turn off Wifi. Then Turn on Wifi.
        d. Hit the back button to go back to the channel screen
        e. Select your channel again.
        f. Look at your wifi analyzer app on another phone and wait until your old channel signal disappears. Note: if you do not wait long enough and immediately return to the Robot Controller main screen, it will say that it is changing the channel, but the channel is not changed.
        g. Hit the back button to get back to your Robot Controller main screen. When your wifi connects, your analyzer app should show the new channel. It will stay at this channel even if exiting the app. Just don't reboot or cycle wifi.

        I hope someone can reproduce this on their Nexus phones. If it turns out there isn't a simple solution for this problem, we will figure out how to come up with new phones before Worlds because our ZTE phones have an intermittent crashing problem when we're trying to grab an image from Vuforia.

        Our Nexus 5's are updated to the latest version of Android that is available which for our phones is 6.0.1 and fully updated with Play Store updates.

        Thanks, and I hope this is helpful.

        fyi - This seems like a good wifi analyzer app with no ads: https://play.google.com/store/apps/d...m.wifianalyzer
        You do not have permission to view this gallery.
        This gallery has 1 photos.

        Comment


        • #5
          We had a very similar experience as to what you describe, but with Samsung Galaxy S5 phones at State where everyone was being told to use the same channel. We were never successful in getting the S5's to change channel, but didn't come up with the steps you suggest. Old ZTE backup phones worked better. In addition to the FTC field systems being on both 2.4 and 5 GHz, there were a number of powerful access points from the venue on 1, 6 and 11, plus the usual random assortment of hot spots from unknowing or uncaring spectators.

          If FTC sticks with WiFi, something needs to change.

          Comment


          • #6
            Totally agree with that.

            In case there is a possibility that this can actually be fixed (because the ZTE phones worked in many cases), what could be done? Possibilities:
            1. Fix the app so that it can actually change the channel on all phones
            2. Find a similar environment to do some testing like basketball game. Go in with 8 pairs of handsets communicating in wifi direct mode and look at ping times.
            3. Reconsider the advice of only using 1, 6, and 11. In our tests 3/4 and 8/9 work well in an environment with lots of access points.
            4. If using 5 channels, assign one fifth of the teams to each channel. Do not assign an entire division to one channel because this even more concentrates the location of multiple wifi direct networks on the same channel.
            5. Periodic announcement to the crowd to turn off Wifi on their phones.
            6. Make sure volunteers do the same (cellular data only). Maybe they already do this.

            5294-jjkd, I'm interested to know if our method of changing channels on the Nexus 5 will also work on the S5? Are the S5's on Android 6.0.1?

            It may be that the newer phones have wifi chips that are more "polite" shall we say. i.e. they see other wifi traffic and wait their turn causing high ping times

            Comment


            • #7
              Based on my 'Google' research, getting the Galaxy S5 to change their WiFi Direct channel seems to require rooting the phone. No amount of fiddling we did seemed to work, but we didn't try your exact sequence.

              1. I don't think this is possible. It is pretty clear that at least some phones require root to do this.
              2. Maybe good for testing possible fixes 'in the wild', but not so helpful for diagnostics. Probably easier to set up repeatable controlled conditions inside a Faraday cage.
              3. 2.4 GHz WiFi is wideband, 1, 6 and 11 are the only (almost) non-overlapping channels. Using any channels closer together generates more 'adjacent channel' interference in addition to the 'co-channel' interference from the devices on the same channel. I think that could actually be worse in some cases.
              4. Spreading out the devices across three channels is probably a good idea.
              5. This has been done but was not effective at any event I have attended. Probably needs calling folks out directly (Hey, John's iPhone - you there in that section, turn off your hotspot!).
              6. Don't know.

              Comment


              • #8
                Thanks for the conversation, feel like it's important to keep this going/get solved before the championship events. We have been using G4 Plays (no 5ghz available) Fooling around in a quiet wifi environment, I'm beginning to think the ZTE phones have a stronger radio than the G4 Play but still need some further testing. I thought I'd at least throw up this screen shot. It seems like our G4 Plays (7.1.1) always choose a 'best' channel when the wifi direct starts. Without other traffic, its usually 1,6 or 11, but here's the picture in our school workroom, three pairs of phones when started up, you can see the schoo'ls wifi access points are reasonably balanced(this was after the school day, so not huge amounts of actual traffic) The ZTE starts up on a repeatable channel (1) but the G4 Plays pick intermediate channels (3,9?).

                In high traffic environments, is it better or worse to use these intervening channels? We see both yes and no answers to this question in our searches, any engineers, FIRST care to comment?

                The large number of phones certainly doesn't help, We suggest that field inspectors really require teams to only have a single pair of phones active at one time. This does not seem like it would be a burden to anyone.

                We'd still love to hear some comments on open wif., Score refreshing, live streaming and its possible impacts on the venue environment.

                Thanks.


                wifi1.jpg

                Comment


                • #9
                  Use of bluetooth should be banned at FTC events. It does not play nicely with 802.11b/g/n in 2.4GHz ISM spectrum. We're busy trying to shutdown hot-spots and so on but totally ignoring other common sources of 2.4GHz noise.

                  Comment


                  • #10
                    Our programmer did a video. Apparently it does work on the S5 with a slight difference. circuitbreakersrobotics.com or https://www.youtube.com/watch?v=vy_X_hBUGWk&t=1s

                    Comment


                    • #11
                      Curious if the channel change method works on the G4 Play.

                      If 5294-jjkd (and what seems to be the prevailing wisdom) is right, the middle channels are not good, causing problems for everyone. So the G4 Play would definitely need to be fixed if it's randomly selecting the channels in between. I have seen arguments both ways, but nothing specific to wi-fi direct and certainly not pertaining to the actual situation being that the two phones are very close to each other and there are 3 other strong wifi direct signals within 12 feet. And with the 3 other fields plus the teams queuing, there are a total of 32 wifi direct signals within 100 feet. Not sure if this matters, but most research is talking about access points which have maybe 3x? the signal strength of a phone.

                      Comment


                      • #12
                        WiFi latencies increase in proportion to the number of APs operating on the same channel or overlapping channel (i.e. in proportion to the number of WiFi Direct group owners [RC phones] operating on the same/overlapping channel).

                        The only way to reduce latencies is to reduce the number of APs [WiFi Direct group owners] operating on the same or overlapping channel at a venue.

                        One way to reduce the number of APs operating on the same or overlapping channel at a venue is to have the DS phones connect to the RC phones via a WiFi router instead of WiFi Direct. Managing the channels of 3 or 4 WiFi routers at a venue (one per field) seems much more feasible to me than managing the channels of dozens of WiFi Direct group owners [RC phones] at a venue.

                        WiFi routers can generate stronger WiFi signals than an RC phone which reduces interference; thus yields lower latencies than WiFi Direct. Also, using a WiFi router allows field personnel to monitor WiFi performance of the channel, and re-run a match if the WiFi channel fails to perform adequately throughout any given match.

                        I realize that there are drawbacks to going back to WiFi routers. I wasn't around back then so I don't know if the drawbacks outweigh the benefits.

                        Comment


                        • #13
                          Originally posted by kkinneer View Post

                          I hope someone can reproduce this on their Nexus phones. If it turns out there isn't a simple solution for this problem, we will figure out how to come up with new phones before Worlds because our ZTE phones have an intermittent crashing problem when we're trying to grab an image from Vuforia.

                          https://play.google.com/store/apps/d...m.wifianalyzer
                          My team's ZTE phones haven't crashed using Vuforia, just saying. We tested our Vuforia programs well over a hundred times (documenting the results). The app sometimes would crash when the robot hit the cryptobox too hard, but not due to a random unknown error.

                          Comment


                          • #14
                            If it turns out there isn't a simple solution for this problem, we will figure out how to come up with new phones before Worlds because our ZTE phones have an intermittent crashing problem when we're trying to grab an image from Vuforia.

                            Originally posted by RoboticsFun View Post

                            My team's ZTE phones haven't crashed using Vuforia, just saying. We tested our Vuforia programs well over a hundred times (documenting the results). The app sometimes would crash when the robot hit the cryptobox too hard, but not due to a random unknown error.
                            But I think one of the earlier versions of the apps did have random crashes, updating the phone apps solved that problem for us early in the season.

                            Comment


                            • #15
                              Originally posted by RoboticsFun View Post

                              My team's ZTE phones haven't crashed using Vuforia, just saying. We tested our Vuforia programs well over a hundred times (documenting the results). The app sometimes would crash when the robot hit the cryptobox too hard, but not due to a random unknown error.
                              I should've been more accurate. "Crashing" is not correct. Our programmer is grabbing a bitmap image buffer from the Vuforia image buffer. This bitmap is then segmented and red/blue counted to determine the jewel color. Once the actual problem was determined, it does not crash. What does happen is that for some reason, intermittently (like 1 out of 6 or 1 out of 10 runs) the image buffer will be null when using the ZTE phones. It was clearly not null when Vuforia read it because it is recognizing the VuMark just fine. This never happens (literally zero times in 2 months and maybe hundreds of runs) with the Nexus phone. I think our programmer was at a dead end and didn't find any way to figure out what was different about the ZTE and didn't see any downside to just using the Nexus.

                              Having said that, now that we know how to actually change the channel on the Nexus phones, we'll probably use them at Worlds unless it becomes a problem.

                              I don't want to get this thread off track, so if anyone has any tips about pulling a bitmap from the camera, pm me.

                              Comment

                              Working...
                              X