MR Gyro - 200 to 250 mSec lag in changes of heading value? Need Help or experiences.

[FTC4106]

We thought to experiment with the Modern Robotics Gyro and we are seeing a huge lag in readings, pretty consistently in the 200 to 260 mSec range. Has anyone else ever seen this and is there a known cause or simple mistake we should avoid making?

We wrote a simple test loop to isolate the problem, as shown in the code below. When viewing the log output on the Android monitor screen, you'll see the timetamps for each log each time through the loop. You'll see several loop iterations and then the heading value will take a big jump, after not given any of the readings in between

The problem occurs whether or not the code with the motors and range sensor is used. Those were added for convenience and a source of double checking. The Thread.sleep was put in to keep log output down. The problem happens with or without it. The opmode.idle() call was added in hopes that it would help. It didn't. When adding the range sensor, we see the range sensor values changing every ~50 mSec, which seems okay. A not unreasonable 20 Hz rate. The extra timer was added to make sure we weren't being fooled by the logging timer somehow.

Gyro documentation says it updates the raw values at 750 Hz, but doesn't say how fast it updates the integrated Z values. Assumption was every time the data changes, but certainly faster than 4 Hz, which is unusable.

Further information. We tested this app with our robot and with a new hardware setup with new sensor, new DIM, new PDM, new cables. The gyro works as expected in this setup. We then moved the gyro, DIM and cable to the PDM on our robot. It doesn't work. It works exactly the same as it does with the sensor, DIM and cable already on the robot. The only difference is the configuration file is configured for 4 motor controllers and 2 servo controllers and some additional sensors, all of which seem to work perfectly fine to run our robot.

All modules in the system have been upgraded by MR
We are using SDK version 2.2

@Autonomous(name="VerifyGyro",group="VV")
public class VerifyMRGyroLag extends LinearOpMode{
public ModernRoboticsI2cGyro gyro;
public ModernRoboticsI2cRangeSensor rangeSensor;
private DcMotor motorRightFront;
private DcMotor motorRightBack;
private DcMotor motorLeftFront;
private DcMotor motorLeftBack;
@Override
public void runOpMode() throws InterruptedException {
motorRightFront = hardwareMap.dcMotor.get("FrontRightMotor");
motorRightBack = hardwareMap.dcMotor.get("BackRightMotor");

motorLeftFront = hardwareMap.dcMotor.get("FrontLeftMotor");
motorLeftBack = hardwareMap.dcMotor.get("BackLeftMotor");
//REVERSE the RIGHT drive motors for Autonomous
motorLeftFront.setDirection(DcMotor.Direction.FORW ARD);
motorLeftBack.setDirection(DcMotor.Direction.FORWA RD);
motorRightFront.setDirection(DcMotor.Direction.REV ERSE);
motorRightBack.setDirection(DcMotor.Direction.REVE RSE);

motorRightFront.setMode(DcMotor.RunMode.RUN_WITHOU T_ENCODER);
motorRightBack.setMode(DcMotor.RunMode.RUN_WITHOUT _ENCODER);
motorLeftFront.setMode(DcMotor.RunMode.RUN_WITHOUT _ENCODER);
motorLeftBack.setMode(DcMotor.RunMode.RUN_WITHOUT_ ENCODER);

gyro = (ModernRoboticsI2cGyro)hardwareMap.gyroSensor.get( "gyro");

// get the range sensor
rangeSensor = hardwareMap.get(ModernRoboticsI2cRangeSensor.class , "BeaconDistanceSensor");

// calibrate the gyro
gyro.calibrate();
// make sure the gyro is calibrated.
while (gyro.isCalibrating()) {
Thread.sleep(50);
idle();
}

waitForStart();

// start the Timer
ElapsedTime timer = new ElapsedTime();

setMotors(-0.6,-0.6,0.6,0.6);
while(opModeIsActive()){
RobotLog.a("T: " + timer.time() +" Gyro" + gyro.getHeading());
RobotLog.a("T: " + timer.time() +" Range" + rangeSensor.getDistance(DistanceUnit.INCH));

Thread.sleep(10);
idle();
}

setMotors(0.0,0.0,0.0,0.0);
}

void setMotors(double lf, double lb, double rf, double rb){
motorRightFront.setPower(rf);
motorRightBack.setPower(rb);
motorLeftFront.setPower(lf);
motorLeftBack.setPower(lb);
}
}

[DanOelkeFTA]

Good work! That would explain some of the effects we've seen.

We ran into similar problems with Adafruit color sensor. (We then learned about setting the integration time)

One thing we did was a bit of logic like this: (note this is not compile-able code - just for illustration purposes)

int oldValue = 0;
int counter = 0;
while (opModeIsActive()) {
    int newValue = sensor.get(); // change to your sensor type
    if (oldValue != newValue) {
        oldValue = newValue;
        counter++;
        telemetry.addData("Update Rate", " %.2f Hz", counter / timer.seconds() );
        telemetry.update();
    }
}

That gave us the update rate in very plain view. Just be sure that timer is non-zero to avoid the divide by zero.

Have you contacted Modern Robotics about this?
What do they say?

[Philbot]

Further information. We tested this app with our robot and with a new hardware setup with new sensor, new DIM, new PDM, new cables. The gyro works as expected in this setup. We then moved the gyro, DIM and cable to the PDM on our robot. It doesn't work. It works exactly the same as it does with the sensor, DIM and cable already on the robot. The only difference is the configuration file is configured for 4 motor controllers and 2 servo controllers and some additional sensors, all of which seem to work perfectly fine to run our robot.

We need some more information.

1) From the quote above it seems like you have one setup that works, and one that doesn't.
Can you explain the difference (other than just the config file.... there must be more.) Phones, MR hardware versions.... etc.

2) Some actual data would help.

Can you program the configuration that does not send commands to the motors in the loop etc and also does not read the range sensor.
So this is just a pure gyro test, logging only gyro data.
Take out the idle(), and use a straight sleep(50) call.
Record 5 seconds worth of turn data.

Now, we need to see the data ....
If you have a PC, put the phone in media mode (PC connection or USB configuration Settings option)), and transfer the file using file explorer.
You can also use ADB, but I'm not sure how.
If you can't transfer the log data to your PC, then how about a screen photo or two showing some problem data.

[FTC4106]

Unfortunately, the team had to abandon this experiment for now due to time pressures. To tie off some of the questions for future readers who may read this.

To Philbot's first question: After seeing the initial problem, we built a mini system with just phone, PDM, DIM and gyro using the same test loop to test it. It worked. So we figured there was something wrong in our electronics or wiring. So we moved the DIM, gyro and phone and cable between DIM and PDM to the robot. Using the same software (no motors or range sensor) it exhibited the lag. The only two physical components that are different are the cable between the two setups are the cable between the phone and the PDM and the PDM itself. Not sure if we switched the config file back to the robot's config or if we were using the test setup's config, but I do know the software was not attempting to initialize any hardware except the MR gyro.

We had performed the test as you say without the idle() and sleep and without the other hardware. That was the starting point. The rest was added later to try to get a reference point from another sensor and to rotate the gyro more consistently.

Not sure when the team will be able to get back to this experiment. Tournament time pressures caused them to switch back to their NavX IMU solution, which seems to work well enough, but of which we are still a bit skeptical. With this sensor, and a very similar test loop as above, we are seeing updates in the 20 to 80 mSec range - as triggered by the NavX's "isUpdateAvailable() method. Every once in a while we see a longer update time, in the 100-200 mSec range, which really is concerning, too.

We want to be clear that we aren't suggesting that there is anything wrong with the MR sensor. We think there is something whacky in our setup, something weird with our ZTE phone configurations or ??. But can't figure out how to isolate it, and unfortunately don't have the time to spend on the MR gyro for now.

[Philbot]

Unfortunately, the team had to abandon this experiment for now due to time pressures. To tie off some of the questions for future readers who may read this.

To Philbot's first question: After seeing the initial problem, we built a mini system with just phone, PDM, DIM and gyro using the same test loop to test it. It worked. So we figured there was something wrong in our electronics or wiring. So we moved the DIM, gyro and phone and cable between DIM and PDM to the robot. Using the same software (no motors or range sensor) it exhibited the lag. The only two physical components that are different are the cable between the two setups are the cable between the phone and the PDM and the PDM itself. Not sure if we switched the config file back to the robot's config or if we were using the test setup's config, but I do know the software was not attempting to initialize any hardware except the MR gyro.

We had performed the test as you say without the idle() and sleep and without the other hardware. That was the starting point. The rest was added later to try to get a reference point from another sensor and to rotate the gyro more consistently.

Not sure when the team will be able to get back to this experiment. Tournament time pressures caused them to switch back to their NavX IMU solution, which seems to work well enough, but of which we are still a bit skeptical. With this sensor, and a very similar test loop as above, we are seeing updates in the 20 to 80 mSec range - as triggered by the NavX's "isUpdateAvailable() method. Every once in a while we see a longer update time, in the 100-200 mSec range, which really is concerning, too.

We want to be clear that we aren't suggesting that there is anything wrong with the MR sensor. We think there is something whacky in our setup, something weird with our ZTE phone configurations or ??. But can't figure out how to isolate it, and unfortunately don't have the time to spend on the MR gyro for now.

Thanks for the clarification. It is a bit disconcerting. We're using a MR Gyro on our robot. I may try to add some log data to see if I see a simlimar anomaly.

Do you know if the overall comunications with the GYRO was slow on the robot, or only the rate at which the values changed?

If there is a partial USB failure causing packets to be lost (if USB even works that way) it could be that it was taking longer to get any valid read of the Gyro. This delayed coms rate would naturally account for the slower updates.
However, if the gyro was being read at the normal faster rate, it is odd as to why the heading was not updating as regularly.

[FTC4106]

Do you know if the overall comunications with the GYRO was slow on the robot, or only the rate at which the values changed?

It's so hard to tell. At first we couldn't tell if it was the gyro not sending the data or if it was sending it, but it was getting buffered/thrown away somehow by the gyro i2c driver or maybe lost in some non-thread safe queue or ?? We bought a cool logic analyzer this summer to debug some other i2c work we were doing and we were tempted to hook it up and see if the data is changing coming from the sensor and just not getting reported. But we just had to save this for another day so we could keep moving forward for competition. Coach's best guess is that it is not the sensor, but some other system factor that remains a mystery -- like maybe the USB driver on the phone or DIM, so like you say :

If there is a partial USB failure causing packets to be lost (if USB even works that way) it could be that it was taking longer to get any valid read of the Gyro. This delayed coms rate would naturally account for the slower updates.
However, if the gyro was being read at the normal faster rate, it is odd as to why the heading was not updating as regularly.

Which is why we added a sensor in to see if that sensor, coming form the same DIM, was updating faster than the gyro was ... and it was. The range sensor, another i2c sensor, was clearly reporting data 4x-5x faster than the gyro. So it sort of ruled out the USB problem...
The reason for using a 2nd i2c sensor of a different type, and not just e.g., an analog sensor, was to try to isolate whether it was a i2c bus related problem or something specific to the gyro driver. The range sensor didn't seem to have problems, implying that the i2c bus wasn't completely screwed up. And we ran the test with and without other devices on the i2c bus to ensure that it wasn't some address conflict causing it. So the MR Gyro i2c driver could still be the issue ... especially if there are threading issues ... which we may not be equipped to determine.

Last night we added the NavX gyro back in and it was updating at a faster rate. 20-80 mSec, but those long pauses still are troubling. And if the raw values are updated at 750 Hz and being integrated at each capture, then why would the values only be updating that slowly... Another clue that we have something that just isn't right in the system and that we need to get that logic analyzer out.

Another troubling real-time issue: How long does it take to send a message to a motor controller and have it acted on? Get the motors up and running in RUN_WITHOUT_ENCODERS and with stop condition for brake. Read and log the encoder values in the loop. Now log the time just before sending a series of setPower(0.0) to the motors. Continue printing out the encoder values and look for the spot where they stop changing, indicating that the motors actually stopped. How much latency should we expect from issuing the stop message to the time the motors stop? How deterministically (range of time) can we expect that message to process? If the latency is deterministic, it potentially can be dealt with (i.e stop early by some amount of time). What we found we weren't happy with. :/ Latency is often over 200 mSec. Range ov times rarely goes under 150 mSec and can stretch to 250 mSec? It seems too slow to us. Is that typical? Or maybe that's an indicator of a deeper problem in our setup ... we would love to find out what other team's setups are giving for these times. Meantime, we are going to set up a new independent little test system to measure this, just to ensure it is the same across other systems.

[FTC4106]

As regards the message lag problem, we wrote a test last night that ran the motors up to speed (1.0) on a NeverRest 20 with wheels in the air, logged the time, issued the setPower(0.0) messages to stop the motors and then continued reading the encoders until they settled and logged that time. What we found was that the time it took from just before the stop message to the settle varied from 30 to about 75 mSec. It's a livable range. Still had a few crazy outliers, which is scary. Posted here for future reference. If someone has found a way to
A) get a more consistent time or
B) eliminate the lag (other than using RUN_TO_POSITION)

Please post how!

[FTC7152Programmers]

Were you using ZTE Speed phones? We are having the same problems with the MR gyro. 200-250ms between updated headings on our full robot when turning in autonomous, but <10ms on a simple test harness with just the gyro and minimal electronics. We tried a faster (non-supported by FTC) Android phone and it does get more responsive, but still 80-100ms between updates.

[4634 Programmer]

I think there's a 250ms delay by default when sending data to the driver staion... not completely sure...

[FTC7152Programmers]

We are logging the heading and timestamps to the robot phone, not using telemetry to the driver phone.

[skatefriday]

gyro.getHeading() is simply going to return the last cached heading read from the device.

The driver for the gyro sits on a thread that continually reads the device if there's no other work to perform. This typically happens in roughly 10ms intervals. Can you please rerun your tests using a non-linear opmode?

[Cheer4FTC]

gyro.getHeading() is simply going to return the last cached heading read from the device.

The driver for the gyro sits on a thread that continually reads the device if there's no other work to perform. This typically happens in roughly 10ms intervals. Can you please rerun your tests using a non-linear opmode?

What if there is "other work to perform"? Could that dramatically slow down the update rate? What types of "other work" would result in this slowdown? Other sensor reads? Motor/servo access? Camera access?

It sounds like the reading of the gyro heading is done in a separate thread. Is it possible that the variable that stores the gyro heading is not labeled "volatile" which could result in other threads reading a stale cached value rather than the most recently read value, depending on what else is happening in other threads? The reported intermittent behavior does sound a bit like a classic "cached value not being updated frequently enough" issue.

Why would you expect that the behavior might be different in a non-linear opmode vs. a linear opmode?

[skatefriday]

What if there is "other work to perform"? Could that dramatically slow down the update rate?

No. The MR gyro implementation implements the notion of an i2c transaction queue. The only user operations that insert transactions into the queue are resetZAxisIntegrator, and calibrate. In all other cases, if the transaction queue is empty, which it will be, the driver inserts a read into the transaction queue.

All of this work is performed in the USB read/write background thread. Profiling of which shows typical 10ms cycle times.

What types of "other work" would result in this slowdown? Other sensor reads? Motor/servo access? Camera access?

If the USB read/write thread was starved and you had delays in the calling of portIsReady() you might see a problem. But in practice we have not encountered this.

It sounds like the reading of the gyro heading is done in a separate thread.

All USB read/writes are done in a separate thread. The LinearOpMode and synchronous work for non-linear opmodes does a bunch of magic to make it look like it's not a separate thread.

Is it possible that the variable that stores the gyro heading is not labeled "volatile" which could result in other threads reading a stale cached value rather than the most recently read value, depending on what else is happening in other threads? The reported intermittent behavior does sound a bit like a classic "cached value not being updated frequently enough" issue.

It is not labeled volatile. If this however were indeed the problem I would think we'd see complaints from a larger segment of the community, and that hasn't been the case.

Why would you expect that the behavior might be different in a non-linear opmode vs. a linear opmode?

They have different synchronization mechanisms. I don't expect it to be different, but it would be a useful test to run.

[Cheer4FTC]

It is not labeled volatile. If this however were indeed the problem I would think we'd see complaints from a larger segment of the community, and that hasn't been the case.

But couldn't this be a problem? Shouldn't it be volatile? I thought that volatile means "update it immediately: don't cache" while a non-volatile variable may only see its value updated in other threads when the OS/task manager got around to updating things. Maybe some folks have other background tasks running that slow down the cache updates while others do not??

[FTC4106]

Hi,

We aren't set up any longer to test this. It could be a while before we get back to it due to time pressures for the upcoming competitions and 2 big events we are hosting. We had to move on to use a different gyro, which was giving us better results. Interesting to hear someone else is seeing this, though, we though we had to have a unique problem in our setup or something.