[info] rusEfi Scheduler Performance & Accuracy

It's all about the code!
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

mck1117 wrote:
Thu Jan 16, 2020 11:09 pm
It's significantly more area and power required for not really any real performance benefit.
The benefit is crank decoding and spark have virtually no lag and no meaningful distribution tail. Interrupts are the devil.
Last edited by mk e on Fri Jan 17, 2020 1:37 am, edited 1 time in total.
mck1117
running engine in first post
running engine in first post
Posts: 1493
Joined: Mon Jan 30, 2017 2:05 am
Location: Seattle-ish

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mck1117 »

mk e wrote:
Thu Jan 16, 2020 11:25 pm
mck1117 wrote:
Thu Jan 16, 2020 11:09 pm
It's significantly more area and power required for not really any real performance benefit.
The benefit it crank decoding and spark have virtually no lag and no meaningful distribution tail. Interrupts are the devil.
But remember, if the timestamp is captured in hardware the interrupt can be pretty lazy about reading it. Likewise, scheduling outputs today currently requires an interrupt to fire to execute it, but wouldn't if that also happened in hardware. It's possible to build a zero jitter, zero error system even when using interrupts, provided your hardware timers are arranged properly.
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

It just seems like a bad idea to pass time critical info between crazy fast and not so fast systems.....way back I learned the expression "put the aces in their places" and it's usually the right answer.
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

Forgot...first things first, figure out where the current system limits are. It certainly works under some or maybe most conditions so its absolutely worth figuring out where the limits are before making it more expensive and complicated.
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

mk e wrote:
Fri Jan 17, 2020 2:49 am
Forgot...first things first, figure out where the current system limits are. It certainly works under some or maybe most conditions so its absolutely worth figuring out where the limits are before making it more expensive and complicated.
How many channels are on the logic analyzer? 4?

I ask because I was pondering test plans last night while watching a movie with and more importantly for the kids. My thought was maybe pick a few items like RPM (I assume its calculated every tooth?), spark, spark calculation and maybe ETB pwm if 4 or add stuff like the SPI and CAN buss. For spark calculation and anything else with no external indication toggle a pin you can measure as the last step of the calculation. Then maybe pick a couple cases...like say a 8cyl, 60 tooth wheel, 2 ETB, and whatever other outputs the system has all on and run it at 3 or 4 rpm points....a point is just a point and 2 points can only make a line so everything looks linear, you need at least 3 points to produce a curve, 4 points is what we shoot for more of the time because the 4th point lets you know about curve fit correlation.

With that you'll know for sure how the system performs as the interrupt freq ups up and where it becomes a concern. It should also let you estimate about where any other setup will fall on the curve you created and that can be confirmed by taking say a more modest 4cyl with 36 tooth wheel, making an estimate of where it should fit then run it. With that you'll kind of know what applications are go to go with rusefi and which if any maybe want to hold off for a fancier version or do what ferrari did for years and run 2 ecus on 8/12 cyl engines to get the performance at higher rpm they were looking for....I remember 1st gen GM electronic ignition stuff was worthless above 5k but same period ferrari was 8 or 9k no problem.
essess
Posts: 13
Joined: Thu Aug 10, 2017 7:12 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by essess »

mck1117 wrote:
Thu Jan 16, 2020 11:28 pm
mk e wrote:
Thu Jan 16, 2020 11:25 pm
mck1117 wrote:
Thu Jan 16, 2020 11:09 pm
It's significantly more area and power required for not really any real performance benefit.
The benefit it crank decoding and spark have virtually no lag and no meaningful distribution tail. Interrupts are the devil.
But remember, if the timestamp is captured in hardware the interrupt can be pretty lazy about reading it. Likewise, scheduling outputs today currently requires an interrupt to fire to execute it, but wouldn't if that also happened in hardware. It's possible to build a zero jitter, zero error system even when using interrupts, provided your hardware timers are arranged properly.
I swear we're all somehow saying the same things?

but when I read this:
mck1117 wrote:It's possible to build a zero jitter, zero error system even when using interrupts, provided your hardware timers are arranged properly.
I know we are. Because the current solution does not do this, the whole FPGA idea now is the solution ... an external solution .. a scalable solution.

I also argure that a solution such as this makes things even easier in the long run. You can treat the fpga as an external peripheral (completely under you design control I might add) and then use any generic core to drive it. You will no longer be beholden to F4/F7/H7/ST parts ... any part with your desired interface to the fpga will do. Division of labor too.
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

essess wrote:
Sat Jan 18, 2020 3:01 pm




I swear we're all somehow saying the same things?

but when I read this:
mck1117 wrote:It's possible to build a zero jitter, zero error system even when using interrupts, provided your hardware timers are arranged properly.
I know we are. Because the current solution does not do this, the whole FPGA idea now is the solution ... an external solution .. a scalable solution.
Anyone with kids...excuse me, papa, papa excuse me, papa, mama - papa won't answer, papa, papa.....knows the issues with interrupts :lol:

Seriously though, I completely see the desire to keep everything is 1 pretty common chip. You guys have done an amazing job but realistically a zero error/lag system is not possible if its interrupt based as there will always be a tail that will growth as load increases but acceptable error/lag I think you've demonstrated and just need a little more testing to really understand the limits then decide if you're happy. You guys certainly have a lot to be proud of getting to where you are.

I have a terrible habit of hunting for "the best" or "the faster" and that leads to projects that never end :oops: . and leads to loving the thought of rusEFI with an fpga....but I honestly would LOVE to see some more data that shows the improved scheduler handles everything you think it should with an accuracy you like without needing to add any more HW to the mix. simple is good....when it works.
mck1117
running engine in first post
running engine in first post
Posts: 1493
Joined: Mon Jan 30, 2017 2:05 am
Location: Seattle-ish

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mck1117 »

mk e wrote:
Sat Jan 18, 2020 5:46 pm
but I honestly would LOVE to see some more data that shows the improved scheduler handles everything you think it should with an accuracy you like without needing to add any more HW to the mix.
Yes - more data coming. The existing improvements I made last week seemed to actually make my engine noticeably smoother - turns out reducing ignition timing jitter by an order of magnitude works wonders.

I've been looking in to where the long tail in the timing inaccuracy is coming from, and the data are pretty interesting. More on that later :D
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

mck1117 wrote:
Sat Jan 18, 2020 9:02 pm

Yes - more data coming. The existing improvements I made last week seemed to actually make my engine noticeably smoother - turns out reducing ignition timing jitter by an order of magnitude works wonders.

I've been looking in to where the long tail in the timing inaccuracy is coming from, and the data are pretty interesting. More on that later :D
A degree at idle makes a big difference, as does a a little bit of fuel, I'm not surprised the improvement is noticeable. Can't wait to see what you found!
mck1117
running engine in first post
running engine in first post
Posts: 1493
Joined: Mon Jan 30, 2017 2:05 am
Location: Seattle-ish

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mck1117 »

I've been on a hunt for where the remaining uncertainty is coming from.

Here's how scheduling ignition timing works:
  1. Edge on trigger pin
  2. Trigger interrupt occurs
  3. Timestamp captured
  4. Ignition event scheduled using timestamp from 3.
    ...then later...
  5. Timer interrupt occurs
  6. Event dequeued & executed, firing ignition
There are a few places that uncertainty can come from.
  • Trigger interrupt latency from 1 -> 2. If interrupts are disabled or the global system lock is locked, the interrupt will be postponed until interrupts are enabled or unlocked.
  • Same thing for 5 -> 6.
  • If another scheduler event is ahead of our ignition event by less than how long it takes to dequeue one event, the second event could be late.
Thankfully all of these effects will cause the edge to be late, not early.

So I started looking at the first possible source: trigger interrupt latency.

Using a second stm32, I generated a 60-2 trigger pattern at 5000 RPM. At the very beginning of the trigger interrupt, I flip an extra IO pin connected to a second channel on the logic analyzer.

Zoomed out view:

Image

Zoomed in view. We're measuring the interval from marker A1 to A2.

Image

Same deal as last time: ran for a little bit, then exported to Excel for analysis.

Here's the histogram of trigger latency, in microseconds:

Image

Note the two marked regions.

Region 1 is the "good part". It's a sort of normal-ish distribution, with the vast majority of samples inside a 0.2us range. Great.
Region 2 is the "bad part". What are all those samples doing way over there...?

Let's look at a different way of presenting the same data. This is the same data, sorted, then graphed on a line graph. If the above graph is roughly the probability distribution, this is the cumulative distribution with the same regions marked:

Image

The large flat part around 1.5us is the well distributed, predictable part at the left side of the histogram (low latency). But we can see now what's going on in region 2: A straight line in a CDF indicates a uniform distribution!

In reality the trigger latency is something like:

= N(1.45, 0.005) + U(0, 16)

The normal term is difficult to measure where it's coming from, since it's relatively narrow (and probably actually many independent impacts, hardware and software). The uniform term, however, should be easier to track down since it's relatively wide.

My best guess is that *somewhere* else in the firmware we're holding a lock for around 16us. Since this would cause a 16us period during which interrupts can't occur, interrupts will arrive uniformly distributed in this window, giving a uniformly distributed error in the range [0, 16] us.

Time to go hunt down locks!
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

First, excellent data, well done collecting it. Don't let this thought be lost when I start talking about what I think I see.


Looking at the data...there is no normal region, here is cause 1 and cause 2. I did some work years ago with 1 sided distribution (weld strength limited by the a hard stop at the wire strength)...you has a hard stop on the "it can't be faster" side cause by the processor, but the tail should not look like that and if you were to do something like a data mirror around the inflection point and exclude cause 2 data to create a full distribution is would likely still not be normal. The tail is basically a linear region not a decay as expected with normal distribution.

The question then as you say is why? You're looking at locks and probably on the right track but with a 60 tooth wheel thwe odds of a trigger tooth interfering with an fuel or spark pulse seems high as does it you are trying to do a PWM the chance that 1 of those is running when something else is due because very high. This is what I was on about with the system limits.....if you just do crank, fuel, spark you will probably get to where you're happy enough. but if you add a bunch more stuff that really has to interfere you probably won't ever be very happy.......and that in a nutshell is why OEM and other aftermarket systems use dedicated timers.

How are you keeping RPM stable? Average over 3 or 4 teeth? if not its got to be bouncing around I think.
Under your test condition do you have any idea what the non-interrupt code is doing as in how long between fuel, spark, throttle. etc calculation?
mck1117
running engine in first post
running engine in first post
Posts: 1493
Joined: Mon Jan 30, 2017 2:05 am
Location: Seattle-ish

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mck1117 »

mk e wrote:
Sun Jan 19, 2020 10:31 pm
First, excellent data, well done collecting it. Don't let this thought be lost when I start talking about what I think I see.


Looking at the data...there is no normal region, here is cause 1 and cause 2. I did some work years ago with 1 sided distribution (weld strength limited by the a hard stop at the wire strength)...you has a hard stop on the "it can't be faster" side cause by the processor, but the tail should not look like that and if you were to do something like a data mirror around the inflection point and exclude cause 2 data to create a full distribution is would likely still not be normal. The tail is basically a linear region not a decay as expected with normal distribution.
I don't mean that it's necessarily exactly normal, but it's normal-ish to within the ability of my logic analyzer to measure. Of course (barring the long linear part) it has a longer tail on the slow side than the fast side, but it is at least bell curve shaped.

Here's a zoom in on region 1:

Image

It's quantized in to only a few bins like that because those bins are spaced out by the sample rate of the logic analyzer. That distribution isn't stacked as hard against the left side because there are normal conditions that change interrupt latency, even if not fully blocked by interrupts being disabled (what instruction is running when the interrupt arrives, whether the cpu is asleep, fpu mode, etc). It's hard to tell the exact shape because it's only 5 different bins.
mk e wrote:
Sun Jan 19, 2020 10:31 pm
The question then as you say is why? You're looking at locks and probably on the right track but with a 60 tooth wheel thwe odds of a trigger tooth interfering with an fuel or spark pulse seems high as does it you are trying to do a PWM the chance that 1 of those is running when something else is due because very high. This is what I was on about with the system limits.....if you just do crank, fuel, spark you will probably get to where you're happy enough. but if you add a bunch more stuff that really has to interfere you probably won't ever be very happy.......and that in a nutshell is why OEM and other aftermarket systems use dedicated timers.

How are you keeping RPM stable? Average over 3 or 4 teeth? if not its got to be bouncing around I think.
Under your test condition do you have any idea what the non-interrupt code is doing as in how long between fuel, spark, throttle. etc calculation?
Since this measurement was just trigger latency alone, the stability doesn't really matter, so long as there's "normal operation" load on the CPU. Work being done in normal threads generally isn't a big deal, since almost none of it ever locks, and is mostly on threads that can be interrupted by any interrupt. The trigger interrupt is the highest priority interrupt we have, so it can preempt any other code running (including interrupts) so long as there isn't a global lock held.
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

mck1117 wrote:
Sun Jan 19, 2020 11:00 pm
I don't mean that it's necessarily exactly normal, but it's normal-ish to within the ability of my logic analyzer to measure. Of course (barring the long linear part) it has a longer tail on the slow side than the fast side, but it is at least bell curve shaped.

Here's a zoom in on region 1:
That part does look normal, but there is that tail that isn't and my concern is it's trying to tell you something...if fact I know its trying to tell you something.

mck1117 wrote:
Sun Jan 19, 2020 11:00 pm
Since this measurement was just trigger latency alone, the stability doesn't really matter, so long as there's "normal operation" load on the CPU.
Why? Its showing the same behavior the ignition is, at least it appears to be...and that is what you'd expect I think. Does that not impact the rpm calculation and therefore the spark timing?......it might be the cause of the tail?
mck1117 wrote:
Sun Jan 19, 2020 11:00 pm
Work being done in normal threads generally isn't a big deal, since almost none of it ever locks, and is mostly on threads that can be interrupted by any interrupt. The trigger interrupt is the highest priority interrupt we have, so it can preempt any other code running (including interrupts) so long as there isn't a global lock held.
The piece of info I was digging for on normal, not interrupt threads is how often is it completing. For sure it can be interrupted but looking at how often it completes tells you an awful lot about the processor loading. Its for sure not as critical as the timing critical stuff...but it does have to get completed on a regular basis like every 10 or 20 msec under worst case conditions. I really think its worth adding an a pin flip to track it because at some rpm it will basically no longer be supported as the interrupt based stuff consumes 100% of the processor's time. A guy I knew worked on the Apollo program and was telling me how a computer design for navigation ended up tasked with a whole lot of other things by the ime the mission occurred....and during the moon landing error codes were pouring in and communication lost because navigation consumed 100% for the computer's time and re-positioning the com antenna was one of the 1st things to go. The priority scheme you have means "normal" fuel, spark and I think throttle position are the items to be dropped and I don't think you want to ever have that happen....delay bit but not drop. Worth checking IMO.
mck1117
running engine in first post
running engine in first post
Posts: 1493
Joined: Mon Jan 30, 2017 2:05 am
Location: Seattle-ish

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mck1117 »

mk e wrote:
Mon Jan 20, 2020 12:23 am

That part does look normal, but there is that tail that isn't and my concern is it's trying to tell you something...if fact I know its trying to tell you something.
Yes, exactly. We're thinking the same thing. The normal-looking bit is fine, the tail is not.
mk e wrote:
Mon Jan 20, 2020 12:23 am
Why? Its showing the same behavior the ignition is, at least it appears to be...and that is what you'd expect I think. Does that not impact the rpm calculation and therefore the spark timing?......it might be the cause of the tail?
It's showing similar behavior to the ignition timing results because this is where at least some of the problem is coming from. This is directly measuring the time between the actual hardware trigger edge, and when we grab the timestamp that we depend on for all subsequent math (trigger decode, rpm, ign scheduling, etc). It isn't necessarily required for the trigger to be of high quality, since this measurement was just of the interrupt/timestamp acquisition latency itself.

While it's not particularly important for measuring interrupt latency, the trigger generated by my second stm32 does probably have some jitter in it. Since it's very important for measuring ignition timing accuracy/jitter, I'm going to whip up some quick verilog to generate a 60-2 from an FPGA instead, which will have essentially zero measurable jitter (on the time scales engines are concerned with).
mk e wrote:
Mon Jan 20, 2020 12:23 am

The piece of info I was digging for on normal, not interrupt threads is how often is it completing. For sure it can be interrupted but looking at how often it completes tells you an awful lot about the processor loading. Its for sure not as critical as the timing critical stuff...but it does have to get completed on a regular basis like every 10 or 20 msec under worst case conditions. I really think its worth adding an a pin flip to track it because at some rpm it will basically no longer be supported as the interrupt based stuff consumes 100% of the processor's time.
Yes - we already have this, in the form of the performance tracing. The processor loading is not bad at all. Our 200hz callback to compute things like ignition timing, fuel quantity, etc usually takes only a few hundred microseconds, even when running an engine at speed like in these tests. I can grab some specifics on that later, but the total CPU load actually isn't very high, maybe only 50%. There's tons of time in the gaps between the interrupts, so I'm not really worried about outright starvation of anything at this point.
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

mck1117 wrote:
Mon Jan 20, 2020 2:18 am

Yes - we already have this, in the form of the performance tracing. The processor loading is not bad at all. Our 200hz callback to compute things like ignition timing, fuel quantity, etc usually takes only a few hundred microseconds, even when running an engine at speed like in these tests. I can grab some specifics on that later, but the total CPU load actually isn't very high, maybe only 50%. There's tons of time in the gaps between the interrupts, so I'm not really worried about outright starvation of anything at this point.
A few hundred microsec seems way too fast. I don't thing any ECU I've seen is looping faster than about 3000-4000 microsec...so.... are you sure?

A simple pin toggle your logic analyzer can see would put it in the exact same data set as all the other performance info you're now collecting and remove all doubt..... I can't count the number of times I've been burned over the years looking at internal system data and ASSUMING the output was right. Or have toy heard the story of the statistician who was found dead with his feet in the freezer and head in the oven giving an average body temp of 70F?...a comfortable 50% average loading might be overloaded 50% of the time and idling the other....I think its worth checking to see the 200hz is about right.
mck1117
running engine in first post
running engine in first post
Posts: 1493
Joined: Mon Jan 30, 2017 2:05 am
Location: Seattle-ish

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mck1117 »

mk e wrote:
Mon Jan 20, 2020 2:40 am

A few hundred microsec seems way too fast. I don't thing any ECU I've seen is looping faster than about 3000-4000 microsec...so.... are you sure?
It's not a tight loop. We do the math in ~150us, then wait until it's time to run again, to rate-limit to 200hz.
mk e wrote:
Mon Jan 20, 2020 2:40 am
A simple pin toggle your logic analyzer can see would put it in the exact same data set as all the other performance info you're now collecting and remove all doubt.

.....

I think its worth checking to see the 200hz is about right.
The profiler got validated when I wrote it (using pin flipping) to confirm that it was measuring the right thing, and it indeed is. All the profiler does on the MCU is take timestamps when you tell it to, so it's hard to accidentally measure the wrong thing with that.
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

mck1117 wrote:
Mon Jan 20, 2020 4:37 am

It's not a tight loop. We do the math in ~150us, then wait until it's time to run again, to rate-limit to 200hz.
That is very fast. Is that just the calcs or does it include updating the sensor values, comms, and everything else while its being interrupted with time critical stuff?

That is much faster than my enginelab stuff running on a PC based simulator and like 20-30x faster than it runs on the real HW, which is 200Mhz, 32 bit so similar to yours. I know enginelab has more background "stuff" going on and I wasn't shy about adding tables for anything that popped into my head but still a big difference. Also much faster than the motec M1 stuff, haltech and adaptronic (they used to advertise 3.5msec loops so they thought it good) which are all 32 bit and I have at least rough data for so it leaves me confuses and I suspect we are not comparing like for like.

If its truly just 150usec and you have a 50% processor utilization then you should be able to up the run rate to say 3000hz...have you tried anything that as another way to confirm? The numbers just sound to good to be real, which doesn't mean they're not but they are much better then I'm doing...I have 8 threads in my setup none of which run in 150usec in the ecu HW which is why that number seems too fast.

edit - I'm pretty sure the enginelab is a renesas SH725X Series...pretty sure. I suspect they are working on a new gen with the new 320Mhz RH850
puff
contributor
contributor
Posts: 2961
Joined: Mon Nov 11, 2013 11:28 am
Location: Moskau

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by puff »

do they have fpu? :-)
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

puff wrote:
Mon Jan 20, 2020 3:49 pm
do they have fpu? :-)
of course....along with all the other stuff MCU chips normally have...built in watchdog, AN signals read, converted, moved to memory with no CPU action needed, buckets of timers in a time processing unit...purpose built for MCU use hence my confusion. The freescale MPC5xxx chips I used to play with were similar and with those chip Floats were actually faster than binary because they were in the fpu.
https://www.renesas.com/us/en/doc/products/mpumcu/doc/superh/r01uh0344ej0300_sh7256.pdf?key=ed4983fa2d8ca6b2dec81fc6b9d4d29e
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

Here are my thread times running in the simulator on a PC.....I have a timer on each thread just cause....I'll try to load it into the HW and show the difference and clearly I need to fix the min. Times are msec.
Attachments
thread times Capture.JPG
thread times Capture.JPG (249.62 KiB) Viewed 23753 times
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

Here's the thread times on the real HW for comparison. Thread 2 is the main fuel/spark calcs and I lied, the units are seconds. It's a simple dt, so time since last run and the thread is on a 7ms sleep. so actual run time is about (forgot to check actual sleep times) 9.64msec-7msec = 2.64msec on the real HW. The HW is faster than I was thinking at 1/3 the speed as the PC simulator. Anyway this is where my confusion with the 150usec came from....its REALLY fast compared to other ECUs.
Thread HW Capture.JPG
Thread HW Capture.JPG (98.98 KiB) Viewed 23736 times
Here are the thread sleep times in msec. Thread 6 with a 999msec sleep would be stuff like coolant temp, thread 1 with a 2msec sleep is cylinder MAP but is run every 5msec it looks like
Thread sleep Capture.JPG
Thread sleep Capture.JPG (49.29 KiB) Viewed 23736 times
mck1117
running engine in first post
running engine in first post
Posts: 1493
Joined: Mon Jan 30, 2017 2:05 am
Location: Seattle-ish

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mck1117 »

What ECU is that?

Remember that we have many threads (on the order of 15) that are all completely independent. Most of them are rate limited, but a few (Tunerstudio, for example) are not. So if TS isn't connected it uses exactly zero cycles.

A bunch of stuff that might be on the same thread for you is running on different threads for us:
  • ADC conversion
  • USB/serial tunerstudio IO
  • Electronic throttle
  • Idle
  • Idle stepper (yes, separate from normal idle control)
  • Blinking LEDs
  • Onboard LCD
  • Alternator controller
  • Bench tests
  • CAN (one for RX, one for TX)
For example, ADC conversion is on a pretty high priority thread, but that thread is waiting while doing the actual conversion, so other threads get scheduled at the same time.

As you've asked of me, I'd like to know exactly what those times are measuring. Is that actual time that the thread was scheduled (self time), or how long the task took to complete, including the amount of time it was preempted (wall clock time)?
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

mck1117 wrote:
Wed Jan 22, 2020 2:38 am
What ECU is that?

Remember that we have many threads (on the order of 15) that are all completely independent. Most of them are rate limited, but a few (Tunerstudio, for example) are not. So if TS isn't connected it uses exactly zero cycles.

A bunch of stuff that might be on the same thread for you is running on different threads for us:
  • ADC conversion
  • USB/serial tunerstudio IO
  • Electronic throttle
  • Idle
  • Idle stepper (yes, separate from normal idle control)
  • Blinking LEDs
  • Onboard LCD
  • Alternator controller
  • Bench tests
  • CAN (one for RX, one for TX)
For example, ADC conversion is on a pretty high priority thread, but that thread is waiting while doing the actual conversion, so other threads get scheduled at the same time.

As you've asked of me, I'd like to know exactly what those times are measuring. Is that actual time that the thread was scheduled (self time), or how long the task took to complete, including the amount of time it was preempted (wall clock time)?
Its an enginelab, same as AEM but the "developer version" that AEM buys and loads there control logic into (sort of programming but limited to library functions basically). I'm pretty sure it uses this chip
https://www.renesas.com/en-us/products/microcontrollers-microprocessors/superh/sh7250/sh7256r.html

32bit, 200MHz

None of the listed items are on the thread....it is Fuel (edit spark is thread 5) calculation. Wall-wetting is in the thread, but calculation of cylinder trims is not, its thread 3.

The OS doesn't run items on a clock, it puts items to sleep on a clock. Thread 2 sleeps 7msec then wakes and is Que'd to run (all threads are currently set to the same priority)

The time is a simple counter function- ""dt" – The time delta dt. This is the time between successive computations of this math item."
https://enginelab.net/documentation/#Math_Calculation

I added a dt to all the threads along with a running average and min/max....min is always 0 from startup but I haven't fixed it. Any this tracks time between runs, subtract up the sleep time (7msec) and that is whats is taking to process including any time waiting it Que...which to me as a system guy not a programmer is most important. I guess I could add a timer to the thread itself which would clear up how long it waits in Que.....I'll have to thing about that because as I said its library based so easy in C is not always easy.

On MCU type chips like this stuff like ADC is hardwired into the chip and uses no processor time and time critical stuff is in the TPU where the timers are, or so I'm told, again I'm not a programmer so just repeating what I've been told. But that means it really doesn't matter to the main processor what RPM or how many cylinders the load is the same, its the co-processors that see that increased load.
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

mk e wrote:
Wed Jan 22, 2020 12:37 pm
I guess I could add a timer to the thread itself which would clear up how long it waits in Que.....I'll have to thin(k) about that because as I said its library based so easy in C is not always easy.
I looked at it a little and so far haven't come up with anything as I only have access at the upper logic level and there are some restriction like I can only 1 function can set a variable value so I can't start a timer at the beginning and then read/reset it at the end. I could just load the fuel model and set the sleep time to 0 maybe, it would take a bit of work to do something with all the links to info from other threads....but its possible I think....maybe.
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

mk e wrote:
Wed Jan 22, 2020 2:53 pm
mk e wrote:
Wed Jan 22, 2020 12:37 pm
I guess I could add a timer to the thread itself which would clear up how long it waits in Que.....I'll have to thin(k) about that because as I said its library based so easy in C is not always easy.
I looked at it a little and so far haven't come up with anything as I only have access at the upper logic level and there are some restriction like I can only 1 function can set a variable value so I can't start a timer at the beginning and then read/reset it at the end. I could just load the fuel model and set the sleep time to 0 maybe, it would take a bit of work to do something with all the links to info from other threads....but its possible I think....maybe.
I think that worked mostly. Min thread sleep time is 1msec so on the PC with only the fuel thread loaded I get 2.174msc minus the 1msec sleep is really 1.174msec on the PC which is a little less from the full model running which was 8.57-7 = 1.57msec, real HW would be a bit slower.
Attachments
ThreadTime PC fuel only.JPG
ThreadTime PC fuel only.JPG (19.47 KiB) Viewed 23693 times
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

I may have more going in what I call fuel calculations and the you do. This is the fuel script that assembles the subroutines I call (with there inputs listed) I think this is all fuel other than the .\Cylinder_Trim.elcfg item and remember its a higher level language that best I can tell is just calling library functions so it doesn't look like C

Something like .\Gasoline_Constants.elcfg just defines variables at startup and is not part of the thread then the next item .\Load.elcfg is just a simple 1 line calculation, but then .\MAF.elcfg output the mass air flow in G/sec so a bit going on there. Anyway the fuel thread basically creates all info needed to fire the injectors but actually firing them is done in the TPU and as I said sensors values are just used other than unit conversions

Code: Select all

#////////
#Fuel Control
#////////

{if defined(FUEL)
.\Gasoline_Constants.elcfg
                #Basic physical propereties of gasoline preset to normal values but can be changed by user
                #Fuel_Molar_Mass [g/mol] - 98.05
                #Fuel_Stoich_Ratio - 14.7
                #Fuel_Density [kg/m3] - 745
                #Fuel_Density_Tep_Coef [kg/m3/C] - 0.9
                #Fuel_Cooling [C] - 13.5, this is the amount of inlet air temp cooling due to fuel vaporization
		#Fuel_Heat_of_Combustion [kJ/g] - 47

.\Load.elcfg
                #Creates a normalized MAP value to separate manifold pressure changes due to barometric pressure from pressure changes due to throttle position
                #Load [%] - MAP/Baro


.\MAF.elcfg
                #Calculates the Mass Air Flow using the predefined speed density function.
                #VE [%] - user lookup table for VE
                #R [J/g*degK]
                #IAT_Cooled [K] applied the fuel cooling effect (Fuel_Cooling [C]) to the IAT
                #Delta_CLT_IAT [C] - temp difference between CLT and IAT
                #CLT_IAT_Bias - user lookup to set amount or CLT ot use in MAF calculation.....I'm not sure this is useful but it's required by the Speed Density function
                #MAF [g/s] - Mass air Follow in [g/s] 
 

.\Cylinder_Base_Fuel.elcfg
                #Calculates the per cylinder fuel required in [g]
                #Lambda_Target - user lookup for desired lambda vs load and rpm
                #Air/Fuel_Ratio - Lambda_Target x Fuel_Stoich_Ratio
                #Cylinder_Base_Fuel [g] - required fuel per cylinder per cycle in [g] used as reference in wall wetting 


.\Start_Enrich.elcfg
		#calculates a fuel correction to use during cranking and post start.  inputes are cycles and CLT, output a correction factor 1=no correction
                #Start_Enrich_Enable
                #Start_Enrich [%] - user lookup for extra fuel required vs temp and cycles
		#Start_Enrich


.\CLT_Fuel_Corr.elcfg
		#calculates a fuel correction to use during warmup.  inputsa are CLT and RPM, output a correction factor 1=0 no correction
                #CLT_Fuel_Enable
                #CLT_Fuel_Correction [%] - user lookup for extra fuel required vs temp
                #CLT_Fuel_Corr - if enabled use corr, if not set value to 0

.\Flood_Clear.elcfg
                #Routine to cut fuel during cranking if throttle is full open to clear flood - 100msec.  Inputs State_Engine and TPS [%], outputs a 0 or 1 correction faction, 1= no corre4ction, 0=cut fuel
                #Flood_Clear_Enable - input to enable feature
                #Flood_Clear - inverts test resuls so 0 = flooded and 1 = not flooded which will set Fuel_Req to 0 when flooded


.\lambda_Fuel_Error.elcfg
                #Calculates the difference between lambda and lambda target in [g] and as a ratio - 50msec
                #Lambda_Error_Bank1 [%]
                #Lambda_Error_Bank2 [%]
                #Fuel_Error_Bank1 [mg]  - difference in [g] between desired lambda and actual lambda
                #Fuel_Error_Bank2 [mg]  - difference in [g] between desired lambda and actual lambda


.\Enable_Injectors.elcfg
		#Enable_Injectors
		#Injector_On

.\Disable_Banks.elcfg
		#creates a trim factor that that is used to set cylinder trim for the bank to 0% shutting off fuel.
		#Disable Bank 1
		#Disable Bank 2


.\Wall_Wetting.elcfg
                #Determines if wetting (accel/decel) fuel corrections should be enabled. - 10msec

                #Fuel_Film_Enable - input to enable feature
                #Fuel_Wetting_Enbale_Time [s] - run time before system enables
                #Fuel_Wetting_Enable_Test - if enable and enable time contions met = 1, else 0
                #Fuel_Wetting_Cycle_Last


                #Fuel_Stored_Base [mg] - fuel stored in manifold vs MAP and RPM
                #Fuel_Stored_CLT_Corr_Enable
                #Fuel_Stored_CLT_Corr [%] - CLT correction for stored fuel

                #Fuel_Stored_IAT_Corr_Enable
                #Fuel_Stored_IAT_Corr [%] - IAT correction for stored fuel

                #Fuel_Stored_Target [g] - Fuel_Stored_base* 0.001 * (1+Fuel_Stored_CLT_Corr [%]*0.01*Fuel_Stored_CLT_Corr_Enable_Clamp) * (1+Fuel_Stored_IAT_Corr [%]*0.01*Fuel_Stored_IAT_Corr_Enable_Clamp)*(1+Fuel_Stored_Angle_Corr [%]*.01*Fuel_Stored_Angle_Corr_Enable_Clamp)

                #Fuel_Stored_Delta [mg] - fuel stored_Target - Fuel_Stored_Current
                #Fuel_Stored_Correction [%] - % correction to make based on Fuel_Stored_Delta and RPM
                #Fuel_Stored Correction [g]
		#Fuel_Stored_Unchanged

                #This is the running total of fuel that is currently stored in each intake path in [g].  The stored fuel is set to the target prior to system becoming active.
                #Fuel_Stored [g] - running total of fuel stored
                #Fuel_Stored_Current [g]  - select Fuel_stored or Fuel_stored_Target to set as current.

		#MAF_Cycle [g/cycle] - 
		#Fuel_Flow [g/cycle]
		#Fuel_Stored_Trim_Calculation
		#Fuel_Stored_Trim_Correction_Test
		#Fuel_Stored_Trim_Correction
		#Fuel_Stored_Trim_Correction_Last


.\Cylinder_MAP_Trim.elcfg
		#Uses Map_Cyl_Error to create a cyinder trim factor
		#Cylinder_MAP_Trim_Enable
		#Cyl_1_MAP_Trim

.\Cylinder_Trim.elcfg
		#
		#Cylinder_Fuel_Trim Enable
		#Cylinder_1_Fuel_Trim [%] - inputs RPM and load

		#Cylinder_1_Trim_Factor

		#FI_Trim_1 - combines table and Map trim factors for LSFS
		#FI_Trim_2
		#FI_Trim_3
		#FI_Trim_4
		#FI_Trim_5
		#FI_Trim_6
		#FI_Trim_7
		#FI_Trim_8
		#FI_Trim_9
		#FI_Trim_10
		#FI_Trim_11
		#FI_Trim_12

.\Fuel_Trim.elcfg

		#Fuel_Trim_Enable
		#Fuel_Trim [%]
		#Fuel_Trim_Factor
		#Global_Trim_Factor


.\Injector_Time.elcfg
                #convert grams of fuel into pulse width - 10msec

                #Enable_Injectors

                #Injector_On - 1 if Enable_Injectors = 1 and rpm > 1 and rpm < redline


                #Differential_Fuel_Pres [kPa] - Fuel Pressure ralative to MAP [kPa]

                #Primary_Flow_Rate [g/s] - injector flow rate vs differentail pressure in [g/s]
		#Secondary_Flow_Rate [g/s]

                #Primary_Offset [msec] - the offset of "deadtime" for the injector vs differentail fuel pressure vs battery voltage
		#Secondary_Offset [msec]
		
		#Injection_Timing [degBTDC]
                #FI_TimingX [degBTDC] - reguired variable, user lookup table for injection end time


		#pri_duty_threshold [%]
		#staged_split [%]
		#pri_duty [%]
		#sec_duty [%]
		#flex_stoic_ratio

		#Injection_Timing [degBTDC]
		#FI_TimingX [degBTDC]

		#FI_Nonlinear_PriPulsewidth [ms]
		#FI_Nonlinear_SecPulsewidth [ms]
                #Primary_Short_Pulse_Adder [msec] - user lookup for short pulse correction 
                #Secondary_Short_Pulse_Adder [msec]
		#FI_Nonlinear_PriTrim_add
		#FI_Nonlinear_SecTrim_add


		#LSFS

                        #Cylinder fuel time for EAL
                #FI_Fuel1 [ms] - Req variable for fuel in [msec]
                #FI_Fuel2 [ms]
                #FI_Fuel3 [ms]
                #FI_Fuel4 [ms]
                #FI_Fuel5 [ms]
                #FI_Fuel6 [ms]
                #FI_Fuel7 [ms]
                #FI_Fuel8 [ms]
                #FI_Fuel9 [ms]
                #FI_Fuel10 [ms]
                #FI_Fuel11 [ms]
                #FI_Fuel12 [ms]



#LAMBDA_TARGET - (MAF/Req_Fuel)/Fuel_Stoic.....Air/Fuel_Ratio
#PRIMARY_DUTY_THRESHOLD [%]
#STAGED_SPLIT_PERCENT [%]

#PRIMARY_FLOWRATE [cc/s] - flow [g/s]/density [kg/m3]
#SECONDARY_FLOWRATE [cc/s]
#TRIM_MULTIPLIER_GLOBAL
#TRIM_PRI_ADDER_GLOBAL [ms]

#TRIM_PRI_ADDER_GLOBAL [ms] - offest+short+pulse+adder


#OUTPUT_PRIMARY_DUTY [%]
#OUTPUT_SECONDARY_DUTY [%]
#FLEX_FUEL_STOICH_AF


.\Fuel_basic.elcfg
                #Values required by EAL for lowZ injectors, not used in this project
                #FI_DutyCycX %high: Duty cycle during PWM after the injector is open. (Applies to peak and hold). Set to 100 for high impedance injectors.
                #FI_CyclTimeX us/10: Cycle time during PWM after injector is open. (Applies to peak and hold). Set to anything >= 100 for high impedance injectors.
                #FI_OpenTimeX us/10: Injector open time. PWM will proceed following this time setting.


}
once the script runs the fuel stuff is about 195 lines the look like this
modelCapture.JPG
modelCapture.JPG (135.58 KiB) Viewed 23682 times
Then something like the MAF item looks like this (if it were running the "current value" line wold show the output of each line in real time and can be logged or displayed or plotted realtime to help with debugging or trouble shooting....so someone like me who makes mostly mistakes its pretty helpful)
MAFCapture.JPG
MAFCapture.JPG (47.88 KiB) Viewed 23682 times
and what the function is doing is here
https://enginelab.net/documentation/#Speed_Density

as an example....so certainly not C and there is no compile, it can be changed on the fly in the sim or HW but they say good practice is to make the changes outside and reload so everything says documented....and that certainly could all make it slower although I don't think a lot slower as it does seem to be a map to a series of library function calls that are done in the "firmware" and this stuff is just called the "model". The new M1 motec stuff does something similar with what they call a "package" that looks a little more like C pseudo code when you dig into them, again loaded with no compiling so for sure its not loading actual programming.

Anyway, that's my fuel function that takes a couple msec to run.
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

Where did you guys ever end up with the scheduler work? Decide the improvements that were made got it where you were happy?
User avatar
AndreyB
Site Admin
Posts: 14292
Joined: Wed Aug 28, 2013 1:28 am
Location: Jersey City
Github Username: rusefillc
Slack: Andrey B

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by AndreyB »

Funny that you've asked today since we are working on a yet another improvement today, after being happy with scheduler for a while :)
Very limited telepathic abilities - please post logs & tunes where appropriate - http://rusefi.com/s/questions

Always looking for C/C++/Java/PHP developers! Please help us see https://rusefi.com/s/howtocontribute
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

Awesome....perfect is never done :)

Someone was asking me about o5e stuff today and I suggest he head over here....which reminded me of the one thing I wasn't sure is better than o5e, timing accuracy. Can't wait to see where you get it.
mk e
Posts: 486
Joined: Tue Dec 06, 2016 7:32 pm

Re: [info] rusEfi Scheduler Performance & Accuracy

Post by mk e »

Got curious this morning and found some speeduino accuracy test data
https://speeduino.com/forum/viewtopic.php?f=13&t=223

At 13000rpm, 3.5 degree worst case and 1 degree typical error. 1/2 that at 1/2 the rpm

Some one else commented that the bosch ecus are 0.75 degree at 12000...which seems high honestly but still 5x better then the speeduino.

Comparison points if nothing else.
Post Reply