Dynamic table creation and axis selection

[ssmith]

I've brought this up in another thread but I think it deserves it's own. I think it's really important from a usability perspective that users have more control over the 2d and 3d maps used by the software. It makes it easier to write software (no need to predict every use case) and it makes it more flexible for tuners (who usually aren't programmers) to configure what they want. To me this means several things:
1. Every axis on every table should be user selectable. You can kind of do this in TunerStudio with massive if blocks but it's a pain and feels like an afterthought as opposed to integrated into the way logging/internal variables are managed by the ECU (the variables should be structured in a tree, any input and any intermediate/computed variable should be selectablle, the same way you select variables to log or tx/rx via CAN)
2. Should be easy to change the shape of a table (more or less columns/rows, up to some limit of course)
3. Should be able to reference the output of any other table as the input for any table (extension of #1)
4. Should have a collection of do-nothing tables that users can enable for the purpose of chaining tables together (either a table used to interpolate between two others, or a table that is used to add a % to another table, or a table that is straight up added to another table (same units), or select between two tables using a digital input.

Lua can kind of be used to provide some of this, but in a way that is much less approachable by non-programmers, and in a way that's harder to make clear in a GUI - i.e. we can name tables and see that name when looking up axis variables. For example, I can name "User Table 1" as "MAP Target", then when I look up variables to use in an axis for another table, I should be able to see "User Table 1 - MAP Target" instead of just "User Table 1."

These sound like minor issues but I think it's easy to underestimate the usefulness of having this level of integration in the GUI itself. It makes it easier to see how it all ties together.

Anyways I mocked up a start of a GUI. It's written in Python / QT, and I only tested it on Linux. I had another GUI I was starting in Python / WxWidgets but I ran into too many issues with their grid/spreadsheet widget. Let me know what you think and if you find this useful as a potential start for a RusEFI specific GUI. It might be that we can get TunerStudio to add the overall feature, as it seems they're currently receptive to improvements?

[ssmith]

(sorry, fixed bug in script, if it didn't work before try again)

[AndreyB]

Hold on, are you saying that
a) dynamic table size is worth looking into in immediate future
and
b) that would require new calibration UI?

As of today I am absolutely not available to think custom calibration UI.

There are some relevant items I am interested in right now but that would be off-topic for this thread.

[ssmith]

For me the bigger issue is axis selection than dynamic table size; besides I believe TunerStudio can handle dynamic table size now if we would just implement the necessary changes in RusEFI.

Also dynamic table allocation is probably also needed as we have limited resources and need to allow different users to allocate tables to the things that are important to them.

Finally, while I recognize how far we've gotten by using TunerStudio so far, I feel like as it is it will not be capable of providing the user experience I expect. I don't know if this is affecting adoption by other users or not, or if it's just me being weird/picky. For me it comes down to
1. Convincing them to add features around CAN message assembly, table axis selection, variable tree organization, etc, or
2. Writing my own GUI.

Either one would require changes to the protocol/layout/etc. Now that TunerStudio is downloading the config over the protocol, maybe it's as simple as having a separate configuration GUI that lets you change axis and size, and then have the product emit a different config to TunerStudio so it can be used to actually tune the values in those tables. But IMO something needs to be done here.

[AndreyB]

1. Convincing them to add features around CAN message assembly, table axis selection, variable tree organization, etc, or
2. Writing my own GUI.

I wonder if there is 3. implement multi-frame CAN protocol and figure out what proper UI could be configured for our needs

See also https://github.com/rusefi/rusefi/issues/214

[nmstec]

I completely agree with the dynamic table sizes. I especially could use some with my negative throttle issue. Most manufacturers use a throttle request vs pedal request map, and we have something similar but its RPM based...? So shit like that. Could also use a bit more resolution in some maps, and way less in others

[ssmith]

I started a GitHub project for a GUI that supports dynamic table creation and axis selection.

https://github.com/racer-coder/tunedemo

As an example, I demoed using an ethanol sensor to interpolate between two VE maps, and then adding coolant temp compensation on top of that, all configured by the user in the GUI, without having to teach the EMS about it.

[AndreyB]

Hello from 2024. Now that we are in 2024 we have at least one custom board with second VE table using board_config.txt extension mechanism

Now that we finally support 2Mb flash chips as of yesterday and with all the recent custom board progress around https://github.com/rusefi/fw-custom-example more dynamic tables are probably much more realistic.

I still do not believe in replacing Tuner Studio completely but hacking around TS limitations should get 91% of the desired outcome with 9% of the cost

I can think of two pieces if anyone would be interested to undertake such a project:

* to make existing table sizes dynamic we have TS side but as is it's wasteful on rusEFI side. Well, we have two options - we can either waste when we run F7/H7, or we would need an address translation layer if we would want to be less compact.

* for completely new tables some sort of a wizard/software/web toolset to code-generate completely new tables bundled together with table-specific nice Lua functions. I guess I should make a nicer board_config.txt/board_tables.ini/board_menu.ini example with an actual table.

[mk e]

WOW!

Personally, I think the table size is more important than making the table dynamic so that would seem like step 1. 16/32 (load/rpm) is more than I've ever needed but I could see some boosted setup wanting at least 16 load points. could TS easily give 2 options of 12x24 and 24/32? or similar?

In my very humble opinion creating new wizards and more Lua is something most users would see more as bugs than features. I saw an article where a company added linux support and found it was under 5% of sales and over 90% of support time.....most people like things that are easier to use and anything that detracts from ease of use costs market share. For me personally:
1) right click, insert row is AWESOME
2) select, cut, move over, paste its tolerable.
3)Go to a wizard somewhere is already beyond my patience and ability to remember what I was doing before I've even used it. bug not feature.

As I was going through rusEFI setup the other day I had several tables that had more rows or columns than I needed or wanted for whatever it was I was doing.....but its not a problem to just fill them with something....not having them when I want them is much worse.

In engine lab the dynamic tables are not actually dynamic. When I create a table I specify the max size, fuel and spark I use 24x32, but there is another selection for rows/columns to hide and that is what the UI is changing when when the right click, add/remove feature is used. It might be worth a conversation with Phil at TS to see if that is something he'd entertain adding as it would work with any ECU as it cold be part of the tuner file rather than stored in the ECU.

Now that that is solved lets talk about how important high speed onboard logging or even better logging to a usb drive or SD card is

[AndreyB]

* for completely new tables some sort of a wizard/software/web toolset to code-generate completely new tables bundled together with table-specific nice Lua functions. I guess I should make a nicer board_config.txt/board_tables.ini/board_menu.ini example with an actual table.

here's an example of a custom table https://github.com/rusefi/rusefi/commit/e977a58a059090123331a1964e74fbf360129010

This one is for an official firmware but same thing could be done in custom firmware. Some sort of automation could make it a bit easier but next question how such custom tables would actually be used.

[mk e]

here's an example of a custom table https://github.com/rusefi/rusefi/commit/e977a58a059090123331a1964e74fbf360129010

What does it do? It doesn't appear to anything that anyone other than the core rusefi programming team cold use?

It might be worth a conversation with Phil at TS to see if that is something he'd entertain adding as it would work with any ECU as it cold be part of the tuner file rather than stored in the ECU.

I shot Phil a note (making it crystal clear I speak NO ONE but myself) asking if he would be interested in adding a hide unused rows/columns feature to TS. That seems like by far the best way to handle if Phil is willing to implement it.

[mk e]

Phil says TS already supports dynamic tables but no one created fw to use the feature, but someone from rusefi did express interest at some point. Now might be the time to have another look at what he can offer?

He's interested in the virtual dynamic table though, not yet sure if he's interested enough to do it but keep you posted.

From a user perspective it makes no difference if it's a true or virtual feature I don't think.

The right answer to custom boards is "knock yourself out" and don't give it another thought as it will consume every available project resource.....working off a team wish list is helpful, asking for team resources for a personal project not so much and you end up with 5% of users consuming 95% of resources and a stalled project. Its the open source paradox, distraction and noise often looks nearly the same as help.

[mk e]

From Phil

As long as you restrict the virtual table size to <= the firmware table on both the X & Y axis, then any interpolation wouldn't be visible to the user.

Dynamic tables on the firmware are possible by adding row & column variables in the cal space and as the associated Contant in the ini file for the firmware. It must be defined before the Z axis 2d array.
The X, Y & Z arrays are all initialized using the values of these variables for sizing the arrays.

The arrays would need to be reinitialized anytime the row or column size changes.

TunerStudio already limits table size to the available RAM and resends the entire new table after a size change.

So, the advantage of doing it firmware size is if you have 16x16 tables now, you could elect to have a 12x21 table within the same RAM space.

The piece of this I was working on, then put it on the back shelf until someone implements dynamic tables is shared RAM.
Say you have 10 dynamic sized tables, have them all share one block of RAM. Then only used tables need to take any space and if you want 1 table very large when using all RAM, you can make another table smaller.

It looks like if you just reach out to him to get details, dynamic tables within a fix table ram size is available today and blocks of ram he sounds open to doing as some as someone asks for it.

He sound pretty open to the virtual dynamic table option too (fixed in the FW but displayed as dynamic up to the fixed FW limits).

Good things come to those who ask