Putting together a 2024 Roadmap

[rocky]

Last year, in response to suggestions from @Li-Xiang-Ideal and @mmatera I said I would start a discussion on the 2024 Roadmap.

It is 2024, so let us begin!

When the discussion settles, this will go into FUTURE.rst and can be updated and can go in the next Mathics3 release.

In the Short Term

First, some very short-term backlogged and smaller goals before the broader and bigger picture. I am hopeful that the short-term goals can be achieved in a relatively short period.

We have not had a major general release since the 6.0.0 release at the end of February 2023. Minor releases since that have been small bug fixes to assist downstream packager of OS distributions.

While I had hoped more would occur in 2023 in the way of the necessary breaking API changes, and minimizing future API-breaking changes, nevertheless a lot has happened in 2023 year.

So unless there are objections, I think we should declare a feature freeze and focus on releasing what we have in place now. This is not trivial because we need to go over not just the Mathics core, but all the other Mathics3 modules and front ends, e.g. Mathics Django and mathicsscript. They are all are impacted by the API change. After that is done, I imagine there will be yet another major release because there are more API changes that are in the works. Namely, API changes in how may Mathics3 modules work in the future to make these more seamless.

In advance of a release, my practice is to get out a new interim docker image. This allows me to experiment with building the documentation and trying to update the package while also allowing people to try things and give us feedback without having to build everything.

From 2023 Roadmap

For more details on the items, see the links in this section header. Here, I am going to just list the items with some small notes.

Forms, Boxing, and Formatting

It feels like this should be doable, or at least we start on this path. I believe we should be able to prototype in Mathics3, but I am open to making small changes in the code. I think something like a change to the implementation of MakeBoxes was requested where we need to be looking at "upvalues" instead of "ourvalues" or maybe it is the other way around. I do however believe there is a way where we can prototype this before committing to a full implementation.

Performance

This is a more nebulous overall and needs to be broken down more. There are some shorter-term and more measurable steps you will below and elsewhere.

Initial Startup Time

With the work done in 2023 to allow mathics.builtin modules to be loaded deterministically through a "load" function, we can probably reduce startup time by as much as 2 seconds or so. And at the same time, addresses problems that would be caused by scaling the number of built-in functions up many times.

More Custom kinds of (compound) Expressions and Fewer Conversions

Python/Numpy literal expressions fall into this category and this is especially useful in data crunching and rendering graphics.

Compilation

With the work done in 2023 towards providing access to instruction-like procedure calls, we probably are in a better position to handle smarter compilation, i.e. something other than turning it into Cython, and hope that Cython and figure out how to speed up what we don't have a handle on.

Further Code Reorganization in Core and Eval

Core object like BaseElement and possibly Symbol, (and
probably others) are still too "fat": they have too many custom methods that
are not applicable for most of the subclasses support. It is likely
another pass will be made over this.

We have started moving "eval" code out of the "eval" methods and into its own module.

Deferred

I hope Pattern Matching and rewrite rules will be started sometime in 2024. Otherwise, not much to say here.

Jupyter integration depends on getting Boxing under control.

Migration of the documentation system to Sphinx

This is important but probably less important than performance, boxing & formatting, and pattern matching.

And in the interim, we will probably be making more smaller fixes. Sigh. I am optimistic that the smaller fixes can work in the direction of making moving out easier. Modularity here.

New Work

Sparse Array Implementation

Li-Xiang-Ideal mentioned this. Perhaps he can elaborate.

Debugger

I, Rocky, Plan on starting to write a debugger for Mathics3. I need something like this to help understand code written in WMA and Mathics3. TracePrint[] may help, but if the past is prologue, something better than that would be nice and not that difficult. This will also be the first time since I joined the project 3 years ago, that I'll be able to do something I have wanted as opposed to tracking existing technical debt (which which we still have a bit of).

I believe this can be done largely outside of Mathics-core (although it heavily relies on Mathics-Core internals.

Operator Precedence Tables

This pulls out of Mathics Core any knowledge of Operator Precedence and instead uses tables found in mathics-scanner.
This would be based on Robert Jacobson's work.

Miscellaneous Small Things

• Revise to support Python 3.12 (may require upgrading packaging) along with dropping Python 3.7 support
• Possible Miscellaneous updates in packages and git commit hooks
• Be on the lookout for Open-Source WMA Packages like KnotTheory or Rubi that can be run in Mathics3

Individual Leads or Responsibilities

Rocky

• Debugger
• Compilation
• Literal Expressions for performance (e.g. data processing and graphics rendering)
• Mathics Autoloading (with mmatera)
• Operator Precedence (with mmatera)
• Releases (with mmatera)

mmatera

• Forms, Boxing and Formatting

Li-Xiang-Ideal

• Sparse Array Implementation

[mmatera]

Last year, in response to suggestions from @Li-Xiang-Ideal and @mmatera I said I would start a discussion on the 2024 Roadmap.

It is 2024, so let us begin!

Great!

As there are several points here, I will split my comments on each section.

In the Short Term

Completely agree. I tested most of the Pymathics modules and they are now compatible with the current master branch of mathics-core. The front ends still need some small adjustments. Also, I looked over the current PRs in mathics-core and I didn´t find anything critical that must be included in the 7.0 release, so freezing master in the current state is OK for me.

[mmatera]

From 2023 Roadmap

For more details on the items, see the the links in this section header. Here, I am going to just list the items with some small notes.

Forms, Boxing and Formatting

Regarding this, to make Mathics compatible with how WMA handles format, we should invert how Format and Makeboxes rules are applied, meaning modifying mathics.eval.makeboxes.format_element and how Set handles MakeBoxes and Format in the lhs . In any case, before working out that, something that I would like to implement is the converters from expressions to InputForm (1D format, like FullForm, but using {} for lists and infix/prefix/sufix form for operators ) and OutputForm (2D format, like in the SymPy pretty print). I have this work partially done in a private branch, and I can put this as an external module for testing when it is closer to being completed.

Performance

This is a more nebulous overall and needs to be broken down more. There are some shorter-term and more measurable steps you will below and elsewhere.

Initial Startup Time

With the work done in 2023 to allow mathics.builtin modules to be loaded deterministically through a "load" function, we can probably reduce startup time by as much as 2 seconds or so. And at the same time, address problems that would be caused by scaling the number of builtin-function up many time.

To have this completed would be great. Also, I would like to improve the interface for the Pymathics modules, in a way that more specialized built-in sections be moved outside the core. In particular, the part of Image handling (which depends on special libraries) be moved outside Mathics core, making the setup and the CI faster.

More Custom kinds of (compound) Expressions and Fewer Conversions

Python/Numpy literal expressions fall into this category and this is especially useful in data crunching nad rendering graphics.

Compilation

With the work done 2023 towards providing access to instruction-like procedure calls, we probably are in a better position to handle smarter compilation, i.e. something other than turn it into Cython and hope that Cython and figure out how to speed up what we don't have a handle on.

This would be also great.

Further Code Reorganization in Core and Eval

Core object like BaseElement and possibly Symbol, (and probably others) are still too "fat": they have too many custom methods that are not applicable for most of the subclasses support. It is likely another pass will be made over this.

We have started moving "eval" code out of the "eval" methods and into its own module.

Deferred

I hope Pattern Matching and rewrite rules will be started sometime in 2024. Otherwise not much to say here.

Something that at some point could help to make the evaluation faster would be to improve the pattern-matching mechanism. WMA has a kind of pattern compiler that makes the pattern-matching loop faster. See https://reference.wolfram.com/language/ref/Dispatch.html

[mmatera]

New Work

Sparse Array Implementation

Li-Xiang-Ideal mentioned this. Perhaps he can elaborate.

In my opinion, this should be implemented as an external module. Here https://github.com/Mathics3/pymathics-sparse I started a module that overwrites the basic one contained in Mathics-core. If the idea is to use Scipy for the low-level implementation, it is better to do it outside Mathics-core to avoid requiring a large library dependency.

Debugger

I, Rocky, Plan on starting to write a debugger for Mathics3. I need something like this to help Understand code written in WMA and Mathics3. TracePrint[] may help, but if past is prologue, something better than that would be nice and not that difficult. This will also be the first time since I joined the project 3 years ago, that I'll be able to do something I have wanted as opposed to tracking existing technical debt (which which we still have a bit of).

I believe this can be done largely outside of Mathics-core (although it heavily relies on Mathics-Core internals.

A previous step to implement Trace and TracePrint would be to implement Stack (https://reference.wolfram.com/language/ref/Stack.html?q=Stack). With it, and a part of the code that we already included in the implementation of TraceEvaluation, I think that at least a basic implementation of Trace and TracePrint is doable.

Operator Precedence Tables

This pulls out of Mathics Core any knowledge of Operator Precedence and instead uses tables found in mathics-scanner. This would be based on Robert Jacobson's work.

Miscellaneous Small Things

* Revise to support Python 3.12 (may require upgrading packaging) along with dropping Python 3.7 support

* Possible Miscellaneous updates in packages and git commit hooks

* Be on the lookout for Open-Source WMA Packages like KnotTheory or Rubi that can be run in Mathics3

• See if it is possible to migrate the home-brew documentation system to something more standard like Sphinx. A previous step would be to implement some translators from the current docstring format to a more standard Markdown format, which would help in the migration.

Individual Leads or Responsibilities

Rocky

* Debugger

I can help with this.

* Literal Expressions for performance (e.g. data processing and graphics rendering)

* Mathics Autoloading

* Operator Precedence

* Releases

mmatera

* Forms, Boxing and Formatting

Agree. Also,
* Migration of the documentation system?

Li-Xiang-Ideal

* Sparse Array Implementation

I could also help with this.

[rocky]

• Migration of the documentation system?

Yes, but realistically this seems to be lesser important than performance, boxing & formating, and pattern matching and I doubt we will deal with these sufficiently. I will add this as a deferred item.

And in the interim I will probably be making more smaller fixes. Sigh. I am hopeful that the smaller fixes can work in the direction of making moving out easier. Modularity here -- separating running doctests, from LaTeX formatting and HTML/ajax formatting -- has been a mess.

[mmatera]

• Migration of the documentation system?

Yes, but realistically this seems to be lesser important than performance, boxing & formating, and pattern matching and I doubt we will deal with these sufficiently. I will add this as a deferred item.

I know that this is less important, but it is something that can be done from outside, without compromising the code, at least in the step of writing a translator back-and-forward to markdown.

And in the interim I will probably be making more smaller fixes. Sigh. (I am optimistic that the smaller fixes can work in the direction of making moving out easier. Modularity here -- separating running doctests, from LaTeX formatting and HTML/ajax formatting -- has been a mess.

This would also be useful.

[rocky]

I know that this is less important, but it is something that can be done from outside, without compromising the code, at least in the step of writing a translator back-and-forward to markdown.

Ok - if you want to make experiments, prototype or beyond, please do.

[rocky]

I can help with this.- [Literal Expressions, Precedence, Autoloading, Releases]

Great - thanks! I have updated the top-level entry. (This will be copied into FUTURE.rst with a link back to this discussion page.)

[aravindh-krishnamoorthy]

Sorry, this update is really near the end of 2024. Nevertheless, here it goes.

Jupyter integration depends on getting Boxing under control.

Running Mathics on Jupyter is really important for me. However, I only need text output + 2D graphics. For this, I took @mmatera's old code, stripped off the support for things other than Mathics, and piped 2D graphics via SVG here: https://github.com/aravindh-krishnamoorthy/imathics. The changes by me are very basic, but for now Mathics works in JupyterLab/JupyterLab Desktop and that's good enough.

I've also started to fix related issues in Jupyter, see, e.g., jupyter/nbconvert#2190, which allows export of SVG 2D graphics. After the fork is stable, I will start a discussion about merging it back here. However, I'm not an expert on Jupyter by far.

• Be on the lookout for Open-Source WMA Packages like KnotTheory or Rubi that can be run in Mathics3

Yesterday, I tried importing this out-of-the-box version of 'Rubi.m' into Mathics (In[1]:= << Rubi.m) with the following minor changes to Rubi.m:

1. Directory[] instead of DirectoryName[System`Private`$InputFileName].
2. $LoadShowSteps = False set.

Rubi loads under a minute if the following rules are NOT loaded:

1. Whole of "3 Logarithms";
2. Whole of "5.3 Inverse tangent";
3. Whole of "7.3 Inverse hyperbolic tangent";
4. Whole of "8.8 Polylogarithm function";

all of which seem to never finish loading (waited about 30 mins each). Interestingly, they all had to do with logarithms and tangents for some reason. I also had strange errors during parsing:

Part::partd: Part specification is longer than depth of object.

which seemed to happen because _list_parts() in mathics/eval/parts.py was called on <Symbol: RubiPrivateSimplifyFlag>, an atom.

Nevertheless, after loading, I was able to get a few integrations done successfully with Int[]. I'll try to get as much as I can done and come back here and start a discussion for this too.

[rocky]

Sorry, this update is really near the end of 2024. Nevertheless, here it goes.

No problem. It is better to get feedback whenever it comes than the default no feedback :-)

Jupyter integration depends on getting Boxing under control.

Running Mathics on Jupyter is really important for me. However, I only need text output + 2D graphics. For this, I took @mmatera's old code, stripped off the support for things other than Mathics, and piped 2D graphics via SVG here: https://github.com/aravindh-krishnamoorthy/imathics. The changes by me are very basic, but for now Mathics works in JupyterLab/JupyterLab Desktop and that's good enough.

I cloned the repository. There is a step in the instructions that I needed to add, namely pip install jupyter and/or pip install notebook. But running this I don't seem to be inside a running mathics_kernel. I didn't have time to investigate much further.

I've also started to fix related issues in Jupyter, see, e.g., jupyter/nbconvert#2190, which allows export of SVG 2D graphics. After the fork is stable, I will start a discussion about merging it back here. However, I'm not an expert on Jupyter by far.

You know more about it than I do, (I still haven't kept up to date with IPython). So as far as I am concerned you are more of an expert than I.

• Be on the lookout for Open-Source WMA Packages like KnotTheory or Rubi that can be run in Mathics3

Yesterday, I tried importing this out-of-the-box version of 'Rubi.m' into Mathics (In[1]:= << Rubi.m) with the following minor changes to Rubi.m:

1. Directory[] instead of DirectoryName[System`Private`$InputFileName].
2. $LoadShowSteps = False set.

Rubi loads under a minute if the following rules are NOT loaded:

1. Whole of "3 Logarithms";
2. Whole of "5.3 Inverse tangent";
3. Whole of "7.3 Inverse hyperbolic tangent";
4. Whole of "8.8 Polylogarithm function";

all of which seem to never finish loading (waited about 30 mins each). Interestingly, they all had to do with logarithms and tangents for some reason. I also had strange errors during parsing:

Part::partd: Part specification is longer than depth of object.

which seemed to happen because _list_parts() in mathics/eval/parts.py was called on <Symbol: RubiPrivateSimplifyFlag>, an atom.

If you can give a small reproducible example, I bet we can get this fixed.

Part of the problem with getting any package working has been a lack of being able to understand how to isolate problems. So I am grateful for you showing how to isolate parts of this package.

BTW, recently we added tracing to Get, so Get["Rubi.m", Trace->True] will show the lines as they are executed. What surprises me here is that there is a lot of time outside of reading the basic rules, such as on line 371 "FixIntRules[]".

Nevertheless, after loading, I was able to get a few integrations done successfully with Int[]. I'll try to get as much as I can done and come back here and start a discussion for this too.

I'd greatly appreciate it if you would do more investigation and let us know what you find. Many thanks.

Possibly we could include a stripped down version of this for the next release which I hope will be sometime in December.