Changes in pod6 files not available after the first call to update-cache

[antoniogamiz]

You can see that happening in Raku/Documentable#73.

Maybe this question is relevant to the problem.

[JJ]

OK, the thing seems to be that as soon as you cal "load" it's cached, and it will not change.

[JJ]

The new version of 040-pod-extraction is failing now with:

Attempt to obtain non-existent POD for <a-pod-file>. Is the source new and failed to compile? Has the cache been updated?
  in method pod at /home/jmerelo/Code/forks/raku/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 214
  in block <unit> at t/040-pod-extraction.t line 34

This is due to the fact that, before, stuff was created and then tested in a different instance of a program. Now that we do it in the same instance it's not going to work. So that's the deeper problem I talk about in #17

[antoniogamiz]

I have been trying to unlock the cache as @JJ said here, without any success. I propose to use Pod::Load to serve the pods that have been modified. I mean, use Pod::Load in get-pods to serve those that have been modified and use the cache for those that have not.

[JJ]

The way I see it, there are several options

• Use a different mechanism to update the cache. Right now we are using high-level CompUnits, we should lower that level so that we don't turn the caches into something immutable.
• Use CompUnits at a higher level. Right now, CompUnits are a mechanism for retrieving precompiled code, which is version-based and has got its own indexing mechanisms. We are avoiding that mechanism altogether and using our own. Maybe we shouldn't; we should piggyback on it and add metainformation to current CompUnit caches.
• Hack the lock put on installed CompUnits and unlock it.

I think that the only reasonable purpose, relying on tested and documented APIs is the second one. The first is undocumented, and probably not frozen or specced. Ditto for third: we overcome some lock, it's bound to break. Besides, second would be also a way to re-use existing installations so that we can "install" documentation, for instance, and create HTML out of it, which would result in all kinds of good things.

[niner]

Can someone describe for me what you are actually trying to achieve and what kind of issues you are encountering? I'm a bit surprised that no one has ever talked to me about the best way to use our precompilation infrastructure. I could probably give you answers, so you don't have to guess and if there are no answers yet improve our API, so it covers your use case (whatever that is).

Also note that the stable API is defined by these classes and roles:

• CompUnit
• CompUnit::DependencySpecification
• CompUnit::Handle
• CompUnit::Loader
• CompUnit::PrecompilationRepository
• CompUnit::PrecompilationStore
• CompUnit::PrecompilationId
• CompUnit::PrecompilationDependency
• CompUnit::PrecompilationUnit
• CompUnit::Repository
• CompUnit::RepositoryRegistry
• Distribution
• Distribution::Resource
• Distribution::Resources

Everything else is subject to change. Everything not covered by spec tests is subject to change.

[JJ]

Thanks @niner, for dropping by. The basic idea is we want to use the precomp API for a POD cache. In a pinch I would describe it more or less like the current CUR infrastructure, but mutable. We would like to be able to cache a document we have already seen, and then check if the new version of the document is different (by date), and if it is, substitute it.

[niner]

Please don't assume that I know anything about our Pod stuff because....I don't :D
Why do you need a POD cache? Where and how should it be used? That's incidentally something that the README also doesn't answer.

Rakudo doesn't use timestamps for invalidating precomp files. The whole thing works only with checksums of file contents and for good reason. We learned the disadvantages of timestamps the hard way and I'd suggest you spare yourselves the pain and move on to checksums as well. After all the precomp infrastructure wants you to do that anyway.

I don't see why you'd have any issues with the precomp infrastructure for the use case as you've described it. After all, things work just fine when you develop a module in a FileSystem repository and make changes to source files. So I'm gonna need more information on your use case and your troubles.

[JJ]

El sáb., 29 feb. 2020 a las 8:52, niner (<[email protected]>) escribió:

Please don't assume that I know anything about our Pod stuff because....I don't :D Why do you need a POD cache? Where and how should it be used? That's incidentally something that the README also doesn't answer.

We need to generate the documentation site fast, so we need precompiled PODs for doing that. We'll open an issue to improve the README.md

Rakudo doesn't use timestamps for invalidating precomp files. The whole thing works only with checksums of file contents and for good reason. We learned the disadvantages of timestamps the hard way and I'd suggest you spare yourselves the pain and move on to checksums as well. After all the precomp infrastructure wants you to do that anyway.

OK. Any easy way of doing that?

I don't see why you'd have any issues with the precomp infrastructure for the use case as you've described it. After all, things work just fine when you develop a module in a FileSystem repository and make changes to source files. So I'm gonna need more information on your use case and your troubles.

Can you please tell us how to replicate the functionality you mention? We're not saying that it does not work, only we have no idea how to do that, which is why we're using the current mechanism, inspired by the code that was used in Pod::to::BigPage. So let's flesh out the use case * Initially, we'll cache (precompile) and index every file in the Raku/doc repo. Could be others, but that's the main use case. From those files, the primary and secondary HTML pages are generated. * When we update from the repo, some files will have changed. We need to know which are these; then we generate HTML pages only for those sites. There's a bit more here and there: generate graphs, things like that. But that's basically it. This issue is an example of what does not really work (and which were kind of mimicking so far, but now it does not, possibly after changes in 2020.02)

[niner]

We need to generate the documentation site fast, so we need precompiled PODs for doing that. We'll open an issue to improve the README.md

So you're regenerating that site quite often and want to only process the files that are actually changed? Sounds like the precomp cache can help, but then you'd still need to generate (presumably) HTML from the bytecode. Why not skip even that step and just not touch the HTML file at all unless the source file changed?

OK. Any easy way of doing that?

If you go through CompUnit::PrecompilationRepository::Default then it will automatically use checksums. You give it the path to the source file and it will do the check and either precompile or just load the thing.

Can you please tell us how to replicate the functionality you mention? We're not saying that it does not work, only we have no idea how to do that, which is why we're using the current mechanism, inspired by the code that was used in Pod::to::BigPage.

Sounds to me as simple as: my $precomp-repo = CompUnit::PrecompilationRepository::Default.new( :store(CompUnit::PrecompilationStore::File.new(:$prefix)), ); my $handle = $precomp-repo.try-load( CompUnit::PrecompilationDependency::File.new( :src($pod-file-path), :id(CompUnit::PrecompilationId.new-from-str($pod-file-path)), :spec(CompUnit::DependencySpecification.new(:$short-name), ), );

So let's flesh out the use case * Initially, we'll cache (precompile) and index every file in the Raku/doc repo. Could be others, but that's the main use case. From those files, the primary and secondary HTML pages are generated. * When we update from the repo, some files will have changed. We need to know which are these; then we generate HTML pages only for those sites.

So you do want to save that HTML generation step? Honestly I'd just use make(1) in that case. After all "The make utility will determine automatically which pieces of a large program need to be recompiled, and issue the commands to recompile them."

There's a bit more here and there: generate graphs, things like that. But that's basically it. This issue is an example of what does not really work (and which were kind of mimicking so far, but now it does not, possibly after changes in 2020.02)

This issue does not exactly explain what's wrong. Neither does it explain how the system is supposed to work. No steps to reproduce, expected or actual results.

[antoniogamiz]

Steps to reproduce it: Raku/Documentable#73 (comment)

[niner]

After installing Documentable::CLI (had to skip tests as Pod::To::Cached doesn't seem to installed), I tried the script you pointed out:

nine@sphinx:~/test> mkdir Pod-To-Cached-issue-73
nine@sphinx:~/test> cd Pod-To-Cached-issue-73/
nine@sphinx:~/test/Pod-To-Cached-issue-73> cat > test.raku
use lib 'lib';
use Documentable::CLI;
use Pod::To::Cached;
use Documentable::Utils::IO;

my $cache-dir = cache-path("docs");

my $pod-cache = Pod::To::Cached.new(
    source      => "docs",
    :verbose,
    path        => $cache-dir
);

$pod-cache.update-cache;

my $pod = $pod-cache.pod("type/documentable");

# the subtitle of the pod 
say $pod[0].contents[1].contents[0].contents.first
nine@sphinx:~/test/Pod-To-Cached-issue-73> raku test.raku 
Source verification failed with:
docs is not a directory
  in submethod TWEAK at /home/nine/rakudo/install/share/perl6/site/sources/E4B781FB1DD47C7D9EAE831448606DD3EBCDC8ED (Pod::To::Cached) line 63
  in block <unit> at test.raku line 8

Is that the error this issue is about?

[antoniogamiz]

You need to run it from the Documenable repo, so you have to clone it first. Sorry, I should have said that.

[niner]

nine@sphinx:~/test/Pod-To-Cached-issue-73/Documentable (master=)> raku ../test.raku
Source verified
Got cache at ./.cache-docs
Caching language/cli
Caching type/documentable/secondary
Caching type/documentable/docpage
Caching language/typegraph
Caching language/configuration-file
Caching type/documentable/config
Caching homepage
Caching language/document-format
Caching type/documentable/index
Caching type/documentable/heading/grammar
Caching type/documentable/to/html/wrapper
Caching type/documentable/primary
Caching type/documentable
Caching language/highlighting
Caching type/documentable/search
Caching type/docpage/factory
Caching type/documentable/registry
Caching 404
Cache fully updated
Represents a self-contained Perl 6 documentation element.
nine@sphinx:~/test/Pod-To-Cached-issue-73/Documentable (master=)> vi docs/Type/Documentable.pod6
nine@sphinx:~/test/Pod-To-Cached-issue-73/Documentable (master=)> raku ../test.raku
Source verified
Got cache at ./.cache-docs
Caching type/documentable
Cache fully updated
Represents a self-contained Perl 6 documentation element.
nine@sphinx:~/test/Pod-To-Cached-issue-73/Documentable (master *=)>

I don't get any error. Seems to work just fine?

[antoniogamiz]

Have you modified the subtitle of the pod? Does that change appear? Nope.

[antoniogamiz]

And that's the error, that we never truly update the cache. I mean, we update it, but those changes are not available to the program running the cache. For those changes to be available, we need to stop running the program and start it again.

[niner]

On Samstag, 29. Februar 2020 19:22:23 CET Antonio wrote: Have you modified the subtitle of the pod?

Yes, as you can see by the call to vi

Does that change appear? Nope.

Appear where?

[antoniogamiz]

When you print it. In the first run of the script you got:

Represents a self-contained Perl 6 documentation element.

And in the second run, supposedly after you modified the subtitle:

Represents a self-contained Perl 6 documentation element.

As you can see, the changes are not available.

[niner]

Unfortunately I've got to go now. Would have been so useful to have this info as real steps to reproduce, so I wouldn't have had to guess so much.

[antoniogamiz]

Sorry for all the trouble and misunderstanding, I should have added that to the issue. I will try to do it better next time, thanks for your time.

[niner]

I replaced Pod::To::Cached with the code I suggested above and it precompiles and picks up changes in the files just fine:

my $pod-file-path = "docs/Type/Documentable.pod6";

my $precomp-repo = CompUnit::PrecompilationRepository::Default.new(
    :store(CompUnit::PrecompilationStore::File.new(:prefix('docs/.precomp'.IO))),
);
my $handle = $precomp-repo.try-load(
    CompUnit::PrecompilationDependency::File.new(
        :src($pod-file-path),
        :id(CompUnit::PrecompilationId.new-from-string($pod-file-path)),
        :spec(CompUnit::DependencySpecification.new(:short-name($pod-file-path))),
    ),
);

use nqp;
my $pod = nqp::atkey($handle.unit, '$=pod');

say $pod[0].contents[1].contents[0].contents.first;

[antoniogamiz]

That's fantastic. I will try to use that code in Pod::To::Cached and see what happens. Thanks a lot for your help :).

[JJ]

El sáb., 29 feb. 2020 a las 18:31, niner (<[email protected]>) escribió:

> We need to generate the documentation site fast, so we need precompiled > PODs for doing that. We'll open an issue to improve the README.md So you're regenerating that site quite often and want to only process the files that are actually changed? Sounds like the precomp cache can help, but then you'd still need to generate (presumably) HTML from the bytecode. Why not skip even that step and just not touch the HTML file at all unless the source file changed?

That's what we do, that's what we would like to keep doing, but haven't been able since some recent change. Also, the thing is a bit more complicated, but that's the gist of it.

> OK. Any easy way of doing that? If you go through CompUnit::PrecompilationRepository::Default then it will automatically use checksums. You give it the path to the source file and it will do the check and either precompile or just load the thing. > Can you please tell us how to replicate the functionality you mention? > We're not saying that it does not work, only we have no idea how to do > that, which is why we're using the current mechanism, inspired by the code > that was used in Pod::to::BigPage. Sounds to me as simple as: my $precomp-repo = CompUnit::PrecompilationRepository::Default.new( :store(CompUnit::PrecompilationStore::File.new(:$prefix)), ); my $handle = $precomp-repo.try-load( CompUnit::PrecompilationDependency::File.new( :src($pod-file-path), :id(CompUnit::PrecompilationId.new-from-str($pod-file-path)), :spec(CompUnit::DependencySpecification.new(:$short-name), ), );

That looks great. We'll check.

So let's flesh out the use case > * Initially, we'll cache (precompile) and index every file in the Raku/doc > repo. Could be others, but that's the main use case. From those files, the > primary and secondary HTML pages are generated. > * When we update from the repo, some files will have changed. We need to > know which are these; then we generate HTML pages only for those sites. So you do want to save that HTML generation step? Honestly I'd just use make(1) in that case. After all "The make utility will determine automatically which pieces of a large program need to be recompiled, and issue the commands to recompile them."

It's a bit more complicated, because multiple files are generated from multiple sources, and the files that are actually generated are not known until the compiled document is checked...

> There's a bit more here and there: generate graphs, things like that. But > that's basically it. This issue is an example of what does not really work > (and which were kind of mimicking so far, but now it does not, possibly > after changes in 2020.02) This issue does not exactly explain what's wrong. Neither does it explain how the system is supposed to work. No steps to reproduce, expected or actual results.

Well, this are the issues of one of the tests that are failing now. But what you mention above is helpful. We'll give it a try, thanks.

[JJ]

Thanks @niner, I appreciate.

[JJ]

We need to fix this pronto, using code suggested by @niner
This is going to imply a major rewrite, including possibly a change in API. So stay tuned.

[JJ]

I'm checking this and the changes needed to get it going, and I'm under the impression it's going to be better to just writing a new cache from scratch to use it from Documentable. Most attributes (for instance, the JSON file with state) is not really going to be needed if we use this.