how to deploy to heroku? #151

buncis · 2021-07-30T16:11:54Z

when I want to deploy to heroku

the application always failed to build with this error message

PyCall::LibPythonFunctionNotFound: Unable to find the required symbol in libpython: _Py_NoneStruct

heroku logs -t

       Using rails 6.1.4
       Using rails_admin 2.2.0
       Installing pycall 1.4.1 with native extensions
       Fetching numpy 0.4.0
       Installing numpy 0.4.0
       Fetching pandas 0.3.8
       Installing pandas 0.3.8
       Bundle complete! 20 Gemfile dependencies, 77 gems now installed.
       Gems in the groups 'development' and 'test' were not installed.
       Bundled gems are installed into `./vendor/bundle`
       Bundle completed (6.72s)
       Cleaning up the bundler cache.
-----> Detecting rake tasks
 !
 !     Could not detect rake tasks
 !     ensure you can run `$ bundle exec rake -P` against your app
 !     and using the production group of your Gemfile.
 !     rake aborted!
 !     PyCall::LibPythonFunctionNotFound: Unable to find the required symbol in libpython: _Py_NoneStruct
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.5/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.5/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.5/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.5/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.5/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
...
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.5/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/rake_module.rb:29:in `load_rakefile'
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:710:in `raw_load_rakefile'
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:104:in `block in load_rakefile'
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:103:in `load_rakefile'
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:82:in `block in run'
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
 !     /tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
 !     /tmp/build_403b9e21/bin/rake:5:in `<main>'
 !
/tmp/codon/tmp/buildpacks/50d5eddf222a9b7326028041d4e6509f915ccf2c/lib/language_pack/helpers/rake_runner.rb:106:in `load_rake_tasks!': Could not detect rake tasks (LanguagePack::Helpers::RakeRunner::CannotLoadRakefileError)
ensure you can run `$ bundle exec rake -P` against your app
and using the production group of your Gemfile.
rake aborted!
PyCall::LibPythonFunctionNotFound: Unable to find the required symbol in libpython: _Py_NoneStruct
/tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.5/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
/tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.5/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
/tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.5/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
/tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.5/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
/tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.5/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
/tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4/lib/active_support/dependencies.rb:332:in `block in require'
...
/tmp/build_403b9e21/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/tmp/build_403b9e21/bin/rake:5:in `<main>'
	from /tmp/codon/tmp/buildpacks/50d5eddf222a9b7326028041d4e6509f915ccf2c/lib/language_pack/ruby.rb:1007:in `rake'
	from /tmp/codon/tmp/buildpacks/50d5eddf222a9b7326028041d4e6509f915ccf2c/lib/language_pack/rails4.rb:78:in `block (2 levels) in run_assets_precompile_rake_task'
	from /tmp/codon/tmp/buildpacks/50d5eddf222a9b7326028041d4e6509f915ccf2c/lib/language_pack/base.rb:190:in `log'
	from /tmp/codon/tmp/buildpacks/50d5eddf222a9b7326028041d4e6509f915ccf2c/lib/language_pack/rails4.rb:72:in `block in run_assets_precompile_rake_task'
	from /tmp/codon/tmp/buildpacks/50d5eddf222a9b7326028041d4e6509f915ccf2c/lib/language_pack/instrument.rb:18:in `block (2 levels) in instrument'
	from /tmp/codon/tmp/buildpacks/50d5eddf222a9b7326028041d4e6509f915ccf2c/lib/language_pack/instrument.rb:40:in `yield_with_block_depth'
	from /tmp/codon/tmp/buildpacks/50d5eddf222a9b7326028041d4e6509f915ccf2c/lib/language_pack/instrument.rb:17:in `block in instrument'
	....
	from /tmp/codon/tmp/buildpacks/50d5eddf222a9b7326028041d4e6509f915ccf2c/bin/support/ruby_compile:15:in `<main>'
 !     Push rejected, failed to compile Ruby app.
 !     Push failed

The text was updated successfully, but these errors were encountered:

buncis · 2021-07-30T16:19:36Z

apparently this is because default python that heroku use is not builded with --enable-shared

thus why our build is failed because pycall.rb need python that use enable-shared

to fix this problem here are the steps

first, set your heroku buildpack

since using pycall.rb our apps is consist of multiple stack (Python, Ruby, Javascript)
to set the buildpack what you need to do are

clean the current buildpack
heroku buildpacks:clear
add the needed buildpack with order (-i set the order)
heroku buildpacks:add heroku/python - i 1
heroku buildpacks:add heroku/nodejs -i 2
heroku buildpacks:add heroku/ruby -i 3

optional step, clear your heroku build cache

heroku sometimes use build cache instead compiling from scratch, to clear it
https://help.heroku.com/18PI5RSY/how-do-i-clear-the-build-cache
first install the plugin
heroku plugins:install heroku-builds
Then use the following command to clear the cache:
heroku builds:cache:purge
The cache will be rebuilt on the next deploy. If you do not have any new code to deploy, you can push an empty commit.

$ git commit --allow-empty -m "Purge cache"
$ git push heroku master

second, add neccessary file & scripts to your rails project

the python buildpacks from heroku need two file to run, build and install the dependencies

runtime.txt this is file where u declared ur python version
my runtime.txt
python-3.8.10
requirements.txt this is file where u declare python package that you use
my requirements.txt

openpyxl==3.0.7
numpy==1.21.1
pandas==1.3.1

put those 2 files on root of your rails app folder, they are on same folder with your Gemfile and .ruby-version

third, make heroku use python that have `--enable-shared` config

this is essentials since our beloved pycall.rb depends on it, it cannot work with python without --enable-shared
there is multiple way to achieve this, the readme page has it owns section, but imho its kinda obsolete and not working

my recommended approach

is using the built in post_compile hooks from official heroku/python buildpacks
to do that add post_compile to your bin/ folder in your rails app

post_compile

set -e
buildpack_url=https://github.com/heroku/heroku-buildpack-python
buildpack_vsn=v197

# rebuild python if it's missing enable-shared
if  ! python3 -msysconfig | grep enable-shared \
    > /dev/null; then
  PYTHON_VERSION="$(< runtime.txt)"
  git clone -b "$buildpack_vsn" "$buildpack_url" _buildpack
  export WORKSPACE_DIR="$PWD/_buildpack/builds"
  rm -f .heroku/python/bin/python   # prevent failing ln after build
  sed -i 's!figure --pre!figure --enable-shared --pre!' \
    "$WORKSPACE_DIR"/runtimes/python3
  "$WORKSPACE_DIR/runtimes/$PYTHON_VERSION" /app/.heroku/python/
  rm -fr _buildpack
fi

with this when the heroku building your app for first time it will check did it have your required python runtime if not then they will install it but they will install it using pre-compile python that not use the --enable-shared options
after this first time script run then it will call the post_compile script

the post_compile script
will check if the installed python is have --enabled-shared config
if not the script will tell to remove the current installed python and compile the python runtime with --enabled-shared

after this your app could build smoothly

another approach

is to compile the python by urself with --enable-shared and upload the precompiled python somewhere then fork the official https://github.com/heroku/heroku-buildpack-python buildpacks and somehow make the scripts aim to download your precompiled python instead using heroku precompiled python

extra tips

if you want to check if your current python runtime has --enable-shared you could do run

heroku run bash then paste this
$ python3 -msysconfig | grep enable-shared
if there is --enable-shared then you good to go

another approach is in your bash open python
then from your python repl

import sysconfig
sysconfig.get_config_vars('Py_ENABLE_SHARED')

if it return [1] then its builded with --enable-shared

buncis · 2021-07-30T16:55:01Z

@mrkn thanks for this gem with this gem I could get the best of two worlds python pandas and numpy + ruby and rails

could we add this into wiki or readme.md?

but I think putting this on readme would be too much

mrkn · 2022-02-10T00:34:12Z

@buncis I'm sorry for my response to being late. PyCall's README already has the section that describes tips for deploying on Heroku. This section was added by the pull-request #134 on March 9, 2021. Did you check this part in README?

buncis · 2022-02-10T03:06:03Z

@mrkn I have read that readme before, problem is I can't make my app work with that README

I think we could close this issue right now and could you enable discussion or wiki in this github repo?

so I could post my issue and guide to there?

mrkn · 2022-02-10T03:15:36Z

@buncis I don't want to open the wiki because the wiki is too difficult to keep managed. Opening the discussion can be acceptable, but I believe README is appropriate for mentioning the way to resolve this problem.

I want to know what information you needed is absent in the current README. Could you please tell me about it?

buncis · 2022-02-10T03:18:26Z

all the buildpacks mentioned in the readme is too old for newer applications or just doesn't works because its not updated the only solution is to make ur own buildpacks or recompile the heroku default buildpack with --enable-shared options

…

On Thu, Feb 10, 2022, 10:15 AM Kenta Murata ***@***.***> wrote: @buncis <https://github.com/buncis> I don't want to open the wiki because the wiki is too difficult to keep managed. Opening the discussion can be acceptable, but I believe README is appropriate for mentioning the way to resolve this problem. I want to know what information you needed is absent in the current README. Could you please tell me about it? — Reply to this email directly, view it on GitHub <#151 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADMZEW34UC6JDFZWHKDLVKDU2MUWHANCNFSM5BI2A4IA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: <mrkn/pycall .***@***.***>

mrkn · 2022-02-10T03:32:03Z

@buncis Thanks. I understand. It should be described in README and I want to do so. Could you please make a pull request to update the Heroku section in README?

buncis · 2022-02-10T03:45:07Z

I wish, but I can't do it right now, I'll do it later

mrkn · 2022-02-10T04:06:03Z

@buncis No problem. Thank you very much!

This was referenced Aug 2, 2021

Using pycall on Dokku (Heroku compatible alternative) #146

Closed

PyCall::LibPythonFunctionNotFound (Unable to find the required symbol in libpython: _Py_NoneStruct) #149

Closed

mrkn added the question label Feb 10, 2022

buncis mentioned this issue Mar 11, 2022

update deploying on heroku readme #156

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

how to deploy to heroku? #151

how to deploy to heroku? #151

buncis commented Jul 30, 2021 •

edited

Loading

buncis commented Jul 30, 2021 •

edited

Loading

buncis commented Jul 30, 2021 •

edited

Loading

mrkn commented Feb 10, 2022

buncis commented Feb 10, 2022

mrkn commented Feb 10, 2022

buncis commented Feb 10, 2022 via email

mrkn commented Feb 10, 2022

buncis commented Feb 10, 2022

mrkn commented Feb 10, 2022

how to deploy to heroku? #151

how to deploy to heroku? #151

Comments

buncis commented Jul 30, 2021 • edited Loading

buncis commented Jul 30, 2021 • edited Loading

first, set your heroku buildpack

optional step, clear your heroku build cache

second, add neccessary file & scripts to your rails project

third, make heroku use python that have --enable-shared config

my recommended approach

another approach

extra tips

buncis commented Jul 30, 2021 • edited Loading

mrkn commented Feb 10, 2022

buncis commented Feb 10, 2022

mrkn commented Feb 10, 2022

buncis commented Feb 10, 2022 via email

mrkn commented Feb 10, 2022

buncis commented Feb 10, 2022

mrkn commented Feb 10, 2022

buncis commented Jul 30, 2021 •

edited

Loading

buncis commented Jul 30, 2021 •

edited

Loading

third, make heroku use python that have `--enable-shared` config

buncis commented Jul 30, 2021 •

edited

Loading