Mkwrapper Stop block: ModuleNotFoundError: No module named 'preview'

[the-snowwhite]

The whole premise in this thread is to see:

A Is it possible to get the qtquickvcp gui's running by adding the missing preview.so lib ?
Edit: Answer: allmost

B.Find out what else is missing.
Edit Answer at bottom

Based on this suggestion
I compiled Emcapp with full python 3 support using this method:

git clone https://github.com/machinekit/emcapplication.git
cd emcapplication
sudo apt install build-essential fakeroot devscripts equivs python3 python3-tk lsb-release apt-cudf
debian/configure machinekit-hal=0.4.21003-1.git39389db84~$(lsb_release -cs) no-docs
mk-build-deps -irs sudo -t 'apt-cudf-get --solver aspcud -o Debug::pkgProblemResolver=0 -o APT::Install-Recommends=0'
cd src
./autogen.sh
./configure --with-hal=machinekit-hal --with-python=python3 --with-boost-python=boost_python-py35
time make -j$(nproc)
sudo make install
source ~/emcapplication/scripts/rip-environment
linuxcnc

Then tested the axis sim config: ran without any problems execpt this message:

1: unspecified exception detail=0x55c7de3db8 ts=0x7facd0b164

---

Mkwrapper:

I used the newest package from machinekit-hal on purpose.

I used this hal config updated for linuxcnc 2.9 (and the newest added soc)
I have a similar version that is tested to work with the compiled in axis gui.

Here is the result of a python3 ./run.py run:

machinekit@mksocfpga-fz3-soc:~/Hm2-soc_FDM/Cramps/HAL/FZ3$ ./run.py
starting mklauncher... done
starting configserver... done
starting linuxcnc... LINUXCNC - 2.9.0~pre0
Machine configuration directory is '/home/machinekit/Hm2-soc_FDM/Cramps/HAL/FZ3'
Machine configuration file is 'ox.ini'
Starting LinuxCNC...
rtapi_msgd command:  /usr/libexec/machinekit/rtapi_msgd --instance=0 --rtmsglevel=1 --usrmsglevel=1 --debug=1 --halsize=524288
rtapi_app command:  /usr/libexec/machinekit/rtapi_app --instance=0 --debug=1
3::3910:rt INFO:  Picked default flavor 'posix' automatically
done
Found file(REL): ./ox.hal
emcTrajSetJoints(3) returned 0
emcTrajSetSpindles(1) returned 0
emcTrajSetAxes(3, 7)
emcTrajSetUnits(1.0000, 1.0000)
emcTrajSetVelocity(0.0000, 20.0000) returned 0
emcTrajSetMaxVelocity(200.0000) returned 0
emcTrajSetAcceleration(999999999999999967336168804116691273849533185806555472917961779471295845921727862608739868455469056.0000) returned 0
emcTrajSetMaxAcceleration(999999999999999967336168804116691273849533185806555472917961779471295845921727862608739868455469056.0000)
emcTrajSetHome(0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000) returned 0
emcJointSetType(0, 1)
emcJointSetUnits(0, 1.0000)
emcJointSetBacklash(0, 0.0600) returned 0
emcJointSetMinPositionLimit(0, -0.5000) returned 0
emcJointSetMaxPositionLimit(0, 320.0000) returned 0
emcJointSetFerror(0, 800.0000) returned 0
emcJointSetMinFerror(0, 200.0000) returned 0
emcJointSetHomingParams(0, 0.0000, 0.0000, -1.0000, -10.0000, 1.0000, 0, 1, 0, 1, 0) returned 0
emcJointSetMaxVelocity(0, 50.0000) returned 0
emcJointSetMaxAcceleration(0, 80.0000) returned 0
emcJointActivate(0) returned 0
emcJointSetType(1, 1)
emcJointSetUnits(1, 1.0000)
emcJointSetBacklash(1, 0.0800) returned 0
emcJointSetMinPositionLimit(1, -1.0000) returned 0
emcJointSetMaxPositionLimit(1, 500.1000) returned 0
emcJointSetFerror(1, 800.0000) returned 0
emcJointSetMinFerror(1, 200.0000) returned 0
emcJointSetHomingParams(1, 0.0000, 0.0000, -1.0000, -20.0000, 1.0000, 0, 1, 0, 1, 0) returned 0
emcJointSetMaxVelocity(1, 50.0000) returned 0
emcJointSetMaxAcceleration(1, 80.0000) returned 0
emcJointActivate(1) returned 0
emcJointSetType(2, 1)
emcJointSetUnits(2, 1.0000)
emcJointSetBacklash(2, 0.0200) returned 0
emcJointSetMinPositionLimit(2, -40.2000) returned 0
emcJointSetMaxPositionLimit(2, 0.2000) returned 0
emcJointSetFerror(2, 800.0000) returned 0
emcJointSetMinFerror(2, 200.0000) returned 0
emcJointSetHomingParams(2, 0.0000, 0.2000, -1.0000, 8.0000, -1.0000, 0, 1, 0, 0, 0) returned 0
emcJointSetMaxVelocity(2, 10.0000) returned 0
emcJointSetMaxAcceleration(2, 20.0000) returned 0
emcJointActivate(2) returned 0
emcAxisSetMinPositionLimit(0, -0.5000) returned 0
emcAxisSetMaxPositionLimit(0, 320.0000) returned 0
emcAxisSetMaxVelocity(0, 50.0000) returned 0
emcAxisSetMaxAcceleration(0, 80.0000) returned 0
emcAxisSetLockingJoint(0, -1) returned 0
emcAxisSetMinPositionLimit(1, -1.0000) returned 0
emcAxisSetMaxPositionLimit(1, 500.1000) returned 0
emcAxisSetMaxVelocity(1, 50.0000) returned 0
emcAxisSetMaxAcceleration(1, 90.0000) returned 0
emcAxisSetLockingJoint(1, -1) returned 0
emcAxisSetMinPositionLimit(2, -40.0000) returned 0
emcAxisSetMaxPositionLimit(2, 0.2000) returned 0
emcAxisSetMaxVelocity(2, 10.0000) returned 0
emcAxisSetMaxAcceleration(2, 20.0000) returned 0
emcAxisSetLockingJoint(2, -1) returned 0
hm2: loading Mesa HostMot2 driver version 0.15

hm2_soc_ol: loading Mesa AnyIO HostMot2 socfpga overlay driver version 0.9

hm2/hm2_fz3_.0: Low Level init 0.15

hm2/hm2_fz3_.0: 36 I/O Pins used:

hm2/hm2_fz3_.0:     IO Pin 000 (GPIO0.P0-01): IOPort

hm2/hm2_fz3_.0:     IO Pin 001 (GPIO0.P0-02): IOPort

hm2/hm2_fz3_.0:     IO Pin 002 (GPIO0.P0-03): IOPort

hm2/hm2_fz3_.0:     IO Pin 003 (GPIO0.P0-04): IOPort

hm2/hm2_fz3_.0:     IO Pin 004 (GPIO0.P0-05): StepGen #0, pin Direction (Output)

hm2/hm2_fz3_.0:     IO Pin 005 (GPIO0.P0-06): StepGen #0, pin Step (Output)

hm2/hm2_fz3_.0:     IO Pin 006 (GPIO0.P0-07): StepGen #1, pin Direction (Output)

hm2/hm2_fz3_.0:     IO Pin 007 (GPIO0.P0-08): StepGen #1, pin Step (Output)

hm2/hm2_fz3_.0:     IO Pin 008 (GPIO0.P0-09): StepGen #2, pin Direction (Output)

hm2/hm2_fz3_.0:     IO Pin 009 (GPIO0.P0-10): StepGen #2, pin Step (Output)

hm2/hm2_fz3_.0:     IO Pin 010 (GPIO0.P0-11): StepGen #3, pin Direction (Output)

hm2/hm2_fz3_.0:     IO Pin 011 (GPIO0.P0-12): StepGen #3, pin Step (Output)

hm2/hm2_fz3_.0:     IO Pin 012 (GPIO0.P0-13): Hostmot2 DPLL #0, pin Ref Out Pin (Output)

hm2/hm2_fz3_.0:     IO Pin 013 (GPIO0.P0-14): Encoder #0, pin A (Input)

hm2/hm2_fz3_.0:     IO Pin 014 (GPIO0.P0-15): Encoder #0, pin B (Input)

hm2/hm2_fz3_.0:     IO Pin 015 (GPIO0.P0-16): Encoder #0, pin Index (Input)

hm2/hm2_fz3_.0:     IO Pin 016 (GPIO0.P0-17): IOPort

hm2/hm2_fz3_.0:     IO Pin 017 (GPIO0.P0-18): PWMGen #0, pin Out0 (PWM or Up) (Output)

hm2/hm2_fz3_.0:     IO Pin 018 (GPIO0.P1-19): IOPort

hm2/hm2_fz3_.0:     IO Pin 019 (GPIO0.P1-20): IOPort

hm2/hm2_fz3_.0:     IO Pin 020 (GPIO0.P1-21): IOPort

/usr/lib/python3/dist-packages/pyftpdlib/authorizers.py:244: RuntimeWarning: write permissions assigned to anonymous user.
  RuntimeWarning)
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/bin/mkwrapper", line 322, in run
    import preview  # must be imported in new process to work properly
ModuleNotFoundError: No module named 'preview'
send_pbcontainer: type: MT_PING

send_pbcontainer: type: MT_PING

task: main loop took 5.014869 seconds
send_pbcontainer: type: MT_PING

send_pbcontainer: type: MT_PING

send_pbcontainer: type: MT_PING

^Cstopping mklauncher... done
stopping configserver... done
stopping linuxcnc... done
machinekit@mksocfpga-fz3-soc:~/Hm2-soc_FDM/Cramps/HAL/FZ3$ task: main loop took 5.003195 seconds
task: 27 cycles, min=0.000181, max=5.014869, avg=0.375194, 2 latency excursions (> 10x expected cycle time of 0.010000s)

Seems like config is parsed ok however mkwrapper cannot fínd the module preview.

Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/bin/mkwrapper", line 322, in run
    import preview  # must be imported in new process to work properly
ModuleNotFoundError: No module named 'preview'

I wonder where this python preview module requested here is supposed to come from.

BTW
With these changes to the linuxcnc script
Im able to run a similar python based hal config
however the end result is the same error traceback

[the-snowwhite]

I had some similar module not found issues with my python configs in the fdm folder
I got module errors in the base.py file

before i changed the lines from:

import rcomps
import storage
import motion

to:

from fdm.config import rcomps
from fdm.config import storage
from fdm.config import motion

Like python 3.7 handles local paths different ?

[the-snowwhite]

This is not a python 2/3 issue:
I revived my Emcapp python2 setup with this patch specified here and the problem persists:
( I even tried copying over an old mkwrapper from my running machinekit-cnc ox-router setup, same result)

/usr/lib/python2.7/dist-packages/pyftpdlib/authorizers.py:244: RuntimeWarning: write permissions assigned to anonymous user.
RuntimeWarning)
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
File "/usr/bin/mkwrapper", line 322, in run
    import preview  # must be imported in new process to work properly
ImportError: No module named preview

@cerna
Can this have something to do with how the python include/search paths are setup in Linuxcnc differing from Machinekit-cnc ?

[the-snowwhite]

Poking around in my old ob-ox setup I can find out where the preview comes from there:

machinekit@mksocfpga-ob-ox:~$ python
Python 2.7.13 (default, Aug 22 2020, 10:03:02) 
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import preview
>>> preview.__file__
'/usr/lib/python2.7/dist-packages/preview.so'
>>> exit()
preview: socket shutdown
machinekit@mksocfpga-ob-ox:~$ cat /etc/*-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

And find out that its missing in my newest setup:

machinekit@mksocfpga-ultra-soc:~/Hm2-soc_FDM/Cramps/HAL/ULTRA$ python
Python 2.7.16 (default, Oct 10 2019, 22:02:15) 
[GCC 8.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import preview
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named preview
>>> exit()
machinekit@mksocfpga-ultra-soc:~/Hm2-soc_FDM/Cramps/HAL/ULTRA$ cat /etc/*-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

However struggling to find out where the: '/usr/lib/python2.7/dist-packages/preview.so' file is from

[the-snowwhite]

Well I can find it in the (from machinekit.io)
machinekit-cnc-rt-preempt_0.1.1553365560.git3686a54-1_stretch_armhf.deb
Still struggling to find out how its compiled

[the-snowwhite]

The preview module is compiled in from:
https://github.com/machinekit/machinekit-cnc/blob/master/src/emc/rs274ngc/Submakefile#L77-L87

[the-snowwhite]

The whole compile blob shows it to depend on haltalk, czmq and the protobuff stuff:

PREVIEWMODULESRCS := \
    emc/rs274ngc/previewmodule.cc
PYSRCS += $(PREVIEWMODULESRCS)

$(call TOOBJSDEPS, $(PREVIEWMODULESRCS)) : Makefile.inc
$(call TOOBJSDEPS, $(PREVIEWMODULESRCS)) : EXTRAFLAGS += -Ihal/haltalk $(CZMQ_CFLAGS) $(PROTOBUF_CFLAGS)

PREVIEWMODULE := ../lib/python/preview.so
$(PREVIEWMODULE): $(call TOOBJS, $(PREVIEWMODULESRCS)) \
    ../lib/librs274.so.0 \
    ../lib/libmachinetalk-pb2++.so.0 \
    ../lib/libmtalk.so
    $(ECHO) Linking python module $(notdir $@)
    $(Q)$(CXX) $(LDFLAGS) -shared -o $@ $^ -lstdc++  $(CZMQ_LIBS) \
        $(PROTOBUF_LIBS)


PYTARGETS += $(PREVIEWMODULE)

@cerna
I'm not sure if the QtQuickvcp repo would be the natural place to compile this code in from. Comes to mind a more natural place would be a haltalk repo based on this folder.

[the-snowwhite]

I'm able to get the preview.so to compile in machinekit-hal however I'm unable to get past this issue:

ImportError: /usr/lib/librs274.so.0: undefined symbol: _Z18GET_EXTERNAL_SPEEDi

/usr/lib/python2.7/dist-packages/pyftpdlib/authorizers.py:244: RuntimeWarning: write permissions assigned to anonymous user.
  RuntimeWarning)
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/bin/mkwrapper", line 322, in run
    import preview  # must be imported in new process to work properly
ImportError: /usr/lib/librs274.so.0: undefined symbol: _Z18GET_EXTERNAL_SPEEDi
^Cstopping mklauncher... done
stopping configserver... done
stopping linuxcnc... task: 3456 cycles, min=0.000134, max=0.010570, avg=0.010017, 0 latency excursions (> 10x expected cycle time of 0.010000s)
exiting HAL component storage
done

$ c++filt _Z18GET_EXTERNAL_SPEEDi
GET_EXTERNAL_SPEED(int)

[the-snowwhite]

Turns out I needed to use the files from the Emc app src/emc folder instead of the ones from mk-cnc and update previewmodule.cc
via looking at the gcodemodule.cc file from emc app.
The "patches" are currently in 2 separate branches: arm64 armhf
These patches are for preview only and intended as somthing to copy and paste from (ie. previewmodule.cc)

[the-snowwhite]

Results running in machinekit-client service view:

There are 4 services missing compared to a correctly running instance:

HAL Rcommand service
HAL Rcomp service
HAL Group service
Log service

[the-snowwhite]

running the gui (MachinekitClient) gives following execption:

Exception in thread process_sockets:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/bin/mkwrapper", line 678, in process_sockets
    self.update_status(self.stat)
  File "/usr/bin/mkwrapper", line 1758, in update_status
    self.update_motion(stat)
  File "/usr/bin/mkwrapper", line 1510, in update_motion
    modified |= self.update_motion_value(name, getattr(stat, name))
AttributeError: 'linuxcnc.stat' object has no attribute 'spindle_brake'

[the-snowwhite]

Looks like mkwrapper needs to be updated to the linuxcnc 2.8 , 2.9 changes:
axis --> joints
motion.spindle.xx --> spindle..xx

[the-snowwhite]

This exception seems to be caused by mkwrapper not being updated to the new lcnc axis --> joints update in 2.8 , 2.9.
A way to see the changes needed seem to be to compare emctaskmain.cc from machinekit-cnc to emcapp

---

Also same update seems to be needed to the protobuf stuff here