Solved the problem for myself and just maybe, it will also be a solution for Ghost.
Between the feeder and the printhead cables and filament run through a (in my case) black tube which goes down in a spiral towards the printhead. My tube was too far down, close to the printhead (about 3 cm). Looks nice, but when the printhead moves to the outer positions, the black tube is too rigid to follow and bend as required. Hence, it puts a force to the magnetic balls to push them loose.
After moving the tube upward leaving about 4cm of free cable and filament tube exposed, it is flexible enough to not loosen the magnets. So, if Ghost still has this issue, check whether the prints are very wide (reaching the outer diameter of the printarea or high reaching the top. In those situations I can imagine the tube will again work the magnets out of position.
Anyway, hope this post helps other (new) users in case the meet the same issue.
PS: Do not shorten the tube or work it up too high, as the losse cables will then pose other issues.