Automatically delete non-expected files from the filesystem

[mattab]

in #11096 and #11107 we introduced a new feature where the file integrity checker now also checks for files which are present in the Piwik filesystem, but are not expected.

Often these are files that used to be in previous Piwik versions but were removed afterwards. our automatic updater should automatically delete these files, but it was only introduced during 2.X cycle so many un-expected files will be present for users who used Piwik for a long time.

See an example of such files reported here: #11107 (comment)

Ideas:

• Maybe we could delete all these files automatically?
• Or maybe we delete files only in some specific folders to avoid over-deleting (eg. vendor/, libs/)

[hpvd]

thanks for opening this issue.

imho every file installed by piwik in any version and not needed anymore in newer version of piwik should be automatically deleted when installing/updating to any newer version.

Since one can not assume every user updates from every version directly to following version of piwik, but there is a big chance users miss one or several minor or even major versions in-between,
maybe one need for the deleting tool a list of files (growing over time) within every piwik installation package which contains name and path of ALL of these files of ANY ever released version (the known ones ;-)

Otherwise users may get irritated or even get afraid that their server has been taken over and all the files mentioned here belong to a dangerous virus/malware...

Would vote strongly to address this issue in the not to far future.

[hpvd]

the other way around when having a positive list of files, one can of course automatically delete all files not within this positive list.
Doing this, one should add a check-mark in installation procedure to "clean installation"
(if there is no check mark, files added by the user on purpose disappear too...)

[hpvd]

or, to keep installation process without any additional check mark:
add a button to "clean installation" of all files not containing in positive list directly to integrity checker result...
(one can add a hint that also additional files added by the user/admin will be deleted)

[mattab]

add a button to "clean installation" of all files not containing in positive list directly to integrity checker result...

That's a nice idea! we'll consider this

[gaumondp]

I'm a bit against the full "blind" automated cleaning. Use case : Maybe an admin put a adminer.php or phpinfo.php file for debug and secured them well enough to keep them there...

OTOH I really like the simple "Clean up my installation".

Maybe a middle ground and flexible way would be to let Admins add some file or folders untouched by whitelisting them in config/config.ini.php

[ExtraFilesFolders]
adminer/
phpinfo.php
my.css

That way it would still be possible to get a clean Piwik installation while having some liberty.

[eldk]

Hello,

I'm a bit against the full "blind" automated cleaning
OTOH I really like the simple "Clean up my installation".

I agree with @gaumondp

Maybe a middle ground (...)

Eric

[mattab]

the fact that HTML entities are displayed in the Update screen file integrity error message will be tackled in #11167

[chriscroome]

A command line option for deleting old, unneeded, files would be good — as far as I'm aware this is the only part of upgrading Matomo which can't be scripted at the moment (please point me in the right direction if it can be).

[tsteur]

I wouldn't delete such files automatically during the update as we in the past often had trouble and falsely reported some files. And with introduction of new features such things may happen again. You could build such a command though.

• First make sure developer mode is enabled, see https://developer.matomo.org/guides/getting-started-part-1
• Create a command for the CoreUpdater plugin https://developer.matomo.org/guides/piwik-on-the-command-line (./console generate:command --pluginname=CoreUpdater)
• In the command get all files through \Piwik\FileIntegrity::getFileIntegrityInformation() (haven't double checked if it is the right method)
• Delete each file / dir... for files we have \Piwik\Filesystem::deleteFileIfExists($file), for directories we have Piwik\FileSystem::unlinkRecursive($dir, $deleteRootToo)

I'd recommend to make a backup of the files personally before executing the command

[chriscroome]

I agree that doing it automatically would be potentially too risky, but having a command line option to get a list of the files that you can access in the web interface would be good enough for me — I could then check the list before deciding to delete the files or not.

For what it is worth these are the Ansible tasks I'm using to upgrade multiple Matomo servers at a time and I'd ideally like to not have to subsequently login into each one and then click through to the system check to get a list of files to be deleted.

[fdellwing]

After 3.9.0 I stumpled across this again.

I propose another solution:

I don't see a big risk for much going wrong as I would not integrate this into the integrety report but write it by hand in the core/Updates/xxx.php.

This way, there is absolute controll over what get's deleted. This means that something maybe does not get deleted (false negative) but nothing gets deleted that should not be deleted (false positive).