the "public" folder is still accessible from the outside world?
Yes , the accessbility of public folder is required
If it is required, what are your recommendations to block access to this folder so that no one can run the setup process again ?
Once you have completed your installation with UVDesk then you can redirect all the requests from web installer to knowledgebase panel by adding a route in file uvdesk.yaml resides in routes folder under config directory, that's how you could restrict the installer part.
Redirection route to be added in uvdesk.yaml (with proper indentation):-