You are correct.... within these clarifications/parameters:
You are on PC/Mac (not console)
Make sure that all of the Sim Households, Lots and Rooms that you want to preserve/save have been saved to your library (these will be in the tray file) or the community gallery to preserve them for future use.
To further clarify...
Deleting files in the saves folder in your 'The Sims 4' folder will only delete your game saves
All Households, Lots and Rooms that you have saved to your "My Library" will be saved into your "Tray" folder in your 'The Sims 4' folder.
Following is some additional information about saves and backups that may be helpful: