The Total Score in your Site Audit campaign is based on the amount of your total errors and total warnings that were found on the pages crawled on your site. Errors have more impact on your Total Score than warnings, so it is recommended that you address errors first if you can.
If you see your total number of issues decrease but your score remain the same or get worse, then there is likely an important error that still needs to be addressed.
Your Site Audit will also provide a list of notices. As far as notices go, you will still want to fix these issues as they still impact your SEO, but they don't have as much of an impact on the Total Score as errors and warnings have.
For the full list of issues that Site Audit checks for, read this FAQ: What Issues Can Site Audit Identify?