It takes time to build up the soil. The best thing you can do is to add organic material like leaves, hay, grass and straw to the soil. It is a good habit to always bring organic material back to the soil as soon as you have harvested something.
Many people only use manure in the spring but that is not a good way to build up the soil or to feed the plants. The plants don’t need a lot of nutrition in the spring which means that a lot is lost through leakage. It is better to add manure and organic material many times during the growing season, especially in the summer when the plants are growing fast.
To create a good soil structure you can also grow clover. Either you plant it in a bed and let it grow a whole season or you can plant it between the rows of vegetables. Cut the clover a couple of times during the season and leave the greens to compose, it enriches the soil.