Heat olive oil in a stockpot over medium-low heat. Cook and stir onion, celery, carrots, and garlic in the hot oil until onion is translucent, about 10 minutes. Add mushrooms to onion mixture and cook, stirring occasionally, until mushrooms are slightly tender, about 5 minutes.
Stir tomato paste and about 1 tablespoon water together in a bowl until tomato paste is thinned; add remaining water, marjoram, rosemary, paprika, sage, bay leaf, cayenne pepper, and pepper and stir.
Stir tomato paste mixture into onion mixture; add chicken. Bring mixture to a boil; reduce heat to medium-low, cover stockpot, and simmer until chicken is no longer pink in the center and vegetables are tender, about 40 minutes.
Stir cornstarch and cold water together in a bowl until smooth. Stir cornstarch mixture and salt into stew; bring to a boil and cook until stew is thickened, 5 to 10 minutes.
