Heat olive oil and butter in a large, heavy-bottomed pot over medium-high heat. Add onion, shallots, green onions, salt, and pepper; saute until softened and beginning to brown, 4 to 6 minutes. Add garlic and herbes de Provence; saute until garlic is fragrant, about 30 seconds. Avoid burning the garlic; it will turn bitter.
Add potatoes to onion mixture and toss well to coat. Pour in vegetable broth and milk; stir well. Cover pot and bring to a boil. Reduce heat to low and simmer, covered, until potatoes are tender and can be easily broken up with a spoon, 10 to 12 minutes.
Use a potato masher or the back of a wooden spoon to roughly mash the potato chunks until the soup is relatively smooth and the largest chunks have been broken up.
Whisk creme fraiche into the soup and taste for seasoning. Add additional pinches of salt or black pepper if the soup tastes flat.
